Visual Informatics: Sustaining Research and Innovations, Part I - IVIC 2011

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris...

Author: Halimah Badioze Zaman | Peter Robinson | Maria Petrou | Patrick Olivier | Timothy K. Shih | Sergio Velastin | Ingela Nyström

11 downloads 1315 Views 17MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbruecken, Germany

7066

Halimah Badioze Zaman Peter Robinson Maria Petrou Patrick Olivier Timothy K. Shih Sergio Velastin Ingela Nyström (Eds.)

Visual Informatics: Sustaining Research and Innovations Second International Visual Informatics Conference, IVIC 2011 Selangor, Malaysia, November 9-11, 2011 Proceedings, Part I

13

Volume Editors Halimah Badioze Zaman Universiti Kebangsaan Malaysia, Bangi, Malaysia; [email protected] Peter Robinson University of Cambridge, UK; [email protected] Maria Petrou Imperial College, London, UK; [email protected] Patrick Olivier Newcastle University upon-Tyne, UK; [email protected] Timothy K. Shih National Central University, Jhongli City, Taiwan; [email protected] Sergio Velastin Kingston University, UK; [email protected] Ingela Nyström Uppsala University, Sweden; [email protected]

ISSN 0302-9743 e-ISSN 1611-3349 ISBN 978-3-642-25190-0 e-ISBN 978-3-642-25191-7 DOI 10.1007/978-3-642-25191-7 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011940133 CR Subject Classification (1998): I.4, I.5, I.2.10, I.3.5, I.3.7, I.7.5, F.2.2 LNCS Sublibrary: SL 6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics

© Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Preface

Visual informatics is currently a multidisciplinary field that is well accepted among researchers and industry in computer science, information technology and engineering. The basic areas of research, such as virtual real image processing and engineering; computer vision and simulation; visual computing and visualization and social computing, have been applied in various domains such as education, medical and health, finance, agriculture and security. We currently also see various Centres of Excellence (CoE) in the field of visual informatics being established in various institutions of higher learning (IHLs) around the world (Europe, USA and UK). Malaysia has just established a similar CoE called the Institute of Visual Informatics (IVI) at Universiti Kebangsaan Malaysia (UKM) or The National University of Malaysia. It is therefore important that researchers from these various CoEs, research institutes, technology centers and industry form networks and share and disseminate new knowledge in this field for the benefit of society. It is for this reason that the Institute of Visual Informatics (IVI) at UKM decided to host the Second International Visual Informatics Conference (IVIC 2011), to bring together experts in this very important research area so that more concerted efforts can be undertaken not just locally but globally. The first IVIC held in 2009 brought together experts from Asia, the UK, Oceania and the USA. This time we also managed to bring in an expert from Sweden. Like the first IVIC, this time too the conference was conducted collaboratively by the visual informatics community from various public and private universities and industry. The second conference was co-sponsored by the Malaysian Information Technology Society (MITS), Multimedia Development Corporation (MDeC), and the Malaysian Research Education Network (MyREN). The conference was co-chaired by seven professors from four different countries (UK, Sweden, Taiwan and Malaysia). The theme of the conference, ‘Visual Informatics: Sustainable Innovations for Wealth Creation,’ reflects the importance of bringing research from ‘laboratories to the market.’ It also portrayed the shared belief of the organizers (both locally and globally) of the importance of creating a seamless value-chain R&D ecosystem: from fundamental, applied research to ‘proof of concept’ and commercialization. With the slow economic trend experienced around the world today, research and innovation are more important than ever before in creating high-income jobs in order to accelerate economic growth. Thus, the relevance of the theme of the conference was apt and timely. The conference focused on four tracks related to the basic areas of visual informatics over two days (November 9 and 10, 2011) and ended with a one-day workshop (November 11, 2011). There were four keynote speakers and 75 paper presentations based on topics covered by the four main tracks mentioned ear-

VI

Preface

lier. The reviewing of the papers was conducted by experts who represented a 150-member Program Committee from Asia, Europe, Oceania and North America. Each paper was reviewed by three reviewers and the rejection rate was 60%. The reviewing process was managed using an electronic conference management system (CoMSTM ) created by the Institute of Visual Informatics, UKM. On behalf of the Organizing and Program Committees of IVIC 2011, we thank all authors for their submissions and camera-ready copies of papers, and all participants for their thought-provoking ideas and active participation in the conference. We also thank the Vice-Chancellor of UKM (host University), and Vice-Chancellors and Deans of all IT faculties of the IHLs for their support in organizing this conference. We also acknowledge the sponsors, members of the Organizing Committees, Program Committee members, support committees and individuals who gave their continuous help and support in making the conference a success. We fervently believe that IVIC will grow from strength to strength and we also hope that one day it will be held in different host countries in Asia, Europe, Oceania or North America. We also hope that IVIC will continue to provide a stimulating and enriching platform for research and innovations that will transcend religions, cultures, race and beliefs to contribute to better general human well-being. November 2011

Halimah Badioze Zaman Peter Robinson Maria Petrou Patrick Olivier Timothy Shih Sergio Velastin Ingela Nystr¨ om

Organization

The Second International Visual Informatics Conference (IVIC 2011) was organized by the Institute of Visual Informatics and Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, in collaboration with 13 local public and private universities in Malaysia, the Malaysian Information Technology Society (MITS), Multimedia Development Corporation (MDeC), Malaysian Institute of Microelectronic Systems (MIMOS) and Malaysian Research Educational Network (MYREN).

Local Executive Committee General Chair Deputy Chair Secretary Assistant Secretary I Assistant Secretary II Treasurer Assistant Treasurer I Assistant Treasurer II

Halimah Badioze Zaman (UKM) Fatimah Dato’Ahmad (UPNM) Azlina Ahmad (UKM) Nazlena Mohamad Ali (UKM) Mohd M. Kadhum (UUM) Haslina Arshad (UKM) Rabiah Abdul Kadir (UPM) Syaimak Abd Shukor (UKM)

Program Committee Program Co-chairs Halimah Badioze Zaman Peter Robinson Patrick Olivier Ingela Nystr¨ om Maria Petrou Timothy Shih Sergio Velastin

Universiti Kebangsaan Malaysia, Malaysia University of Cambridge, UK University of Newcastle Upon-Tyne, UK Uppsala University, Sweden Imperial College, UK National Central University, Taiwan Kingston University, UK

Members/Referees Europe Ahmad Khurshid Alan Smeaton Burkhard Wuensche Daniel Thalmann

Edie Rasmussen Gregor Rainer Hassan Ugail Ingela Nystr¨ om

VIII

Organization

Jian Jun Zheng Jonathon Furner Ligang He Neil Andrew Gordon Peter Robinson Rainer Malaka Sergio Velastin Tony Pridmore Ann Blandford Carol Peters Donatella Castelli Gerald Schaefer

Harold Timbleby Ingeborg Solvberg Jian J. zhang John Wilson Keith van Rjsbergen Maria Petrou Patrick Olivier Qingde Li Roy Sterritt Stephen McKenna Wenyu Liu

USA Archan Misra Dick Simmons Hshinchun Chen Josep Torellas Micheal H. Hinchey Per-Ake (Paul) Larson Vicky Markstein

Carl K. Chang Eric Wong James Hughes Joseph Urban Paul R. Croll T. Kesavadas

Asia and Oceania Abd Razak Yaakub Abdul Razak Hamdan Abdul Samad Hasan Basari Abdul Samad Shibghatullah Abdullah Gani Aboamama Atahar Ahmed Alex Orailoglu Amirah Ismail Ang Mei Choo Anup Kumar Asim Smailagic Azizah Jaafar Azizi Abdullah Azlina Ahmad Azreen Azman Azurah Abu Samah

Bahari Belaton Bryon Purves Burairah Hussin Burhanuddin Mohd Aboobaider Chen Chwen Jen Choo Wou Onn Choo Yun Huoy Christopher C. Yang Chung Jen-Yao Dayang Norhayati Abg Jawawi Dayang Rohaya Awang Rambli Dhanesh Ramachandram Dzulkifli Mohamad Edwin Mit Effirul Ikhwan Ramlan

Elankovan A. Sundararajan Faaizah Shahbodin Faieza Abdul Aziz Farid Ghani Fatimah Dato’ Ahmad Faudziah Ahmad Hajah Norasiken Bakar Halimah Badioze Zaman Hamid Abdalla Jalab Hanspeter Pfister Haslina Arshad Hwee Hua Pang Jane Labadin Jie-Wu Juan Antonio Carballo Khairulmizam Samsudin Lai Jian Ming Li Jian Zhong

Organization

Lili Nurliyana Abdullah Ling Teck Chaw Li-Zhu Zhou M. Iqbal Bin Saripan Maizatul Hayati Mohamad Yatim Maryam Nazari Masatoshi Yoshikawa Masnizah Mohd Mazleena Salleh Md. Nazrul Islam Mohamad Ishak Desa Mohammad Khatim Hasan Mohd Faizal Abdollah Mohd Khanapi Abdul Ghani Mohd Shafry Mohd Rahim Mohd. Taufik Abdullah Mun-Kew Leong Muriati Mukhtar Narhum Gershon Nazlena Mohamad Ali Nazlia Omar Ng Giap Weng Ning Zhong Nor Aniza Abdullah Nor Azan Hj Mat Zin Nor Faezah M. Yatim Nor Hasbiah Ubaidullah Norafida Ithnin Noraidah Sahari Ashaari

Norasikin Fabil Norrozila Sulaiman Norshahriah Abdul Wahab Nur’Aini Abdul Rashid Nurazzah Abd Rahman Nursuriati Jamil Osman Ghazali Patricia Anthony Phillip C.-Y. Sheu Puteh Saad Rabiah Abd Kadir Ramlah Mailok Reggie Caudill Riaza Rias Ridzuan Hussin Riza Sulaiman Roselina Sallehuddin Shahrin Sahib Shahrul Azman Mohd Noah Shahrul Azmi Mohd. Yusof Sharifah Mumtazah Syed Ahmad Abdul Rahman Sharlini R. Urs Sim Kok Swee Siti Mariyam Hj Shamsuddin Sobihatun Nur Abdul Salam Suliman Hawamdeh

Syaimak Abdul Shukor Syamsiah Mashohor Syed Nasir Alsagoff Tan Tien Ping Teddy Surya Gunawan Tengku Siti Meriam Tengku Wook Timothy Shih Tutut Herawan Wai Lam Wan Abdul Rahim Wan Mohd Isa Wan Azlan Wan Zainal Abidin Wan Fatimah Wan Ahmad Wan Mohd Nazmee Wan Zainon Wee Mee Chin Wei Zhao Willian Hayward Wong Kok Sheik Yin Chai Wang Yin-Leng Theng Zailani Mohd Nordin Zainab Abu Bakar Zainul Abidin Zaipatimah Ali Zarinah Mohd Kasirun Zulikha Jamaluddin Zulkarnain Md Ali

Local Arrangements Committee Technical Committee Head Members

IX

Halimah Badioze Zaman (UKM) Azlina Ahmad (UKM) Muriati Mukhtar (UKM) Riza Sulaiman (UKM) Nazlena Mohamad Ali (UKM) Mohd M. Kadhum (UUM) M. Iqbal Saripan (UPM)

X

Organization

Haslina Arshad (UKM) Syaimak Abd Shukor (UKM) Rabiah Abdul Kadir (UPM) Elankovan A. Sundararajan (UKM) Norazan Mat Zin (UKM) Mohammad Khatim Hassan (UKM) Wan Mohd Nazmee Wan Zainon (USM) Tengku Siti Meriam Tengku Wook (UKM) Azizah Jaafar (UKM) Bahari Belaton (USM) Wan Fatimah Wan Ahmad (UTP) Fatimah Ahmad (UPNM) Noraidah Sahari Ashaari (UKM) Ang Mei Choo (UKM) Nursuriati Jamil (UiTM) Syed Nasir Alsagoff (UPNM) Azreen Azman (UPM) Dayang Rohaya Bt Awang Rambli (UTP) Suziah Sulaiman (UTP) Riaza Mohd Rias (UiTM) Faaizah Shahbodin (UTeM) Hajah Norasiken Bakar (UTeM) Norshahriah Wahab (UPNM) Nurazzah Abdul Rahman (UiTM) Publicity Head Members

Logistic Head Members

Elankovan A. Sundararajan (UKM) Norazan Mat Zin (UKM) Mohammad Khatim Hassan (UKM) Wan Mohd Nazmee Wan Zainon (USM) Tengku Siti Meriam Tengku Wook (UKM) Azlina Ahmad (UKM) Aidanismah Yahya (UKM) Nurdiyana Mohd Yassin (UKM) Ang Mei Choo (UKM) Nursuriati Jamil (UiTM) Syed Nasir Alsagoff (UPNM) Riaza Mohd Rias (UiTM) Maslina Abdul Aziz (UiTM) Norshahriah Wahab (UPNM) Mohd Hanif Md Saad (UKM)

Organization

Financial Head Members

Workshop Head Members

Secretariat

Azizah Jaafar (UKM) Halimah Badioze Zaman (UKM) Azlina Ahmad (UKM) Wan Fatimah Wan Ahmad (UTP) Fatimah Dato Ahmad (UPNM) Riza Sulaiman (UKM) Wan Mohd Nazmee Wan Zainon (USM) Faaizah Shahbodin (UTeM) Choo Wou Onn (UTAR) Nurul Aini Kasran (UKM) Aw Kien Sin (UKM)

Conference Management System (CoMS

TM

)

Institute of Visual Informatics, Universiti Kebangsaan Malaysia

Sponsoring Institutions Universiti Kebangsaan Malaysia (UKM) Universiti Putra Malaysia (UPM) Universiti Sains Malaysia (USM) University Teknologi PETRONAS (UTP) Universiti Teknologi MARA (UiTM) Universiti Pertahanan Nasional Malaysia (UPNM) Universiti Teknologi Malaysia (UTM) Universiti Malaysia Sarawak (UNIMAS) University Malaya (UM) Universiti Utara Malaysia (UUM) Universiti Teknikal Malaysia (UTeM) Universiti Tunku Abdul Rahman (UTAR) Multimedia University (MMU) Malaysian Information Technology Society (MITS) Multimedia Corporation Malaysia (MDeC) Malaysian Research Educational Network (MyREN) Malaysian Institute of Microelectronics (MIMOS)

XI

Table of Contents – Part I

Keynotes Visualization and Haptics for Interactive Medical Image Analysis: Image Segmentation in Cranio-Maxillofacial Surgery Planning . . . . . . . . . Ingela Nystr¨ om, Johan Nysj¨ o, and Filip Malmberg

1

Evaluation of Unsupervised Segmentation Algorithms for Silhouette Extraction in Human Action Video Sequences . . . . . . . . . . . . . . . . . . . . . . . Adolfo Mart´ınez-Us´ o, G. Salgues, and S.A. Velastin

13

Video Forgery and Motion Editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joseph C. Tsai and Timothy K. Shih

23

Computer Vision and Simulation Improved Incremental Orthogonal Centroid Algorithm for Visualising Pipeline Sensor Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A. Folorunso Olufemi, Mohd Shahrizal Sunar, and Normal Mat Jusoh 3D Visualization of Simple Natural Language Statement Using Semantic Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rabiah Abdul Kadir, Abdul Rahman Mad Hashim, Rahmita Wirza, and Aida Mustapha Character Recognition of License Plate Number Using Convolutional Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Syafeeza Ahmad Radzi and Mohamed Khalil-Hani Simulation Strategy of Membrane Computing to Characterize the Structure and Non-deterministic Behavior of Biological Systems: A Case Study with Ligand-Receptor Network of Protein TGF-β . . . . . . . Muniyandi Ravie Chandren and Mohd. Zin Abdullah Development of 3D Tawaf Simulation for Hajj Training Application Using Virtual Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohd Shafry Mohd Rahim, Ahmad Zakwan Azizul Fata, Ahmad Hoirul Basori, Arief Salleh Rosman, Tamar Jaya Nizar, and Farah Wahida Mohd Yusof A Grammar-Based Process Modeling and Simulation Methodology for Supply Chain Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohsen Mohammadi, Muriati Bt. Mukhtar, and Hamid Reza Peikari

24

36

45

56

67

77

XIV


A Parallel Coordinates Visualization for the Uncapaciated Examination Timetabling Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J. Joshua Thomas, Ahamad Tajudin Khader, and Bahari Belaton A Modified Edge-Based Region Growing Segmentation of Geometric Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nursuriati Jamil, Hazwani Che Soh, Tengku Mohd Tengku Sembok, and Zainab Abu Bakar Comparison on Performance of Radial Basis Function Neural Network and Discriminant Function in Classification of CSEM Data . . . . . . . . . . . . Muhammad Abdulkarim, Afza Shafie, Radzuan Razali, Wan Fatimah Wan Ahmad, and Agus Arif Simulation for Laparoscopy Surgery with Haptic Element for Medical Students in HUKM: A Preliminary Analysis . . . . . . . . . . . . . . . . . . . . . . . . . A.R. Norkhairani, Halimah Badioze Zaman, and Azlina Ahmad

87

99

113

125

Virtual Image Processing and Engineering Detection and Classification of Granulation Tissue in Chronic Ulcers . . . Ahmad Fadzil M. Hani, Leena Arshad, Aamir Saeed Malik, Adawiyah Jamil, and Felix Yap Boon Bin

139

New Color Image Histogram-Based Detectors . . . . . . . . . . . . . . . . . . . . . . . . Taha H. Rassem and Bee Ee Khoo

151

Digital Training Tool Framework for Jawi Character Formation . . . . . . . . Norizan Mat Diah and Nor Azan Mat Zin

164

Empirical Performance Evaluation of Raster to Vector Conversion with Different Scanning Resolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bilal Abdulrahman T. Al-Douri, Hasan S.M. Al-Khaffaf, and Abdullah Zawawi Talib

176

Visualizing the Construction of Incremental Disorder Trie Itemset Data Structure (DOSTrieIT) for Frequent Pattern Tree (FP-Tree) . . . . . . . . . . . Zailani Abdullah, Tutut Herawan, and Mustafa Mat Deris

183

The Gradient of the Maximal Curvature Estimation for Crest Lines Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pan Zheng, Bahari Belaton, Iman Yi Liao, and Zainul Ahmad Rajion

196

AdaBoost-Based Approach for Detecting Lithiasis and Polyps in USG Images of the Gallbladder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Ciecholewski

206


Assessing Educators’ Acceptance of Virtual Reality (VR) in the Classroom Using the Unified Theory of Acceptance and Use of Technology (UTAUT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Niwala Haswita Hussin, Jafreezal Jaafar, and Alan G. Downe A Fuzzy Similarity Based Image Segmentation Scheme Using Self-organizing Map with Iterative Region Merging . . . . . . . . . . . . . . . . . . . Wooi-Haw Tan, Gouenou Coatrieux, Basel Solaiman, and Rosli Besar Enhancing Learning Experience of Novice Surgeon with Virtual Training System for Jig an Fixture Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . Intan Syaherra Ramli, Haslina Arshad, Abu Bakar Sulong, Nor Hamdan Mohd. Yahaya, and Che Hassan Che Haron

XV

216

226

238

Improved Gait Recognition with Automatic Body Joint Identification . . . Tze-Wei Yeoh, Wooi-Haw Tan, Hu Ng, Hau-Lee Tong, and Chee-Pun Ooi

245

A Real-Time Vision-Based Framework for Human-Robot Interaction . . . Meng Chun Lam, Anton Satria Prabuwono, Haslina Arshad, and Chee Seng Chan

257

Automated Hemorrhage Slices Detection for CT Brain Images . . . . . . . . . Hau-Lee Tong, Mohammad Faizal Ahmad Fauzi, and Su-Cheng Haw

268

CBIR for an Automated Solid Waste Bin Level Detection System Using GLCM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maher Arebey, M.A. Hannan, R.A. Begum, and Hassan Basri Image Enhancement of Underwater Habitat Using Color Correction Based on Histogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Norsila bt Shamsuddin, Wan Fatimah bt Wan Ahmad, Baharum b Baharudin, Mohd Kushairi b Mohd Rajuddin, and Farahwahida bt Mohd A New Application for Real-Time Shadow and Sun’s Position in Virtual Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hoshang Kolivand and Mohd Shahrizal Bin Sunar A New and Improved Size Detection Algorithm for Acetabular Implant in Total Hip Replacement Preoperative Planning . . . . . . . . . . . . . . . . . . . . . Azrulhizam Shapi’i, Riza Sulaiman, Mohammad Khatim Hasan, Anton Satria Prabuwono, Abdul Yazid Mohd Kassim, and Hamzaini Abdul Hamid Visual Application in Multi-touch Tabletop for Mathematics Learning: A Preliminary Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Khoo Shiang Tyng, Halimah Badioze Zaman, and Azlina Ahmad

280

289

300

307

319

XVI


Analysing Tabletop Based Computer Supported Collaborative Learning Data through Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ammar Al-Qaraghuli, Halimah Badioze Zaman, Patrick Olivier, Ahmed Kharrufa, and Azlina Ahmad High Order Polynomial Surface Fitting for Measuring Roughness of Psoriasis Lesion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ahmad Fadzil M. Hani, Esa Prakasa, Hurriyatul Fitriyah, Hermawan Nugroho, Azura Mohd Affandi, and Suraiya Hani Hussein Modelling of Reflectance Spectra of Skin Phototypes III . . . . . . . . . . . . . . M.H. Ahmad Fadzil, Hermawan Nugroho, Romuald Jolivot, Franck Marzani, Norashikin Shamsuddin, and Roshidah Baba

329

341

352

Virtual Method to Compare Treatment Options to Assist Maxillofacial Surgery Planning and Decision Making Process for Implant and Screw Placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuwaraj Kumar Balakrishnan, Alwin Kumar Rathinam, Tan Su Tung, Vicknes Waran, and Zainal Ariff Abdul Rahman

361

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

369

Table of Contents – Part II

Visual Computing Capturing Mini Brand Using a Parametric Shape Grammar . . . . . . . . . . . Mei Choo Ang, Huai Yong Chong, Alison McKay, and Kok Weng Ng Development and Usability Evaluation of Virtual Environment for Early Diagnosis of Dementia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Syadiah Nor Wan Shamsuddin, Hassan Ugail, and Valerie Lesk Usability Study of Mobile Learning Course Content Application as a Revision Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ahmad Sobri Hashim, Wan Fatimah Wan Ahmad, and Rohiza Ahmad Game Design Framework: A Pilot Study on Users’ Perceptions . . . . . . . . Ibrahim Ahmad and Azizah Jaafar

1

13

23

33

The Development of History Educational Game as a Revision Tool for Malaysia School Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.A.R. Hadi, Wan Mohd Fazdli Wan Daud, and Nurul Huda Ibrahim

39

Ontology Construction Using Computational Linguistics for E-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L. Jegatha Deborah, R. Baskaran, and A. Kannan

50

Eye Tracking in Educational Games Environment: Evaluating User Interface Design through Eye Tracking Patterns . . . . . . . . . . . . . . . . . . . . . Nurul Hidayah Mat Zain, Fariza Hanis Abdul Razak, Azizah Jaafar, and Mohd Firdaus Zulkipli Use of Content Analysis Tools for Visual Interaction Design . . . . . . . . . . . Nazlena Mohamad Ali, Hyowon Lee, and Alan F. Smeaton Improving Accessibility through Aggregative E-Learning for All Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Khairuddin Kamaludin, Noor Faezah Mohd Yatim, and Md. Jan Nordin Exploiting the Query Expansion through Knowledgebases for Images . . . Roohullah and J. Jaafar Usability Evaluation for ‘Komputer Saya’: Multimedia Courseware for Slow Learners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Norfarhana Abdollah, Wan Fatimah Wan Ahmad, and Emelia Akashah Patah Akhir

64

74

85

93

104

XVIII


Reconstruction of 3D Faces Using Face Space Coefficient, Texture Space and Shape Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sheng Hung Chung and Ean Teng Khor Research Finding for Usability Testing on ILC-WBLE . . . . . . . . . . . . . . . . Ming Chee Hoh, Wou Onn Choo, and Pei Hwa Siew Factors Affecting Undergraduates’ Acceptance of Educational Game: An Application of Technology Acceptance Model (TAM) . . . . . . . . . . . . . . Roslina Ibrahim, Rasimah Che Mohd Yusoff, Khalili Khalil, and Azizah Jaafar

114 123

135

Usability of Educational Computer Game (Usa ECG): Applying Analytic Hierarchy Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hasiah Mohamed Omar and Azizah Jaafar

147

Visual Learning through Augmented Reality Storybook for Remedial Student . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hafiza Abas and Halimah Badioze Zaman

157

Visualisation and Social Computing Preliminary Study on Haptic Approach in Learning Jawi Handwriting Skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maizan Mat Amin, Halimah Badioze Zaman, and Azlina Ahmad Scaffolding in Early Reading Activities for Down Syndrome . . . . . . . . . . . Rahmah Lob Yussof and Halimah Badioze Zaman

168 180

EduTism: An Assistive Educational System for the Treatment of Autism Children with Intelligent Approach . . . . . . . . . . . . . . . . . . . . . . . . . . I. Siti Iradah and A.K. Rabiah

193

Investigating the Roles of Assistance in a Digital Storytelling Authoring System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jumail, Dayang Rohaya Awang Rambli, and Suziah Sulaiman

205

MYNDA - An Intelligent Data Mining Application Generator . . . . . . . . . Zulaiha Ali Othman, Abdul Razak Hamdan, Azuraliza Abu Bakar, Suhaila Zainudin, Hafiz Mohd Sarim, Mohd Zakree Ahmad Nazri, Zalinda Othman, Salwani Abdullah, Masri Ayob, and Ahmad Tarmizi Abdul Ghani

217

Scaffolding Poetry Lessons Using Desktop Virtual Reality . . . . . . . . . . . . . Nazrul Azha Mohamed Shaari and Halimah Badioze Zaman

231

Augmented Reality Remedial Worksheet for Negative Numbers: Subtraction Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Elango Periasamy and Halimah Badioze Zaman

242


Developing Conceptual Model of Virtual Museum Environment Based on User Interaction Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Normala Rahim, Tengku Siti Meriam Tengku Wook, and Nor Azan Mat Zin

XIX

253

Use of RSVP Techniques on Children’s Digital Flashcards . . . . . . . . . . . . . Siti Zahidah Abdullah and Nazlena Mohamad Ali

261

Cultural Learning in Virtual Heritage: An Overview . . . . . . . . . . . . . . . . . . Nazrita Ibrahim, Nazlena Mohamad Ali, and Noor Faezah Mohd Yatim

273

i-JEN: Visual Interactive Malaysia Crime News Retrieval System . . . . . . Nazlena Mohamad Ali, Masnizah Mohd, Hyowon Lee, Alan F. Smeaton, Fabio Crestani, and Shahrul Azman Mohd Noah

284

Measurement Model to Evaluate Success of E-Government Applications through Visual Relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Norshita Mat Nayan, Halimah Badioze Zaman, and Tengku Mohd Tengku Sembok

295

A Conceptual Design for Augmented Reality Games Using Motion Detection as User Interface and Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . Azfar Bin Tomi and Dayang Rohaya Awang Rambli

305

FaceSnap: Game-Based Courseware as a Learning Tool for Children with Social Impairment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Y.Y. Chen, Wan Fatimah Wan Ahmad, and Nur Zareen Zulkarnain

316

A Visual Measurement Model on Human Capital and ICT Dimensions of a Knowledge Society (KS) Framework for Malaysia towards an Innovative Digital Economy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Halimah Badioze Zaman, A.H. Norsiah, Azlina Ahmad, S. Riza, M.A. Nazlena, J. Azizah, and M.C. Ang

323

Visualization of the Hadith Chain of Narrators . . . . . . . . . . . . . . . . . . . . . . Zarina Shukur, Norasikin Fabil, Juhana Salim, and Shahrul Azman Noah

340

A New Framework for Phylogenetic Tree Visualization . . . . . . . . . . . . . . . . Wan Mohd Nazmee Wan Zainon, Abdullah Zawawi Talib, and Bahari Belaton

348

Technical Skills in Developing Augmented Reality Application: Teachers’ Readiness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Norabeerah Saforrudin, Halimah Badioze Zaman, and Azlina Ahmad

360

Scenario-Based Learning Approach for Virtual Biology Laboratory (VLab-Bio) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Murniza Muhamad, Halimah Badioze Zaman, and Azlina Ahmad

371

XX


Towards a Multimodality Ontology Image Retrieval . . . . . . . . . . . . . . . . . . Yanti Idaya Aspura Mohd Khalid, Shahrul Azman Noah, and Siti Norulhuda Sheikh Abdullah A Visual Art Education Tool to Create Logo (APH-Pensil) Based on the Fundamental Design Theory Approach . . . . . . . . . . . . . . . . . . . . . . . . . . Halimah Badioze Zaman, H. Ridzuan, Azlina Ahmad, S. Riza, M.A. Nazlena, J. Azizah, M.C. Ang, and Haslina Arshad Different Visualization Types in Multimedia Learning: A Comparative Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Riaza Mohd Rias and Halimah Badioze Zaman Optimal Command and Control Method for Malaysian Army Small Units in a Malaysian Forest Environment: Small Unit Tactical Management System (SUTaMs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Syed Nasir Alsagoff Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

382

394

408

419

429

Visualization and Haptics for Interactive Medical Image Analysis: Image Segmentation in Cranio-Maxillofacial Surgery Planning Ingela Nystr¨ om, Johan Nysj¨ o , and Filip Malmberg Centre for Image Analysis, Uppsala University, Sweden [email protected]

Abstract. A central problem in cranio-maxillofacial (CMF) surgery is to restore the normal anatomy of the skeleton after defects, e.g., trauma to the face. With careful pre-operative planning, the precision and predictability of the craniofacial reconstruction can be significantly improved. In addition, morbidity can be reduced thanks to shorter operation time. An important component in surgery planning is to be able to accurately measure the extent of anatomical structures. Of particular interest are the shape and volume of the orbits (eye sockets). These properties can be measured in 3D CT images of the skull, provided that an accurate segmentation of the orbits is available. Here, we present a system for interactive segmentation of the orbit in CT images. The system utilizes 3D visualization and haptic feedback to facilitate efficient exploration and manipulation of 3D data.

1

Introduction

A central problem in cranio-maxillofacial (CMF) surgery is to restore the normal anatomy of the facial skeleton after defects, e.g., malformations, tumours, and trauma to the face. CMF surgery can be exceedingly difficult and timeconsuming to perform when a fracture displaces bone segments from their proper position, when bone segments are missing, or when a bone segment is located in such a position that any attempt to restore it into its original position poses considerable risk for causing further damage to vital anatomical structures, e.g., the eye or the central nervous system. There is ample evidence that careful preoperative planning can significantly improve the precision and predictability of CMF surgery as well as reduce the post-operative morbidity. In addition, the time in the operating room can be reduced and thereby also costs. An important component in surgery planning is to be able to accurately measure the extent of certain anatomical structures. Of particular interest in CMF surgery planning is to measure the shape and volume of the orbit (eye socket), comparing an intact side with an injured side. These properties can be measured in three-dimensional (3D) computed tomography (CT) images of the skull. This, however, requires accurate segmentation of the orbits. Today, orbit segmentation is usually performed by experts in CMF surgery planning who manually trace the orbit boundaries in a large number of CT H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 1–12, 2011. c Springer-Verlag Berlin Heidelberg 2011

2

I. Nystr¨ om, J. Nysj¨ o, and F. Malmberg

image slices. This manual segmentation method is accurate but time-consuming, tedious, and sensitive to operator errors. Fully automatic orbit segmentation, on the other hand, is difficult to achieve, mainly because of the high shape variability of the orbit, the thin nature of the orbital walls, the lack of an exact definition of the orbital opening, and the presence of CT imaging artifacts such as noise and partial volume effects. To overcome these issues, we propose an interactive, or semi-automatic, method for orbit segmentation, where a human guides the segmentation process. When we interact with real-life 3D objects, the sense of touch (haptics) is an essential addition to our visual perception. During the last two decades, the development of haptic devices and haptic rendering algorithms has made it possible to create haptic interfaces in which the user can feel, touch, and manipulate virtual 3D objects in a natural and intuitive way. It has been demonstrated [12] that such interfaces greatly facilitate interactive segmentation of objects in medical volume images. Here, we present a system for interactive segmentation of the orbits in CT images. The system first extracts the boundaries of the orbital bone structures and then segments the orbit by fitting an interactive deformable simplex mesh to the extracted boundaries. A graphical user interface (GUI) with volume visualization tools and haptic feedback allows the user to efficiently explore the input CT image, define anatomical landmarks, and guide the deformable simplex mesh through the segmentation.

2

Segmentation Method

In this Section, the proposed method for orbit segmentation is described. The method consists of the following main steps: 1. Segment the bone structures around the orbit. 2. Calculate, based on user defined anatomical landmarks, a planar barrier that defines the extent of the orbital opening. 3. Fit a deformable surface model to the orbit, using the data obtained in the previous steps. The steps of the segmentation method are illustrated in Figure 1. In the remainder of this Section, the above steps are described in detail. 2.1

Bone Segmentation

In this step, extract a binary image representing the bone structures around the orbit. Since the intensity of bone is well-defined in CT images, the bone structures may be segmented using hysteresis thresholding [1] with fixed threshold values. See Figure 2. A one voxel thick boundary is extracted from the segmented bone region using mathematical morphology, resulting in a boundary map.

Visualization and Haptics for Interactive Medical Image Analysis

(a)

(b)

(c)

(d)

3

Fig. 1. Overview of the proposed method for interactive orbit segmentation. (a) The user defines the extent of the orbital opening by placing four landmarks on the orbital rim. Haptic feedback facilitates the landmark positioning. (b) After the landmarks have been positioned, the user interactively initializes a deformable simplex mesh as a coarse sphere inside the orbit. (c) The simplex mesh is deformed, using information from the underlying CT data, to fit the orbit. If necessary, the user can guide the deformation process interactively, using the haptic stylus to apply (interactive) forces on selected mesh faces. (d) Resulting orbit segmentation.

4


Fig. 2. Segmentation of bone structures by hysteresis thresholding. (Left) Close-up image of the orbit in an axial slice of the volume. (Right) Result of hysteresis thresholding.

2.2

Defining the Orbital Opening

Inside the skull, the orbit is surrounded by bone that will prevent the deformable model from extending too far. At the orbital opening, however, there are no bone structures preventing the deformable model from growing indefinitely. Thus, we must explicitly define the extent of the orbital opening. The user selects four landmarks on the orbital rim. These landmarks are subsequently used to define a barrier, beyond which the deformable model is not allowed to grow. The barrier is calculated in two steps: 1. An implicit planar barrier is fitted to the four landmarks through singular value decomposition (SVD). The resulting barrier has three attributes: a normal, a centroid, and a radius. The centroid is defined as the mean value of the landmark positions. The radius is defined as the maximum distance between the centroid and the landmarks. 2. The three attributes are used to generate an explicit triangle mesh for the barrier. This mesh is rasterized, and the resulting voxels are added to the boundary map. 2.3

Deformable Model

Deformable contour models (also known as 2D snakes) were introduced by Kass, et al. [4] and extended to deformable surface models (also known as 3D snakes) by Terzopoulos, et al. [9]. Since their introduction in the late 1980s, deformable surface models have been frequently used in medical image segmentation. A deformable surface model may be thought of as an elastic surface (defined in the image domain) that is driven by the minimization of a cost function, of which local or global minima ideally should correspond to object boundaries.


5

Fig. 3. (Left) A simplex mesh and its dual triangle mesh. (Right) Interactive deformable simplex mesh segmentation of the left orbit in a CT image. The simplex mesh is deformed by external forces computed from the initial CT image and by haptic interactive forces that the user apply on selected mesh faces. Internal forces keep the simplex mesh together and prevent it from bending to much, forming loops, or leaking through boundary gaps.

Deformable surface models make few assumptions about object’s shape and can therefore be used to segment objects of widely different shapes as well as objects of high shape variability. By constraining the extracted object boundaries to be smooth, deformable surface models also offer robustness to boundary gaps and noise [14]. These properties make the deformable models a suitable tool for orbit segmentation. Several different geometric representations of deformable surface models have been proposed [5]. In this project, we have mainly used deformable 2-simplex meshes, a discrete surface model representation introduced by Delingette [2]. See Figure 3. For the remainder of this paper, we will refer to 2-simplex meshes as simplex meshes. Starting from a user-defined initial mesh, we wish to deform this mesh to accurately represent the shape of the orbit. More specifically, we seek a surface that satisfies the following properties: 1. The distance from each vertex of the mesh to the closest point in the boundary map should be minimized. 2. The surface should have some degree of smoothness. To find a surface that matches these criteria, the initial deformable simplex mesh is deformed according to Newton’s second law of motion (Newtonian evolution), μ

∂ 2 pi = Fdamp (pi ) + Fint (pi ) + Fext (pi ), ∂t2

(1)

6


(a)

(b)

(c)

Fig. 4. External forces used to deform the simplex mesh. (a) A boundary map computed from a CT image of the orbit. The barrier in front of the orbit defines the orbital opening. (b) Distance transform of the boundary map. (c) External forces obtained by taking the (inverse) gradient of the distance transform.

where Fdamp is a damping force, Fint an internal force, and Fext an external force [14]. The damping force prevents oscillations of the model. The internal force assures some geometric continuity by penalizing non-smooth surfaces. The external force takes the underlying data into account by penalizing surfaces that are far away from the boundary map. More specifically, the external force at each point in the image is defined as the (inverse) gradient of a distance transform of the boundary map. See Figure 4. In addition, we include an interactive force which constrains the mesh by user supplied forces or forces that come from higher level image understanding processes. Here, these forces are contributed via the haptic device described in Section 3.2.

3 3.1

User Interface Visualization

To visualize the CT volume image during the segmentation process, we use multi-planar reformatting (MPR) [10] and hardware accelerated direct volume rendering. See Figure 5. MPR is a simple yet effective volume visualization technique that uses arbitrarily oriented 2D planes to simultaneously display several cross-sections of a volume image. In our implementation, the CT image is reformatted in three planes, each of which is orthogonal to one of the image axes and can be translated along that axis by the user to explore the contents of the image. Direct volume rendering is used here to visualize the surface of the skull. The skull is rendered as a shaded iso-surface, using the GPU-accelerated technique described in [6]. Combined with haptic feedback, this form of visualization provides an efficient interface for positioning the anatomical landmarks used to define the barrier at the orbital opening.


7

Fig. 5. (Left) Multi-planar reformatting (MPR) of an 8-bit 512 × 512 × 133 CT image of the orbits. The user can explore the CT image by translating the three orthogonal planes along the image axes. (Right) Hardware-accelerated direct volume rendering of the skull in a CT image, here, as a shaded iso-surface.

3.2

Haptics

Haptic rendering is the process of computing and generating kinesthetic or tactile feedback in response to a user’s interaction with virtual 3D objects through a haptic device. Here, the PHANTOM Desktop haptic device (see Figure 6) from Sensable Technologies [7] is used. This device has 6 degrees-of-freedom (DOF) for input and 3 DOF for output, i.e., a position and an orientation for input and a force vector for output. The 6 DOF input facilitates natural and efficient interaction with 3D data — the user can grab, move, and rotate the volume image freely to inspect and manipulate it. In the proposed system, haptic feedback is used when placing the landmarks that define the orbital opening. The skull is then rendered as a solid haptic surface. Additionally, haptic feedback is used to convey the effects of interactive forces to the user during deformable model segmentation.

Fig. 6. (Left) The PHANTOM Desktop haptic device from Sensable Technologies. (Right) A user working with a haptic device.

8


"

!

#

$

%( %(

$

%( , %( '() ( *+ %&&

---

Fig. 7. The software architecture of WISH and WISHH3D

4

Implementation Details

We have implemented the proposed system using WISH [12,6], an open-source software package for developing haptic-enabled image processing software. The diagram in Figure 7 illustrates the software architecture of WISH. The core of WISH is a stand-alone C++ class library for image analysis, volume visualization, and haptics. Through an interface called WISHH3D, WISH is integrated with H3DAPI 1.5 [8], an open source scene graph API for graphics and haptics programming. H3DAPI implements the X3D [13] scene graph standard and uses OpenGL for graphics rendering and OpenHaptics [7] for haptics rendering. There are three programming interfaces to H3DAPI: X3D, Python, and C++. WISHH3D consists of H3DAPI scene graph nodes for the methods and algorithms implemented in WISH. These nodes are written in C++ and compiled into a dynamically linked library that can be loaded into H3DAPI applications via X3D files.


9

Table 1. Intra-operator results of the segmentation evaluation. The coefficient of variation (CV) is given in parenthesis. User Precision U1 0.964 (1.8 %) U2 0.972 (1.3 %) U3 0.969 (2.1 %)

Sensitivity 0.992 (0.3 %) 0.993 (0.4 %) 0.993 (0.4 %)

Deviation 1.5 (27 %) 1.4 (24 %) 1.4 (23 %)

Table 2. Inter-operator precision for the proposed method, compared to manual delineation. The coefficient of variation (CV) is given in parenthesis. User pair U1U2 U1U3 U2U3 Total

5

Manual 0.901 (1.6 %) 0.907 (1.5 %) 0.930 (0.9 %) 0.913 (1.9 %)

Interactive 0.954 (1.8 %) 0.950 (1.9 %) 0.969 (2.0 %) 0.957 (2.1 %)

Experiment and Results

To evaluate the performance of our segmentation system, we asked three users to segment both orbits in a set of seven CT volume images.1 To assess the repeatability of the proposed method, each user segmented each orbit twice. Thereby, a total of 84 orbits were segmented to validate the system. Additionally, the users performed manual segmentation of each orbit using the medical imaging software ITK-SNAP. Prior to this manual segmentation, the orbital openings were defined with planar barriers as described in Section 2.2. These barriers were shown to the users during the manual segmentation. We constructed crisp ground truth segmentations by shape-based interpolation [3] averaging of the manual segmentations. The time for completing the segmentation was recorded for both the interactive and the manual segmentation. The average time for manual segmentation of an orbit was approximately 17 minutes. With the proposed interactive method, the same task was completed on average in 3 minutes, i.e., the required user time for segmentation was reduced by a factor 5. The precision, accuracy, and efficiency of the proposed segmentation method was evaluated using the framework by Udupa, et al. [11]. We measured accuracy in terms of sensitivity, i.e., true positive volume fraction, and deviation, i.e., the distance between the boundaries of two segmentations. Tables 1 and 2 show some results of the evaluation. Figure 8 shows the segmentation results in a slice from a volume where the orbit is intact. The segmentation results shown in Figure 9 illustrates the ability of the proposed method to accurately segment fractured orbits. A 3D visualization of the segmentation results is shown in Figure 10. 1

The dimension of the CT images was 512 × 512 × Nz , where Nz was in the range 26–193 slices. The slice thickness Δz was 0.4–2.0 mm and the in-plane resolution in the range 0.31–0.43 mm.

10


Fig. 8. Segmentation of an intact orbit. (Left) An axial slice of a crisp ground truth segmentation constructed by shape-based interpolation of three manual segmentations. (Right) Corresponding axial slice of a rasterized simplex mesh segmentation. The left and right segmentation results are similar, except that the simplex mesh segmentation reaches deeper into boundary concavities.

Fig. 9. Segmentation of a fractured orbit. (Left) An axial slice of a crisp ground truth segmentation constructed by shape-based interpolation of three manual segmentations. (Right) Corresponding axial slice of a rasterized simplex mesh segmentation. Despite this being a difficult case to segment, the left and right segmentation results are similar with only minor leakage problems at the boundary of the simplex mesh segmentation.


(a)

(b)

(c)

(d)

11

Fig. 10. Surface renderings of (a) a crisp “true” orbit segmentation and (b)–(d) simplex mesh segmentations performed by user U1, U2, and U3, respectively. (b)–(d) highlight the robustness of the method by showing similar results despite being segmentations by different users.

6

Conclusions and Future Work

We have presented an interactive system2 for orbit segmentation in CT images intended for planning of craniomaxillo-facial (CMF) surgery. The segmentation is driven by a deformable model in a 3D visual and haptic environment. We report on high accuracy, very high precision, and several times improved efficiency (compared with manual segmentation) in an evaluation study where three users segmented almost 100 orbits. Next, we propose to make a model for the restoration of an injured face by mirroring an intact side to the fractured side. The resulting model should be used as a template for interactive registration of the bone fragments using haptic interaction. Our bottom-line is that haptic enabled 3D input devices offer exciting possibilities for interactive manipulation and exploration of 3D data.

2

Source code available at http://www.cb.uu.se/research/haptics/orbitproject

12


Acknowledgments. We are grateful to Prof. Jan-Michaél Hirsch and Dr. Elias Messo at the Department of Surgical Sciences; Oral & Maxillofacial Surgery, Uppsala University, Sweden, for providing the CT datasets. The project was supported by a grant from NovaMedTech (http://www.novamedtech.se).

References 1. Canny, J.: A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 8(6), 679–698 (1986) 2. Delingette, H.: General Object R econstruction Based on Simplex Meshes. International Journal of Computer Vision 32(2), 111–146 (1999) 3. Herman, G.T., Zheng, J., Bucholtz, C.A.: Shape-based Interpolation. IEEE Computer Graphics and Applications 12(3), 69–79 (1992) 4. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active Contour Models. International Journal of Computer Vision 1, 321–331 (1987) 5. Montagnat, J., Delingette, H., Ayache, N.: A Review of Deformable Surfaces: Topology, Geometry and Deformation. Image and Vision Computing 19, 1023– 1040 (2001) 6. Nystr¨ om, I., Malmberg, F., Vidholm, E., Bengtsson, E.: Segmentation and Visualization of 3D Medical Images through Haptic Rendering. In: Proceedings of the 10th International Conference on Pattern Recognition and Information Proceeding (PRIP 2009), pp. 43–48. Publishing Center of BSU (2009) 7. SensAble Technologies: SensAble (2011), http://www.sensable.com (accessed on April 27, 2011) 8. SenseGraphics AB: H3D API (2011), http://www.h3dapi.org (accessed on April 27, 2011) 9. Terzopoulos, D., Witkin, A., Kass, M.: Constraints on Deformable Models: Recovering 3D Shape and Nonrigid Motion. Artificial Intelligence 36(1), 91–123 (1988) 10. Udupa, J.K., Herman, G.T. (eds.): 3D Imaging in Medicine, 2nd edn. CRC Press, Inc. (2000) 11. Udupa, J.K., LeBlanc, V.R., Zhuge, Y., Imielinska, C., Schmidt, H., Currie, L.M., Hirsch, B.E., Woodburn, J.: A framework for evaluating image segmentation algorithms. Computerized Medical Imaging and Graphics 30(2), 75–87 (2006) 12. Vidholm, E.: Visualization and Haptics for Interactive Medical I age Analysis. Ph.D. thesis, Uppsala University (2008) 13. Web3D Consortium: X3D (2011), http://www.web3d.org/x3d/ (accessed on April 27, 2011) 14. Xu, C., Prince, J.L.: Snakes, Shapes, and Gradient Vector Flow. IEE Transactions on Image Processing 7(3), 359–369 (1998)

Evaluation of Unsupervised Segmentation Algorithms for Silhouette Extraction in Human Action Video Sequences Adolfo Mart´ınez-Usó, G. Salgues, and S.A. Velastin 1 Institute of New Imaging Technologies Universitat Jaume I, 12071 Castellón, Spain [email protected] 2 Ecole Nationale Suprieure de Physique De Strasbourg 67412 Illkirch, France 3 Digital Imaging Research Centre (DIRC) Faculty of Computing, Information Systems and Mathematics Kingston University, Surrey, KT1 2EE, UK [email protected]

Abstract. The main motivation of this work is to find and evaluate solutions for generating binary masks (silhouettes) of foreground targets in an automatic way. To this end, four renowned unsupervised image segmentation algorithms are applied to foreground segmentation. A comparison among these algorithms is carried out using the MuHAVi dataset of multi-camera human action video sequences. This dataset presents significant challenges in terms of harsh illumination resulting for example in high contrast and deep shadows. The segmentation results have been objectively evaluated against manually derived ground-truth silhouettes.

1 Introduction The growing importance of imaging technology for many applications has widely been recognised in the literature, nowadays being a promising research field which needs further development before practical deployment [5]. The importance of this research has been highlighted with the rapid increase of the use of CCTV systems almost everywhere. These non-intrusive imaging devices are able to detect objects or identify people in a relatively cheap way. However, the huge amounts of data generated by CCTV cameras require sophisticated image analysis techniques to potentially recognise significant events, objects or people in a video sequence. There exists many image analysis methodologies based on video sequences. These methodologies are being developed for applications such as motion capture, surveillance, traffic monitoring, video-based biometric person authentication, etc. An essential step in video-based applications is foreground segmentation, which is the process that extracts a target of interest from a given image sequence. The quality of the final result produced by a video-based application depends mostly on the accuracy obtained segmenting the tarjet [2]. There have been many attempts over the years to segment the foreground, mostly based on features derived from movement or background H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 13–22, 2011. c Springer-Verlag Berlin Heidelberg 2011

14

A. Mart´ınez-Usó, G. Salgues, and S.A. Velastin

subtraction [13,9]. Thus, some robust approaches on this task can be found [3], however, they work in a restrictive context or under important limitations [10]. Therefore, this problem is far from being solved in a general sense and novel approaches on this field are needed in order to find effective solutions particularly in noisy or low contrast conditions. The foreground segmentation task can be divided into two parts: foreground detection and foreground segmentation. The work presented here focuses on the latter part of this task, also known as silhouette extraction, through a comparison of competing algorithms. The experimental part of the work is based on the public MuHAVi human action recognition dataset [12]. In our approach, we consider a video sequence as a set of individual images (frames). The proposed framework starts from a coarse foreground detection using a state-of-theart algorithm [15] that finds out which rectangular part (blob) of the image will be segmented. The segmentation algorithms used in the comparison are well-known and widely used for image segmentation of static images [4,6,7,11]. All these algorithms are unsupervised segmentation methods that can be applied to any research field. The final segmentation results of each algorithm are evaluated using three wellknown metrics [14,8,15]. These metrics provide an objective measure of the quality of the segmentation by comparing it to the ground-truth. Therefore, a novel combination of foreground detection and image segmentation algorithms for foreground segmentation is proposed in this work. To the best of our knowledge, this idea have not been extensively evaluated for the segmentation of moving objects. In addition, this work also contributes to the growth of interesting solutions on this topic by means of knowing which of these well-accepted image segmentation algorithms performs the best for this task. A major part of the MuHAVi dataset remains without ground-truth reference, as manually labelling frame by frame is a very time-consuming task and therefore an automatic silhouette extraction is highly desirable. Thus, in addition to comparison purposes, other main motivations of this work are: – Using the results for significantly improving the final foreground segmentation of the MuHAVi dataset. – Guiding the decision on which method and parameters are appropriate necessary for this automatic silhouette extraction. – Establishing a baseline comparison for future researchers in the field.

2 Comparison Schema The proposed approach consists of several stages for generating the final binary silhouettes. These stages can be seen schematically in Fig. 1. Square objects represent instances whereas oval objects represent processes. Likewise, shapes filled with orange colour are data provided by the MuHAVi dataset. Firstly, a standard algorithm for computing roughly where the silhouette might be is used. This returns a rough binary mask (M ) of the foreground target and its bounding box coordinates (dashed lines in the figure). The frames thus extracted from the video sequences are cropped using the bounding box coordinates provided by the foreground

Evaluation of Unsupervised Segmentation Algorithms for Silhouette Extraction

15

Fig. 1. Flowchart schema of the experiments. Shapes filled with orange colour are data provided by the MuHAVi dataset.

detector. The cropped part of the frame is the input data for the segmentation algorithms. This cropped part also contains the binary mask M , obtained by the foreground detector. Examples of the M mask can be seen in Fig. 2, 3 and 4 framed in red colour. The M mask is quite rudimentary and inaccurate, thus a morphological closing operation is used to obtain M , where small holes have been reduced and gaps in contours have been attenuated. The segmentation process is applied to the whole cropped area but only those regions of the segmentation that intersect with M by more than a fraction p are selected to be part of the final foreground mask. Parameter p is a threshold value which has been found experimentally to work well when set to 80%. Fig. 2 in its bottom row shows the segmentation result (second column) and the final intersection result in a binary mask (fourth column). This resulting mask is worked out for each frame of the video sequence. An estimation of the quality of each binary mask obtained is then calculated using the ground truth reference of the frame and an objective evaluation metric. Finally, the average of the values obtained for each frame is given as a measure of the quality of the silhouette segmentation on each video sequence.

3 Segmentation Algorithms and Objective Evaluation Metrics Four well-known colour segmentation algorithms are used in our comparison. All of them are general-purpose unsupervised methods whose results have been quoted often as the reference to beat [1]. These algorithms are: 1. MS, segmentation method based on the mean shift algorithm proposed by Comaniciu and Meer in [4]. 2. In [7] Felzenszwalb and Huttenlocher presented an algorithm (FH) that adaptively adjusts the segmentation criterion based on the degree of variability in neighbouring regions of the image. 3. SRM algorithm was proposed by Nock and Nielsen in [11] based on the idea of using perceptual grouping and region merging for image segmentation. 4. JSEG algorithm proposed by Deng and Manjunath in [6] provides colour-texture homogeneous regions which are useful for salient region detection.

16


It is important to point out that the parameters of the algorithms have been set to the default values provided in the original papers, therefore, no parameters have been tuned up. The evaluation of the quality of the results has been done by means of comparing the resulting silhouettes (R) to the ground-truth binary mask (GT ). To this end, three objective metrics have been used: – The first metric is an overlapping measure (OV) that has been often used in the literature [15]. It is quite simple but efficient. For each frame k of a sequence the following expression is worked out: OV (k) = 1 −

GT ∩ R GT ∪ R

(1)

– The objective evaluation metric MPEGqm (MPEG quality measure) was proposed in [14] and has been widely adopted by the research community due to its simplicity. In addition to the error added in each frame, MPEGqm takes into account the fluctuation of the error between the current frame and the previous one. The spatial error is defined for a frame k as: Sqm(k) =

fn + f p GT ∪ R

(2)

where fn is the number of false negative pixels and fp is the number of false positive pixels. The temporal error is defined by: T qm(k) = abs(Sqm(k) − Sqm(k − 1))

(3)

Finally, the MPEGqm criteria for a frame k is: M P EGqm(k) =

Sqm(k) + T qm(k) 2

(4)

– The third metric is the measure PST (Perceptual Spatio-Temporal). It is based on objective errors and perceptual errors and has demonstrated superior performance than MPEGqm [8]. Errors produced by spatial artefacts and temporal artefacts are separated, giving different weights to each one. Four spatial artefacts are analysed regarding the ground truth reference: added regions, added background, inside holes and border holes. At the same time, the temporal artefacts are also studied regarding consecutive frames. Examples of spatial artefacts are the false pixel detections on the border of the silhouettes. These artefacts are strongly penalised because they affect the shape of the silhouettes. Examples of temporal artefacts are the fluctuation of the spatial errors from one frame to the previous one in terms of flickering and expectation effect produced. Therefore, the total annoyance found in a video object segmentation is estimated by a weighted linear combination of artefacts. PST metric has a quite long mathematical explanation, thus the reader is directed to [8] for details.


17

4 Comparison Results The MuHAVi multi-camera, multi-action dataset [12] consists of 17 action classes, 14 actors and 8 cameras for a total amount of 952 video sequences. Each actor performs an action several times in the action zone. This is collected by 8 CCTV cameras located at 4 sides and 4 corners of a rectangular platform. The video segments from each camera have been manually synchronised. Illumination is quite harsh, as the experimental setup uses standard night street lighting, intentionally so to simulate realistic conditions and to test robustness of segmentation and action recognition. Table 1 shows some of the actions of the dataset. For these video sequences, the MuHAVi dataset provides manually labelled silhouettes for an interval of frames as a ground-truth reference (column 4 in Table 1). Table 1. Evaluation sequences. First column for the segmented actions, second for the actor/camera number, third for the sequence identifier and fourth for the frames with ground truth reference. Action Act/Cam Seq. Id Run stop 1/3 1 Run stop 1/4 2 Run stop 4/3 3 Run stop 4/4 4 Kick 1/3 5 Kick 1/4 6 Kick 4/3 7 Kick 4/4 8 Punch 1/3 9 Punch 1/4 10 Punch 4/3 11 Punch 4/4 12 Shotgun Collapse 1/3 13 Shotgun Collapse 4/3 14 Shotgun Collapse 4/4 15 Walk Turn Back 1/3 16 Walk Turn Back 1/4 17 Walk Turn Back 4/3 18 Walk Turn Back 4/4 19

Frames 980 to 1417 980 to 1417 293 to 617 293 to 618 2370 to 2910 2370 to 2910 200 to 628 200 to 628 2140 to 2606 2140 to 2606 92 to 535 92 to 535 319 to 1208 267 to 1103 267 to 1103 216 to 681 216 to 681 207 to 672 207 to 672

A foreground detection has been performed on these video sequences using an experimental industrial tracker [15]. However, any reasonable foreground detector (e.g. as available in OpenCV) would be valid as well. The foreground detector outputs i) the bounding box coordinates of the region of the image that will be segmented and ii) a binary mask with a roughly detection of the target. Fig. 2, 3 and 4 show three examples of original frames extracted from the video sequences. These figures also show the foreground detection framed in red, the groundtruth silhouette with no frame and the resulting silhouettes extracted from these actions framed in green. As it can be seen, the resulting silhouettes improve significantly the

18


Fig. 2. Sample for ”RunStop” sequence (actor 4, camera 3, frame 392). Second row, from left to right: binary mask (M ), example of segmentation result for the SRM algorithm, ground-truth reference (GT) and final segmentation as a binary mask (R).

foreground detection. Some common segmentation mistakes resulting from illumination changes and shadows have been successfully avoided. The silhouettes obtained with the proposed methodology are good enough to: – give rise to a baseline silhouette segmentation of the MuHAVi database. – be directly used by video-based applications, for instance in human action recognition. In Table 2, the four segmentation algorithms have been compared by means of the proposed evaluation metrics. It shows the average values of the evaluation metrics obtained by each segmentation algorithm. The comparison has been carried out using the 19 video sequences that have ground-truths in the MuHAVi dataset (described in Table 1). The lower the values, the better the segmentation result in terms of similarity to the ground-truth reference. The first column shows a number for identifying each sequence using Table 1. For each action and for each evaluation measure, the average values obtained by each segmentation algorithm are shown, with the best one shown in bold. The SRM algorithm mostly obtains the best rates in this table, followed by the FH algorithm. Together, they account for more than 90% of the best results.


19

Fig. 3. Top row shows a sample for ”WalkTurnBack” sequence (actor 1, camera 3, frame 253). Second row, from left to right: binary mask (M ), ground-truth reference (GT ) and final segmentation as a binary mask (R).

4.1 Implementation and Computational Issues One of the key implementation issues in this comparison is that three of the algorithms have been implemented using C++ (FH, JSEG and MS) whereas the SRM algorithm has been run using MATLAB. Likewise, the computing cost analysis here only considers the segmentation process and not the preliminary blob detection, so as to focus on segmentation. Taking this in mind, an illustrative example of the computational costs for these image segmentation algorithms is presented. It has been performed using an Intel Core i7 4 CPU 2.80GHz. The cropped part of the frame images of the Punch video sequence (actor = 4, camera = 4, id = 12) has been used (the part that is actually segmented), that means 444 images of different sizes. These cropped parts of the sequence have an average height × width of 254.77 × 91.33 pixels with a standard deviation of 9.11 × 32.80 pixels for each dimension respectively. Table 3 gives a simple quantitative analysis of the computational cost of each algorithm using the CPU time obtained for the entire sequence (ES) and the average time per frame (PF). Note that no optimisations were carried out but a direct implementation of the algorithms described were run1 . As it can be seen, FH and MS algorithms are computationally more affordable than the SRM and JSEG algorithms. 1

Regarding optimisations, the authors are aware that there exits projects where MS algorithm has even been used in a real-time application (see CASBliP project at http://casblipdif.webs.upv.es/).

20


Table 2. Objective evaluation of the segmentation quality. First column for the sequence identifiers and second to fourth for the evaluation metrics. Each column of the evaluation metrics is in turn divided into four columns, one for each segmentation algorithm.

Seq. id 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

FH 0.294 0.323 0.333 0.229 0.202 0.181 0.197 0.147 0.321 0.172 0.196 0.174 0.266 0.286 0.300 0.503 0.282 0.297 0.258

OV JSEG MS 0.260 0.280 0.324 0.275 0.301 0.331 0.258 0.263 0.204 0.242 0.200 0.194 0.193 0.216 0.203 0.192 0.251 0.263 0.173 0.177 0.191 0.193 0.221 0.210 0.233 0.335 0.243 0.388 0.328 0.375 0.470 0.491 0.274 0.240 0.280 0.285 0.285 0.264

SRM 0.250 0.279 0.280 0.250 0.175 0.175 0.176 0.193 0.208 0.156 0.171 0.216 0.209 0.237 0.311 0.447 0.242 0.242 0.273

FH 0.158 0.175 0.180 0.129 0.110 0.103 0.110 0.084 0.167 0.094 0.128 0.098 0.144 0.151 0.159 0.264 0.156 0.161 0.143

MPEGqm JSEG MS 0.154 0.174 0.189 0.160 0.179 0.206 0.152 0.157 0.117 0.165 0.122 0.122 0.114 0.137 0.125 0.115 0.148 0.170 0.104 0.105 0.110 0.112 0.131 0.118 0.143 0.209 0.147 0.255 0.194 0.214 0.258 0.294 0.163 0.147 0.164 0.194 0.168 0.153

SRM 0.149 0.162 0.171 0.150 0.100 0.102 0.104 0.115 0.131 0.089 0.099 0.125 0.124 0.135 0.174 0.252 0.142 0.148 0.158

FH 0.686 0.235 0.798 0.104 0.595 0.087 0.665 0.087 0.542 0.056 0.704 0.569 0.304 0.599 0.575 0.857 0.142 0.643 0.174

PST JSEG MS 0.694 0.805 0.292 0.263 0.855 0.933 0.129 0.448 0.582 0.657 0.116 0.417 0.616 0.772 0.128 0.139 0.532 0.685 0.097 0.322 0.681 0.690 0.596 0.630 0.329 0.873 0.342 0.934 0.683 0.851 0.888 0.990 0.169 0.501 0.642 0.866 0.212 0.199

SRM 0.681 0.260 0.792 0.157 0.580 0.102 0.595 0.129 0.520 0.109 0.698 0.632 0.298 0.317 0.658 0.863 0.162 0.641 0.207

Table 3. Processing times for each of the analysed methods using id = 12 video sequence. First column shows the image segmentation algorithms, second column shows the time for the entire sequence (ES) whereas third column shows the average time per frame (PF). Time (ES) FH 7.79s JSEG 3m 2.98s MS 8.23s SRM 1m 33.02s

Time (PF) 0.0175s 0.4121s 0.0185s 0.2095s


21

Fig. 4. Top row shows a sample for a ”ShotGunCollapse” sequence (actor 1, camera 3, frame 375), the rest of the rows, from top to bottom: binary mask (M ), ground-truth (GT ) and the final segmentation (R)

5 Conclusions In this work, a comparison of four well-known image segmentation methods has been presented. The comparison has been done in a silhouette extraction framework using the MuHAVi dataset of multi-camera human action video sequences and by means of three well-known objective evaluation metrics from the literature. The segmentation algorithms take profit from a binary mask provided by a foreground blob detector to just segment the part of the frame where the target is and to select the regions that form part of the foreground silhouette. A methodology and a baseline comparison for achieving an automatic silhouette extraction have been shown in this work. Likewise, experiments demonstrate that combining foreground blob detection and image segmentation algorithms is efficient in obtaining an improved binary mask of the foreground target. The comparison shows that

22


the SRM and FH algorithms (in this order) are the most suitable ones for this task, obtaining the best performance in almost all the tested actions. An analysis of the computational cost of each algorithm has also been offered. MS and FH algorithms have shown very similar performance, being computationally more affordable than the SRM and JSEG algorithms. A further refinement of the final silhouette extraction is planned as a future work by means of using more robust foreground detectors. These results will be made available as a part of the MuHAVi dataset. Acknowledgements. This work was supported by the Spanish Ministry of Science and Innovation under the project Consolider Ingenio 2010 CSD2007-00018 and under the ”José Castillejo” sub-programme for the mobility of Spanish university lecturers.

References 1. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: From contours to regions: An empirical evaluation, pp. 2294–2301 (2009) 2. Chen, C., Liang, J., Zhao, H., Hu, H., Tian, J.: Frame difference energy image for gait recognition with incomplete silhouettes. Pattern Recognition Letters 30(11), 977–984 (2009) 3. Chen, D., Denman, S., Fooes, C., Sridharan, S.: Accurate silhouettes for surveillance - improved motion egmentation using graph cuts. In: Proceedings of the 2010 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2010, pp. 369– 374. IEEE Computer Society (2010) 4. Comaniciu, D., Meer, P.: Robust analysis of feature spaces: Color image segmentation. In: IEEE Conf. Computer Vision and Pattern Recognition, pp. 750–755 (1997) 5. Davies, E.R.: Image analysis in crime: progress, problems and prospects. In: IEE Seminar Digests, pp. 105–112 (2005) 6. Deng, Y., Manjunath, B.S.: Unsupervised segmentation of color-texture regions in images and video. IEEE Trans. on PAMI 23(8), 800–810 (2001) 7. Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. International Journal of Computer Vision 59(2), 167–181 (2004) 8. Gelasca, E.D., Ebrahimi, T.: On evaluating metrics for video segmentation algorithms. Invited paper of the Workshop on Video Processing and Quality Metrics, VPQM (2006) 9. Hu, W., Tan, T., Wang, L., Maybank, S.: A survey on visual surveillance of object motion and behaviours. IEEE Trans. on System, Man, and Cybernetics, Part C: Applications and Reviews 34(3), 334–352 (2004) 10. Lee, W., Woo, W., Boyer, E.: Silhouette segmentation in multiple views. IEEE Trans. on PAMI 33(7), 1429–1441 (2011) 11. Nock, R., Nielsen, F.: Statistical region merging. IEEE Trans. on PAMI 26(11), 1452–1458 (2004) 12. Singh, S., Velastin, S.A., Ragheb, H.: MuHAVi: A multicamera human action video dataset for the evaluation of action recognition methods, pp. 48–55 (2010) 13. Wang, L., Hu, W., Tan, T.: Recent developments in human motion analysis. Pattern Recognition 36(3), 585–601 (2003) 14. Wollborn, M., Mech, R.: Refined procedure for objective evaluation of video object generation algorithms. Doc. ISO/IEC JTC1/SC29/WG11 M3448 1648 (1998) 15. Yin, F., Makris, D., Velastin, S.A., Orwell, J.: Quantitative evaluation of different aspects of motion trackers under various challenges. British Machine Vision Association (5), 1–11 (2010)

Video Forgery and Motion Editing Joseph C. Tsai1 and Timothy K. Shih2 1

Multimedia Information Network Lab Tamkang University, Taiwan [email protected] 2 Multimedia Information Network Lab National Central University, Taiwan [email protected]

Abstract. Video Forgery is a technique for generating fake video by altering, combining, or creating new video contents. We change the behavior of actors in a video. For instance, the outcome of a 100-meter race in the Olympic Game can be falsified. We track objects and segment motions using a modified mean shift mechanism. The resulting video layers can be played in different speeds and at different reference points with respect to the original video. In order to obtain a smooth movement of target objects, a motion interpolation mechanism is proposed based on reference stick figures (i.e., a structure of human skeleton) and video inpainting mechanism. The video inpainting mechanism is performed in a quasi-3D space via guided 3D patch matching. Interpolated target objects and background layers are fused. It is hard to tell whether a falsified video is the original. In addition, in this talk, we demonstrate a new technique to allow users to change the dynamic texture used in a video background for special effect production. For instance, the dynamic texture of fire, smoke, water, cloud, and others can be edited through a series of automatic algorithms. Motion estimations of global and local textures are used. Video blending techniques are used in conjunction with a color balancing technique. The editing procedure will search for suitable patches in irregular shape blocks, to reproduce a realistic dynamic background, such as large waterfall, fire scene, or smoky background. The technique is suitable for making science fiction movies. We demonstrate the original and the falsified videos in our website at http://www.csie.ncu.edu.tw/~tshih. Although video falsifying may create a moral problem, our intension is to create special effects in movie industry. Keywords: Video Forgery, Object Tracking, Motion Estimation, Special Effect Production.

H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, p. 23, 2011. © Springer-Verlag Berlin Heidelberg 2011

Improved Incremental Orthogonal Centroid Algorithm for Visualising Pipeline Sensor Datasets A. Folorunso Olufemi, Mohd Shahrizal Sunar, and Normal Mat Jusoh Department of Graphics & Multimedia, Faculty of Computer Science and Information Systems, Universiti Teknologi Malaysia, 81310, Skudai, Johor [email protected], [email protected], [email protected]

Abstract. Each year, millions of people suffer from after-effects of pipeline leakages, spills, and eruptions. Leakages Detection Systems (LDS) are often used to understand and analyse these phenomena but unfortunately could not offer complete solution to reducing the scale of the problem. One recent approach was to collect datasets from these pipeline sensors and analyse offline, the approach yielded questionable results due to vast nature of the datasets. These datasets together with the necessity for powerful exploration tools made most pipelines operating companies “data rich but information poor”. Researchers have therefore identified problem of dimensional reduction for pipeline sensor datasets as a major research issue. Hence, systematic gap filling data mining development approaches are required to transform data “tombs” into “golden nuggets” of knowledge. This paper proposes an algorithm for this purpose based on the Incremental Orthogonal Centroid (IOC). Search time for specific data patterns may be enhanced using this algorithm. Keywords: Pigging, heuristics, incremental, centroid, Visual Informatics.

1 Introduction Pipelines are essential components of the energy supply chain and the monitoring of their integrities have become major tasks for the pipeline management and control systems. Nowadays pipelines are being laid over very long distances in remote areas affected by landslides and harsh environmental conditions where soil texture that changes between different weathers increase the probability of hazards not to mention the possibility of third party intrusion such as vandalism and deliberate attempt of diversions of pipeline products. It is widely accepted that leakages from pipelines have huge environmental, cost and image impacts. Conventional monitoring techniques such as the LDSs could neither offer continuous pipeline monitoring over the whole pipeline distance nor present the required sensitivity for pipeline leakages or ground movement detection. Leakages can have various causes, including excessive deformations caused by earthquakes, landslides, corrosion, fatigue, H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 24–35, 2011. © Springer-Verlag Berlin Heidelberg 2011

Improved Incremental Orthogonal Centroid Algorithm

25

material flaws or even intentional or malicious damaging. Pipeline sensors datasets are structurally different and fundamentally unique for so many reasons. In the first place, these data are generated asynchronously, example of sensor datasets obtained from the velocity-vane anemometer is shown in Table 1. Secondly, they are filled with noises. Thirdly, they come in unrelated units and formats, making comparison very difficult. Example, the temperature is measured in degree Celsius while the Velocity is measured in m/s2. Table 1. Data Attributes and Variables from the Velocity-Vane Anemometer

Pressure (N/m2 )

Temp. (0C)

1.002312 1.002202 0.903421 1.002212 0.960620 1.002801 1.002376 .

19.302978 19.302990 19.302990 19.302978 18.999996 18.999996 19.302978 18.999996 .

Vol. (M3/H) x E-03 0.0055546 0.0055544 0.0055546 0.0055544 .

Flow Velocity (m/s) 12.002302 12.002302 12.003421 12.004523 12.005620 12.002302 12.002302 .

External body Force EBF (N) 0.000344 0.002765 0.003452 0.003564 0.005423 0.005642 .

A central problem in scientific visualisation is to develop an acceptable and resources efficient representation for such complex datasets [1, 2, 3]. The challenges of high dimensional datasets vary significantly across many factors and fields. Some researchers including [4, 5] viewed these challenges as scientifically significant for positive theoretical developments.

2 Literature Review Historically, the Principal Components Analysis (PCA) originally credited to Pearson (1901) whose first appearance in modern literatures dates back to the work by Hotelling (1933) was a popular approach to reducing dimensionality. It was formerly called the Karhunen-Loeve procedure, eigenvector analysis and empirical orthogonal functions. The PCA is a linear technique that regards a component as linear combinations of the original variables . The goal of PCA is to find a subspace whose basic vectors correspond to the directions with maximal variances. Let X be an dxp matrix obtained from sensor datasets for example, where d represents the individual data attributes (columns) and p the observations (or variables) that is being measured. Let us further denote the covariance matrix C that defined X explicitly as:

26

A. Folorunso Olufemi, M.S. Sunar, and N.M. Jusoh

C

∑

x

x x

x

T

(1.0)

Where x Є X and x is the mean of x , T is the positional orders of x Є X, and X is the covariance matrix of the sampled data. We can thus define an objective function as: G w

W T CW

(2.0)

The PCA’s aims is to maximise this stated objective function G(w) in a solution space defined by: H

WЄ R

, WT W

I

(3.0)

It has been proved that the column vectors of W are the p higher or maxima eigenvectors of covariance matrix C defined above [see 8]). However, for very large and massive datasets like the pipeline sensors datasets, an enhancement of the PCA called the Incremental PCA developed by [9, 10] could be a useful approach. The IPCA is an incremental learning algorithm with many variations. The variations differ by their ways of incrementing the internal representations of the covariance matrix. Although both the PCA and the IPCAs are very effective for most data mining applications, but, because they ignore the valuable class label information in the entire data space, they are inapplicable for sensor datasets. The Linear Discriminant Analysis (LDA) emerged as another approach commonly used to carry out dimensionality reduction. Its background could be traced to the PCA and it works by discriminating samples in their different classes. Its goal is to maximize the Fisher criterion specified by the objective function: (4.0) ∑ p m x m x T and s ∑ p E x m x m T with Where s x Є ci are called the Inter class scatter matrix and Intra class scatter matrix respectively. E denotes the expectation and p x is the prior probability of a variable (x) belonging to attribute (i). W can therefore be computed by solving arg max G w in the solution space H WЄ R , W T W I , in most w reports; this is always accomplished by providing solution to the generalized eigenvalue decomposition problem represented by the equation: Sbw =λSww

(5.0)

When the captured data is very large like in the case of sensors datasets considered in this research, LDA becomes inapplicable because it is harder and computationally expensive to determine the Singular Value Decomposition (SVD) of the covariance matrix more efficiently. LDA uses attribute label information of the samples, which has been found unsuitable by many researchers including [5] for numerical datasets. [11] had developed a variant of the LDA called the Incremental LDA (ILDA) to solve the problem of inability to handle massive datasets, but, its stability for this kind of application remains an issue till present date. The Orthogonal Centroid (OC) algorithm by [12 and13] is another acceptable algorithm that uses orthogonal transformation on centroid of the covariance matrix. It has been proved to be very effective for classification problems by [14] and it is based


27

on the vector space computation in linear algebra by using the QR matrix decomposition where Q is an orthogonal matrix and R is an upper triangular matrix (Right Triangular Matrix) of the covariance matrix. The Orthogonal Centroid algorithm for dimensionality reduction has been successfully applied on text data (see [12]). But, the time and space cost of QR decomposition are too expensive for largescale data such as Web documents. Further, its application to numerical data or multivariate and multidimensional datasets of this sort remains a research challenge till date. In 2006, a highly scalable incremental algorithm based on the OC algorithm called the Incremental OC (IOC) was proposed by [5]. Because OC largely depends on the PCA, it is therefore not out of focus to state that the IOC is also a relaxed version of the conventional PCA. IOC is a one-pass algorithm. As dimensionality increases and defiles batch algorithms, IOC becomes an immediate alternative. The increase in data dimensionality could now be treated as a continuous stream of datasets similar to those obtainable from the velocity vane thermo-anemometer (VVTA) sensors and other data capturing devices, and then we can compute the low dimensional representation from the samples given, one at a time with user defined selection criterion Area of Interest (AOI) (iteratively). This reassures that the IOC is able to handle extremely large datasets. However, because of its neglect of the variables with extremely low eigenvalues, it is poised to be insensitive to outliers. Unfortunately, this is the case with the kind of data used in this research. There is therefore a necessity to improve the IOC algorithm to accommodate the insurgencies and the peculiarity presented by pipeline sensor datasets. The derivation of the IOC algorithm as well as the improvement proposed to the algorithm is discussed in detail in the following subsections.

3 IOC Derivation and the Proposed (HPDR) Improvement Basic Assumption 1: The IOC optimization problem could be restated as max ∑

WT S W

(6.0)

The aim of this is to optimise equation 6.0 with W Є Xdxp, where the parameters have their usual meanings. However, this is conditional upon wiwiT=1 with i=1,2,3,….p. Now, p belongs to the infinitely defined subspace of X, but, since it is not possible to select the entire variables for a particular data attribute at a time, we introduced a bias called Area of Interest (AOI) to limit each selection from the entire data space. A Lagrange function L is then introduced such that: Or

L w ,λ L w ,λ

∑ ∑

w S wT w S wT

λ w wT 1 λ w wT λ

(7.0)

(Observe that if wkwkT=1, then equation (7.0) is identically (6.0)) With λ being the Lagrange multipliers, at the saddle point, L must = 0. Therefore, it means S w T λ w T necessarily. Since obviously p >>>AOI at any point in time, this means that, w, the columns or attributes of W are p leading vectors of S . S n Can be computed therefore by using:

28


∑AOI p n

S n

m n

m n

m n

m n

T

(8.0)

Where mj(n) is the mean of data attribute j at step i and m(i) is the mean of variables at step i. T is the order of the variable in the covariance matrix defined by data space X. To dance around this problem, the Eigen Value Decomposition (EVD) is the approach that is commonly used although it has been reported to have high computation complexity problems. The EVD is computed by following the following procedure: Given any finite data samples X={x1,x2,x3,...,xn} we first compute the mean of xi by using the conventional formula: ∑ x

µ

(9.0)

This is followed by the computation of the covariance C defined as: C

∑ x

x x

T

x

(10.0)

Next, we compute the eigenvalue λ(s) and eigenvectors e(s) of the matrix C and iteratively solve: Ce=λe

(11.0)

PCA then orders λ by their magnitudes such that λ1 >λ2>λ3>... >λn , and reduces the dimensionality by keeping direction e such that λ ^

Fig. 1. Comparing Dimensionality Reduction Algorithms

From the graph in Figure 1, the HPDR shows a lot of promises for higher selection of AOI although this has not been tested beyond 15 rows of selected variables at any single time due to the limitations imposed by the renderer. Acknowledgements. This work is supported by the UTMViCubeLab, FSKSM, Universiti Teknologi Malaysia. Special thanks to (MoHE), Malaysia and the Research Management Centre (RMC), UTM, through Vot.No. Q.J130000.7128.00J57, for providing financial support and necessary atmosphere for this research.

References 1. Goodyer, C., Hodrien, J., Jason, W., Brodlie, K.: Using high resolution display for high resolution 3d cardiac data. The Powerwall, pp. 5–16. University of Leeds (2009); The Powerwall Built from standard PC components of 7computers

34


2. Ebert, D.S., Rohrer, R.M., Shaw, C.D., Panda, P., Kukla, J.M., Roberts, D.A.: Procedural shape generation for multi-dimensional data visualisation. Computers and Graphics 24, 375–384 (2000) 3. Masashi, S.: Dimensionality reduction of multimodal labeled data by local Fisher Discriminant analysis. Journal of Machine Learning Research 8, 1027–1061 (2007) 4. Donoho, D.L.: High-dimensional data analysis. The curses and blessings of dimensionality. In: Lecture delivered at the Mathematical Challenges of the 21st Century Conference, August 6-11. The American Math. Society, Los Angeles (2000) 5. Yan, J., Benyu, Z., Ning, L., Shuicheng, Y., Qiansheng, C., Weiguo, F., Qiang, Y., Xi, W., Zheng, C.: Effective and Efficient Dimensionality Reduction for Large-Scale and Streaming Data Preprocessing. IEEE Transactions on Knowledge And Data Engineering 18(3), 320–333 (2006) 6. da Silva-Claudionor, R., Jorge, A., Silva, C., Selma, R.A.: Reduction of the dimensionality of hyperspectral data for the classification of agricultural scenes. In: 13th Symposium on Deformation Measurements and Analysis, and 14th IAG Symposium on Geodesy for Geotechnical and Structural Engineering, LNEC Libson May, 2008LBEC, LIBSON, May 12-15, pp. 1–10 (2008) 7. Giraldo, L., Felipe, L.F., Quijano, N.: Foraging theory for dimensionality reduction of clustered data. Machine Learning 82, 71–90 (2011), doi:10.1007/s10994-009-5156-0 8. Vaccaro, R.J.: SVD and Signal Processing II: Algorithms, Analysis andApplications. Elsevier Science (1991) 9. Artae, M., Jogan, M., Leonardis, A.: Incremental PCA for OnLine Visual Learning and Recognition. In: Proceedings of the 16th International Conference on Pattern Recognition, pp. 781–784 (2002) 10. Weng, J., Zhang, Y., Hwang, W.S.: Candid Covariance Free Incremental Principal Component Analysis. IEEE Transaction on Pattern Analysis and Machine Intelligence 25, 1034–1040 (2003) 11. Hiraoka, K., Hidai, K., Hamahira, M., Mizoguchi, H., Mishima, T., Yoshizawa, S.: Successive Learning of Linear Discriminant Analysis: Sanger-Type Algorithm. In: Proceedings of the 14th International Conference on Pattern Recognition, pp. 2664–2667 (2004) 12. Jeon, M., Park, H., Rosen, J.B.: Dimension Reduction Based on Centroids and Least Squares for Efficient Processing of Text Data. Technical Report MN TR 01-010, Univ. of Minnesota, Minneapolis (February 2001) 13. Park, H., Jeon, M., Rosen, J.: Lower Dimensional Representationof Text Data Based on Centroids and Least Squares. BIT Numerical Math. 43, 427–448 (2003) 14. Howland, P., Park, H.: Generalizing Discriminant Analysis Using the Generalized Singular Value Decomposition. IEEE Trans. Pattern Analysis and Machine Intelligence 26, 995– 1006 (2004) 15. Han, J., Kamber, M.: Data Mining, Concepts and Techniques. Morgan Kaufmann (2001) 16. Mardia, K.V., Kent, J.T., Bibby, J.M.: Multivariate Analysis. Probability and Mathematical Statistics. Academic Press (1995) 17. Friedrnan, J.H., Tibshiirani, R.: Elements of Statistical Learning: Prediction. Inference and Data Mining. Springer, Heidelberg (2001) 18. Boulesteeix, A.: PLS Dimension reduction for classification with microarray data. Statistical Applications in Genetics and Molecular Biology 3(1), Article 33, 1–30 (2004) 19. Hand, D.J.: Discrimination and Classification. John Wiley, New York (1981) 20. Quinlan, J.R.: Induction of decision trees. Machine Learning 1, 81–106 (1986) 21. Quinlan, J.R.: Programs for Machine Learning. Morgan Kaufman (1993)


35

22. Cox, T.F., Cox, M.A.A.: Multidimensional Scaling, 2nd edn. Chapman and Hall (2001) 23. Hoppe, H.: New quadric metric for simplifiying meshes with appearance attributes. In: Proceedings IEEE Visualisation 1999. IEEE Computer Society Press (1999) 24. Hyvärinen, A.: Survey on independent component analysis. Neural Computing Surveys 2, 94–128 (1999) 25. Levoy, M.P.K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., Davis, J., Ginsberg, J., Shade, J., Fulk, D.: The Digital Michelangelo Project. 3D scanning of large statues. In: Proceedings of ACM SIGGRAPH 2000. Computer Graphics Proceedings, Annual Conference Series, pp. 131–144. ACM (2000) 26. Lee, T.W.: Independent Component Analysis: Theory and Applications. Kluwer Academic Publishers (2001)

3D Visualization of Simple Natural Language Statement Using Semantic Description Rabiah Abdul Kadir, Abdul Rahman Mad Hashim, Rahmita Wirza, and Aida Mustapha Faculty of Computer Science and Information Technology Universiti Putra Malaysia, 43400 UPM Serdang, Selangor [email protected], [email protected], {rahmita,aida}@fsktm.upm.edu.my

Abstract. Visualizing natural language description is a process of generating 3D scene from natural language statement. Firstly, we should consider the real world visualization and find out what are the keys of visual information that can be extracted from the sentences which represents the most fundamental concepts in both virtual and real environments. This paper focuses on method of generating the 3D scene visualization using semantic simplification description based on logical representation. Based on semantic simplification’s description, a concept of parent-child objects relationship is derived. This concept is used as the foundation in determining spatial relationships between objects. Aim of this study is to analyze and match the visual expression and the key information using visual semantic simplification description which presented in logical form. Based on the experimental result, it shows that 60% of the phrases were able to give appropriate depiction as the meaning of the phrase. Keywords: 3D Visualization, Natural Language Representation, Logical Approach, Semantic Technology, Visual Informatics.

1 Introduction A transformation tool of text to 3D visual is very helpful in storyboarding for game, animation and film making industry. Usually human can easily perceived messages through scene depiction rather than analyzing messages through the whole lines of text. The aim of this study is to investigate the 3D scene visualization using semantic simplification that based on the logical representation. This study is demonstrated through a prototype tool that we developed to illustrate a text document in natural language input, which presented in logical form into 3D visual expression. Form the visualization we are to establish a correspondence between word and 3D object models, and a sentence / phrase and the scene it may evoke. Finally we want to analyze and match the visual expression and the key information using visual semantic simplification description based on logical representation. We divided this paper as follows; Section 2 is about the related works, Section 3 explanation on the implementation of the 3D visualization, Section 4 the experimental result, the discussion in Section 5 and finally the conclusion and briefly future work of this study in Section 6. H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 36–44, 2011. © Springer-Verlag Berlin Heidelberg 2011

3D Visualization of Simple Natural Language Statement Using Semantic Description

37

2 Related Works Natural language input has been studied in a number of very early 3D graphics systems [4][6][7] and the Put system [3], which was limited to spatial arrangements of existing objects in a pre-constructed environment. Also, input was restricted to an artificial subset of English consisting of expressions of the form Put(X, P, Y), where X and Y are objects and P is a spatial preposition. Several more recent systems target animation rather than scene construction. Work at the University of Pennsylvania’s Center of Human Modeling and Simulation [2] used language to control animated characters in a closed virtual environment. CarSim [11] is domain-specific system where short animations were created from natural language descriptions of accident reports. CONFUCIUS [14] is a multimodal animation system that takes as input a single sentence containing an action verb. The system blends animation channels to animate virtual human characters. Another recent system, from the University of Melbourne [18], uses a machine learning-based approach to create animated storyboards on a pre-made virtual stage. In these systems the referenced objects and attributes are typically relatively small in number or targeted to specific pre-existing domains.

3 Implementation The visualization input is a natural language text that will be translated into a logical representation. The translation from text to logical representation is done using the work by [1] and the process is not discussed in this paper. The visualization tool developed in this study was using 3D modeling tool called Blender3D [20] that uses the built-in Python scripting in order to work with their Blender API. We will discuss on how the objects and attributes are obtained from the logical representation, building the 3D object through mapping the vocabularies and the 3D objects, and also handling issues with 3D objects. 3.1 Logical Representations The natural language text is an input categorized of prepositional phrase consists of two possible nouns; either a simple noun or a noun with modifier. There are three types of logical representation is generated from the input phrases as follows. i. Noun with modifier. Logical representation of this type will be converted into such a way: u(v), where u=adjective, v=noun. e.g. Natural Language: Blue box. Logical Representation: blue(box) ii. Prepositional phrase with simple noun Logical representation of this type will be converted in such a way: w(x,y), where w=preposition, x=first noun, y=second noun e.g. Natural Language: Boy on a table. Logical Representation: on(boy,table) iii. Prepositional phrase with noun and modifier Logical representation of this type will converted be in such a way: x1=u(v), where u=adjective, v=noun.

38

R. Abdul Kadir et al.

w(x1,y), where w=preposition, x1=noun with modifier, y=second noun e.g. Natural Language: Blue box under the white table. Logical Representation: x1=blue(box) y1=white(table) under(x1,y1) With the logical representation in place the 3D visualization will need to determine which of the structure of the logical representation is the adjective, preposition, object and subject of the sentence. Using Python regular expression function, the logical representation pattern is processed to extract these attributes out and keep them in the variables that will be called later for mapping the vocabularies with their respective 3D objects, attributes and spatial positioning. 3.2 3D Visualization 3D model is created based on the logical representation. We are using Blender 3D to constructs 3D scene. During the implementation, 3D object is called for scene composition. The object file contains a collection of vertices that largely describe the 3D object. We are implementing the Python Blender API to called the object and brought it onto the 3D space through the bpy.ops.wm.link_append() function. Coordinate space in Blender is an important key in positioning the 3D objects. Blender represents locations in a scene by their coordinates. The coordinates of a location consist of three numbers that define its distance and direction from a fixed origin. More precisely: The first (or x-) coordinate of the location is defined as its distance from the YZ plane (the one containing both the Y and Z axes). Locations on the +X side of this plane are assigned positive x-coordinates, and those on the -X side are given negative ones. Its second (or y-) coordinate is its distance from the XZ plane, with locations on the -Y side of this plane having negative y-coordinates. Its third (or z-) coordinate is its distance from the XY plane, with locations on the -Z side of this plane having negative z-coordinates. 3.2.1 3D Object Dimension 3D models used in this project are made of the 3D modeling tool call Blender 3D. Blender 3D is a 3D model software than can model, shade, animate, render and composite 3D objects like most of 3D software does. It is free software released under GNU General Public License. In Blender 3D, 3D model is saved in file with the extension of .blend. In this .blend file the data that are used by Blender 3D to construct a 3D scene are kept in folders. The following are common list of folder structure inside a Blender3D model file. 3.2.2 Uneven Surfaces If two objects “Book on the box.” the following steps will apply. 1. Conversion the natural language to the logical representation, will generate a logical representation of on(book,box).


39

2. From the logical representation of on(book,box), the identified items will be on as the preposition, book as the child object and table as the parent object. 3. According to the parent – child approach, the parent object has to be pun the 3D space first and the parent objects properties should be collected prior to visualizing the child object. The object properties needed are position point of local coordinate and the dimension of x, y and z. 4. The moving direction for the child object is determined by the preposition word. The moving direction will result in minus or increment of location point on axis x, y or z to get to the point for child object. 5. Then the distance for the local coordinate of parent object is calculated either by minus or plus the dimension figure of the parent object with local coordinate of the parent object on the axis x, y or z. Consider a phrase of “Book on a chair.” which gives on(book,chair) as its logical representation. In this case the chair as the parent object is differ from the box in previous example in such a way that chair object is made of a complex geometry with uneven surfaces on every angle. Child object (book) has been correctly placed on the relative point of the parent object (chair) in every position but it will be improper positioning for the prepositions of On and Under. This shows that the parent object dimension like chair, can infer a proper placement for a child object on every directions and places except for preposition of On and Under. This is due the uneven surface at the bottom and top part of parent object (chair). Using the dimension figure of chairs object makes its child object (book) being placed right under the chair’s leg. This makes it looks like the chair is standing on the book. It is correct to show the spatial relation but it would make more sense if the book were being placed right under the chair, on the same level the chair’s leg is standing. The same goes to the preposition of On. The book should be placed on seat of the chair not on its top rail. For the issue of uneven surface highlighted above, 3D object annotation is used as a countermeasure. Every 3D object that is used in this study will be annotated of its bottom surface type. The topmost height of object will also be tagged to ensure the placement of child object on the top of object giving a more reasonable look. The annotation will be discussed further in the next section. 3.2.3 Data Structure To visualize the natural language phrase into a 3D scene dynamically, this prototype uses a couple set of predefined data that has been organized into the following: 1. 2. 3.

3D objects Direction Colors

Every 3D object is kept in a folder and they are called by Blender into the 3D space using Python script. Path to this 3D object is predefined together with some other metatags as listed in the Table 3.1 to Table 3.3.

40

R. Abdul Kadir et al. Table 3.1. Data structure for 3D objects Data ObjectName

Data Type String

ObjectPath TopMost

String Double

EmptySpaceBott om

Bool

Description

The name of the object. This name is as what received from the logical representation. Directory path to the respective .blend file. The annotated length on z-axis dimension referring the highest part of the object that a child-object should be placed on. TRUE if the object have an empty space at the bottom part. Something like chair objects structure. Otherwise it is FALSE.

Table 3.2. Data structure for prepositions Data PrepositionName

Data Type String

PrepositionAxis

Char

PrepositionDirect Integer ion

Description The name of the preposition. This name is as what received from the logical representation. Axis that the object is moved on. Either x, y or z. Value consists of -1,1,0 and 2. -1 = moving to the negative value on the axis; 1 = moving to the positive value on the axis; 0 = random direct, toward either to negative or to the positive value of the axis; and 2 = object is not moved.

Table 3.3. Data structure for prepositions Data ColorName

Data Type String

ColorRed

Double

ColorGreen

Double

ColorBlue

Double

Description The name of the color (adjective). This name is as what received from the logical representation. Value color of red to perform RGB color to the object. Value color of green to perform RGB color to the object. Value color of blue to perform RGB color to the object.

In order to let Blender now of what axis and either it is positive or negative value should an object to move on the axis and moving direction for preposition is predefined in a table data. Table shows the data dictionary that holds the collection of predefined preposition metadata. When a preposition contain noun with a modifier (color) the modifier name is queried in the database to get the predefined of its equivalent RGB. This RGB code will then be passed to the Blender material and construct a new material with color and apply it to the respective 3D object. Table shows the data structure for colors.


41

4 Experimental Results

Figure 3.1

Figure 3.2

Figure 3.3

From the list of experimental data, Figure 3.1 until Figure 3.3 is some of the depictions we have got. From observation on the 3D scene depictions, below are the things that we have noticed. 1. All of the objects have been placed according to their spatial relation except for the prepositions under. The child object was displayed right towards the –Z axis under the coordinate of (0,0,0). 2. Using the tree object as the parent would cause the child to be place far a bit the tree. 3. Improper visualization for placement with the preposition of on and under. For the phrase of Box on chair, though the box is placed on the seat of the chair, the box eats partially of the backrest of the chair. The same goes to the phrase of Box under the chair, where the legs of the chairs disappeared inside the box.

5 Discussion For issue number 1 above, the child object was place under the coordinate (0,0,0) downwards on axis-z because of the reasons below. 1. When visualizing two objects of parent-child relations, the parent will be brought onto the 3D space first. 2. The parent object’s local coordinate will always be place at the centre coordinate, which is (0,0,0). This makes the parent object being place right on the coordinate of (0,0,0) already. 3. Placement of child object always refers to the position point of parent object’s local. 4. Preposition of under will definitely moving the child object in decremented coordinate on axis-z (moving downwards). Tweak of this issue would be putting the visualization procedure of parent – child relational objects into an exception. The child object will be brought into the 3D space first Instead of the parent object. In this case the child object is taking the parents role because being an object of under other object it always has to be on the coordinate of (0,0,0).

42


Issue of number 2 happens because the child use the parent object’s dimension-x or dimension-y to get away from parent object on the axis-x or axis-y as how its spatial relation is. Hence, the bigger of the parent objects dimension-x or dimension-y the farther the child object will get away. To avoid this problem, the calculation that the child object needs, would be as the in following steps; 1. Take half of the parent object’s dimension-x or dimension-y. 2. Take half of the child object’s dimension-x or dimension-y. 3. Sum these two figures and use this figure to move the child objects on the axis-x or axis-y. The reason of taking half of both objects is that the starting point that child object refer to move away from is the parent object local coordinate. Every 3D object in this prototype their local coordinate is set at their bottom centre. Hence combining the half dimension-x or dimension-y of each object would position them around together in a reasonable distance. Issue like in number 3 would occur when the child object has a bigger dimension-x or dimension-y than the parent object. The countermeasures for such an issue would be; 1. Before visualizing the child object, compare the child object dimension-x and dimension-y with those of parent object. If child object turns out to be larger than of the parent object, scale the child object’s dimensions down (resize the object), or 2. By adding a few more annotation to the parent object. This annotation tags the dimension-x and dimension-y of certain height of the objects. These dimensions will provide the only space that a child object can fit in. For instance, tags the dimension-x and dimension-y at the height of the seat part of the chair. Then before visualizing the child objects compare the child objects dimensions with the parent objects dimensions that were tagged as explained above. If the child’s dimensions are larger scale it down until the child’s dimension get lesser than those tagged dimensions of parent object. Using the solution in number 1 above would not always work since some parent object would have a shrink geometrical shape on certain heights. Just like human body, the shoulder part usually broader than the waist. Blender will return an objects’ dimensions like in bounding box dimension regardless the object have even surface or not. The countermeasure number 1 can be done right away through Blender Python command script without involve additional data from SQLite. Unlike the countermeasure number 2, the process involves a predefined data tagged dimensions as has been explained above. However the countermeasure number 2 would consistently give a reasonable proper visualization of object placement.

6 Conclusion and Future Work This study has demonstrated the 3D scene visualization using semantic simplification description based on logical representation using the technique proposed by [1] as well as the work by [17]. From the semantic simplification’s description, parent child relationship that helps in satisfying the 3D visualization dependencies of two


43

objects, represents their spatial relation. Object constraint like the uneven geometrical surface issue was taken into account and it is solved by object annotation. Annotation was also done on the prepositioning to determine the direction of object moving and the affect axis. Other annotation subject was color. A natural language vocabulary of color was represented by a combination of numerical value of red, green and blue. This annotation approach is adaption of method employed by [3] and from their work in graphic visualization using visual semantic parameterization. The work by [2] utilizes XML as their formatted data structure tool while in this study the concept is applied by utilizing SQLite database engine. In this study we found that the positioning child object using on and under by manipulating the parent objects dimension would give improper visualization when the parent object consist of uneven space. Annotation the 3D objects by tagging the height for child object placement and tagging the surface geometrical bottom side of the object would solve the problem. However there is another issue to take into account concerning the proper visualization. When a child object's dimensions are larger than the parent object's dimensions a particular height some part of the parent object might be invisible covered by the surfaces of child object. The best example was as in the discussion of the Box on the chair and the Box under the chair. Further annotation would be needed to handle this issue as elaborated in the discussion previously. From a recent reading concerning the uneven surface constraint, [18] utilized physic collision and voxelization in their work that involves positioning 3D objects to comply the spatial relations. Through collision over the voxelized part, the objects are partitioned into 9 spatial regions subset. The collision detection handling on the respective region would visualize objects’ spatial region properly. Using this approach in conjunction with the study in 3D scene visualization using logical representation text based would be definitely one of the main future works for consideration. Other future works that could be possibly a complement to this study would be extending the visualization on object’s texture, comparative adjective in terms of size and color intensity. This work can also be extended in visualizing 3D scene with animation from a verb phrase.

References 1. Rabiah, A.K., Sembok, T.M.T., Halimah, B.Z.: Generating 3D Visual Expression Using Semantic Simplification Description Based on Logical. Work, 58–65 (2009) 2. Mehdi, Q.H., Gough, N.E.: From Visual Semantic Parameterization to Graphic Visualization, pp. 1–6 (2005) 3. Clay, S.R.: Put: Language- Based Interactive Manipulation of Objects. IEEE Computer Graphics and Applications (1996) 4. Boberg, R.: Generating Line Drawings from Abstract Scene Descriptions. Master’s thesis, Dept. of Elec. Eng. MIT, Cambridge, MA (1972) 5. Simmons, R.: The clownsmicroworld. In: Proceedings of TINLAP, pp. 17–19 (1998) 6. Kahn, K.: Creation of Computer Animation from Story Descriptions. Ph.D. thesis, MIT, AI Lab, Cambridge, MA (1979)

44


7. Adorni, G., Di Manzo, M., Giunchiglia, F.: Natural language driven image generation. In: COLING, pp. 495–500 (1984) 8. Hanser, E., Kevitt, P.M., Lunney, T., Condell, J.: SceneMaker: Automatic Visualisation of Screenplays. Communication (2009) 9. Hanser, E., Kevitt, P.M., Lunney, T., Condell, J.: SceneMaker: Intelligent Multimodal Visualisation of Natural Language Scripts. Structure, 144–153 (2010) 10. Rowe, N.: The Visualization and Animation of Algorithmically Generated 3D Models Using Open Source Software. In: 2009 First International Conference on Advances in Multimedia, pp. 44–49 (July 2009) 11. Dupuy, S., Egges, A., Legendre, V., Nugues, P.: Generating a 3d simulation of a car accident from a written description in natural language: The carsim system. In: Proceedings of ACL Workshop on Temporal and Spatial Information Processing, pp. 1–8 (2001) 12. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm? arnumber=5206914 13. Ye, P., Baldwin, T.: Towards Automatic Animated Storyboarding. Artificial Intelligence, 578–583 (2008) 14. Ma, M.: Automatic Conversion of Natural Language to 3D Animation. Ph.D. thesis, University of Ulster (2006) 15. Coyne, B., Sproat, R.: WordsEye: An Automatic Text-to-Scene Conversion System. Linguistic Analysis (2001) 16. Zeng, X., Mling, T.: A Review of Scene Visualization Based on Language Descriptions. In: 2009 Sixth International Conference on Computer Graphics, Imaging and Visualization, pp. 429–433 (August 2009), http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm? arnumber=5298772 17. Coyne, B., Rambow, O., Hirschberg, J., Sproat, R.: Frame Semantics in Text-to-Scene Generation. In: Setchi, R., Jordanov, I., Howlett, R.J., Jain, L.C. (eds.) KES 2010 Part IV. LNCS, vol. 6279, pp. 375–384. Springer, Heidelberg (2010) 18. Seversky, L.M.: Real-time Automatic 3D Scene Generation from Natural Language Voice and Text Descriptions. Interface, 61–64 (2006) 19. Ye, P., Baldwin, T.: Towards automatic animated storyboarding. In: Proceedings of the 23rd National Conference on Artificial Intelligence, vol. 1, pp. 578–583 (2008) 20. Blender 3D, http://www.blender3D.org

Character Recognition of License Plate Number Using Convolutional Neural Network Syafeeza Ahmad Radzi1,2 and Mohamed Khalil-Hani2 1

Faculty of Electronics & Computer Engineering Universiti Teknikal Malaysia Melaka (UTeM) 2 VLSI-eCAD Research Laboratory (VeCAD) Faculty of Electrical Engineering Universiti Teknologi Malaysia (UTM) [email protected], [email protected]

Abstract. This paper presents machine-printed character recognition acquired from license plate using convolutional neural network (CNN). CNN is a special type of feed-forward multilayer perceptron trained in supervised mode using a gradient descent Backpropagation learning algorithm that enables automated feature extraction. Common methods usually apply a combination of handcrafted feature extractor and trainable classifier. This may result in sub-optimal result and low accuracy. CNN has proved to achieve state-of-the-art results in such tasks such as optical character recognition, generic objects recognition, real-time face detection and pose estimation, speech recognition, license plate recognition etc. CNN combines three architectural concept namely local receptive field, shared weights and subsampling. The combination of these concepts and optimization method resulted in accuracy around 98%. In this paper, the method implemented to increase the performance of character recognition using CNN is proposed and discussed. Keywords: Character recognition, convolutional neural network, back propagation, license plate recognition, parallel architecture, Visual Informatics.

1 Introduction Convolutional Neural Network (CNN) is a special type of multilayer perceptron; a feed-forward neural network trained in supervised mode using a gradient descent Backpropagation learning algorithm that minimizes a loss function [1, 2]. It is one of the most successive machine learning architectures in computer vision and has achieve state-of-the-art results in such tasks as optical character recognition [1], generic objects recognition, real-time face detection [3] and pose estimation, speech recognition, license plate recognition [4-6] etc. Its strategy is to extract simple features at higher resolution and transform them to complex features at lower resolution. Lower resolution is obtained by applying subsampling at previous layer of such feature maps. The CNN like other artificial neural networks could benefit in speed by implementing parallel architectures. A parallel implementation helps to speed up CNNs simulation,

H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 45–55, 2011. © Springer-Verlag Berlin Heidelberg 2011

46

S.A. Radzi and M. Khalil-Hani

allowing to use more complicated architectures in real-time. It also significantly speeds up training process, which could take days using unparallel architectures. This work is the initial step towards license plate recognition in which the individual characters are manually extracted and recognized using CNNs. The outline of this paper is as follows: Section 2 discusses the theory of CNN. The methodology of the proposed architecture is discussed in Section 3. Section 4 represents the results and discussion, and the final section gives the conclusion of the overall work.

2 Theory The CNN consist layer of neurons and it is optimized for two-dimensional pattern recognition. CNN has three types of layer namely convolutional layer, subsampling layer, and fully connected layer. These layers are arranged in a feed-forward structure as shown in Fig. 1.

Input 1@32x32

C1:feature maps 6@28x28

S2:feature maps 6@14x14

C3:feature maps 16@10x10

S4:feature maps 16@5x5

F5:layer 120

2x2 subsampling 5x5 convolution

F6:layer 10

Full connection 5x5 convolution

2x2 subsampling

Fig. 1. Architecture of CNN [2]

A convolutional layer consists of several two dimensional planes of neurons known as feature maps. Each neuron from a feature map is connected to a neighborhood of neurons from previous layer, resulting to a so called receptive field. Convolution between input feature maps and the respective kernel is computed. These convolution outputs are then summed up together with a trainable bias term which is then passed to a non-linear activation function such as hyperbolic tangent to obtain a new feature value [2]. Weights are shared in convolution matrices, so that large images can be processed with a reduced set of weights. Convolutional layer acts as a feature extractor that extracts salient features of the inputs such as corners, edges, endpoints or nonvisual features in other signals using the concept of local receptive field and shared weights [1, 3]. Shared weights have several advantages as it reduces the number of free parameters to train on, reduce the complexity of the machine, and reducing the gap between test error and training error. An interesting property of convolutional layers is that if the input image is shifted, the feature map output will be shifted by the same amount, but it will be left unchanged otherwise. This property is the basic robustness of CNN for shifts and distortions of the input. One feature map in a layer will have identical weight vectors. A complete convolutional layer is composed of several feature maps (with different weight vectors) so that multiple features can be extracted at each location. Clear view of the statement is represented in Fig. 2. The output yn(l) of a feature map n in a convolutional layer l (as described in [2, 3]) is given by

Character Recognition of License Plate Number Using Convolutional Neural Network     y n (l ) ( x , y ) = φ (l )  wmn (l ) (i, j ) ⋅ y m (l −1) ( x ⋅h (l ) + i , y ⋅ v (l ) + j ) + bn (l )   m∈M ( l ) (i , j )∈K ( l )  n  

(1)

 

where

{

K (l ) = (i, j ) ∈ Ν 2 | 0 ≤ i < k x (l ) ;0 ≤ j < k y ( l )

},

47

k x (l ) and k y (l ) are the width and the height of

(l )

the convolution kernels

wmn of layer l and bn (l ) is the bias of feature map n in layer l.

The set M n (l ) contains the feature maps in the preceding layer l-1 that are connected to feature map n in layer l. The values h (l ) and v (l ) describe the horizontal and vertical step size of the convolution in layer l while φ (l ) is the activation function of layer l.

φ

(a)

(b)

Fig. 2. (a) Spatial convolution [1] (b) CNN convolutional layer [2]

Once a feature has been detected, its exact location becomes less important. Only its approximate position relative to other features is relevant. A simple way to reduce the precision is to reduce the spatial resolution of the feature map. This can be achieved with a so-called subsampling layer which performs local averaging. The subsampling layer reduces the resolution of the image thus reduces the precision of the translation (shift and distortion) effect since the feature maps are sensitive to translation in input [1]. This layer reduces the output of adjacent neuron from previous layer (normally 2x2) by averaging it into a single value. Next, the value is multiplied by a trainable weight (trainable coefficient), adds a bias and then passes the result to a non-linear activation function; such as hyperbolic tangent. The trainable coefficient and bias control the effect of the sigmoid nonlinearity. If the coefficient is small, the unit operates in a quasi-linear mode and the subsampling merely blurs the input. If the coefficient is large, subsampling units can be seen as performing a “noisy OR” or a “noisy AND” function depending on the value of the bias. The illustration can be viewed in Fig. 3. The output yn(l) of a feature map n in a subsampling layer l (as described in [2, 3]) is given by   yn (l ) ( x, y) = φ (l )  wn (l ) ⋅ yn (l −1) (x ⋅ s x + i, y ⋅ s y + j ) + bn (l )    (i , j )∈S ( l )  



(2)

where S (l ) = {(i, j ) ∈ Ν 2 | 0 ≤ i < s x (l ) ;0 ≤ j < s y (l ) }, s x (l ) and s y (l ) define width and height of the subsampling kernel of layer l and bn (l ) is the bias of feature map n in layer l.

48


The value wn (l ) is the weight of feature map n in layer l and φ (l ) is the activation function of layer l.

φ

(a)

(b)

Fig. 3. (a) Example of subsampling process [1] (b) CNN subsampling layer [2]

The final layer is a fully connected layer. In this layer, the neurons from the previous layer are fully connected to every neuron in the current layer. The fully connected layer acts as a normal classifier similar to the layers in traditional Multilayer Perceptron (MLP) networks. The equation of fully connected layer (as described in [2]), is given by  N l −1  y ( l ) ( j ) = φ (l )  y (l −1) (i) ⋅ w(l ) (i, j ) + b (l ) ( j )     i =1 



(3)

where N (l −1) is the number of neurons in the preceding layer l-1, w (l ) (i, j ) is the

weight for connection from neuron i in layer l-1 to neuron j in layer l and b (l ) ( j ) is the bias of neuron j in layer l, and φ (l ) represents the activation function of layer l. The number of layers depends on the application and each neuron in a layer is an input to the following layer. Besides, the current layer only receives input from the preceding layer plus a bias that is usually 1. Each neuron applies weights to each of its inputs and summed up all the weighted inputs. The total weighted value is subject to a non-linear function; such as sigmoid function to limit the neuron’s output to a range of values. Multiple planes are used in each layer so that multiple features can be detected.

3 Methodology This section describes the research methodology to develop the character recognition system. 3.1 Preparing the Training and Test Data Set

The data that are available for training are divided into two different sets: train set and validation set. There should not be any overlapping between these two datasets in order to improve generalization capacity of a neural-network. This technique is called cross validation [3]. The true performance of a network is only revealed when the network is tested with test data to measure how well the network performs on data that were not seen during training. The testing is designed to access the generalization capability of

Character Recognition of License Plate Number Using Convolutional Neural Network

49

the network. Good generalization means that the network performs correctly on data that are similar to, but different from the training data. The training and test data are limited to Malaysian license plate. The alphanumeric characters involved are all the alphabets except for ‘I’, ‘O’ and ‘Z’ as those three characters are not common for Malaysian license plate. The numeric characters are from 0 to 9. Therefore, the total characters are 33. Initially, character recognition is performed to test the algorithm. For that purpose, the characters are extracted from different-angle of license plate and it is binarized, resized from 22x12 pixels to 24x14 pixels and labeled. When padding the input, the feature units are centered on the border and each convolution layer reduces the feature size from n to (n-4)/2. Since the initial input size is 22x12, the nearest value which generates an integer size after two layers of convolution is 24x14.This is to ease the training process. The total numbers of training and test datasets involved are 750 images and 434 images, respectively. CNN can actually deal with raw data but to make it simple, it is suggested to perform simple image processing on train and test data as done in [1]. Fig. 4 shows a sample of test set images.

Fig. 4. Sample of testing set images

3.2 Developing a MATLAB Model

CNN algorithm is implemented by constructing a Matlab program to simulate and evaluate the features extraction process performance upon character image database. Fig. 5 shows the CNN architecture for character recognition. CNN are trained with gradient-based backpropagation method. The purpose of implementing this learning algorithm is to minimize the error function after each training example by adjusting the weights of neural network. The simplest error function used is Mean Square Error. All training patterns along with the expected outputs are fed into the network. Next, the network error (the difference between actual and expected output) is backpropagated through the network and the gradient of the network error is computed with respect to the weights. This gradient is then used to update the weight values according to specific rules such as stochastic, momentum, appropriate learning rate and activation function, etc. The training process will stop until the network is well trained [1]. The supervised training is shown in Fig. 6. Once the network is trained, a test image will be fed to the trained system to perform pattern classification. This statement describes Fig. 7. 3.3 The Proposed Architecture

The architecture shown in Fig. 5 comprises 5 layers, excluding the input, all of which contains trainable weights. The actual character size is 22x12 and padding it to 24x14 to extract the feature in the border of the character image. Layer C1 is a convolutional layer with 6 feature maps. The size of the feature maps is 20x10 pixels. The total number of neurons is 1200 (20x10x6). There are ((5x5+1)x6)=156 trainable weights. The "+1" is for the bias. Each 1200 neurons have

50


Fig. 5. The proposed architecture

Fig. 6. Supervised training diagram

Fig. 7. Testing process diagram

26 connections which make up 31200 total connections from layer C1 to prior layer. At this point, one of the benefits of a convolutional "shared weight" neural network should become clearer: because the weights are shared, even though there are 31200 connections, only 156 weights/parameters are needed to control those connections. As a consequence, only 156 weights need training. In comparison, a traditional "fully connected" neural network would have needed a unique weight for each connection, and would therefore have required training for 31200 different weights. Layer S2 is a subsampling layer with 6 feature maps of size 10x5 pixels. Each unit in each feature map is connected to a 2x2 neighbourhood in the corresponding feature map in C1. The 2x2 receptive fields are non-overlapping, therefore feature maps in S2 have half the number of rows and columns of feature maps in C1. Therefore, there are a total of 10x5x6 = 300 neurons in layer S2, 2x6 = 12 weights, and 300x(2x2+1) = 1500 connections.


51

Layer C3 is a convolutional layer with 16 feature maps. Each unit in each feature maps is connected to several 5x5 neighbourhoods at identical locations in a subset of S2 feature maps. The reason for this is to keep the number of connections with reasonable bounds. There are therefore 6x1x16 = 96 neurons in layer C3, 1516 weights, and 9096 connections. Table 1 shows the connection between S2 and C3. Table 1. Each column indicates which feature map in S2 are combined by the units in a particular feature map of C3 [1]

Layer C4 is a fully-connected layer with 120 units. The choice for this number is due to optimal capacity reached with 120 hidden units for 33 classes. Since it is fullyconnected, each of the 120 neurons in the layer is connected to all 96 neurons in the previous layer. Therefore, there are a total of 120 neurons in layer C5, 120x(96+1) = 11640 weights, and 120x97 = 11640 connections. Layer Output represents the output layer. This layer is a fully-connected layer with 33 units. Since it is fully-connected, each of the 33 neurons in the layer is connected to all 120 neurons in the previous layer. There are therefore 33 neurons in layer F6, 33x(120+1) = 3993 weights, and 33x121 = 3993 connections. Neurons with the value of “+1” corresponds to the “winning” neurons while “-1” corresponds to other neurons. No specific rules were given on deciding the number of layers and feature maps in order to obtain optimum architecture. As long as sufficient information could be extracted for classification task, the architecture is considered accepted. This could be determined by the misclassification rate obtained by the network. However, minimum number of layers and feature maps are much preferred to ease the computation process. Each unit in each feature map is connected to a 5x5 kernel in the input. This kernel size is chosen to be centered on a unit (odd size) in order to have sufficient overlap (around 70%) for not losing information. The kernel size of 3x3 would be too small with only one unit overlap while 7x7 kernel size would be too large that would add the computation complexity [7]. 3.4 Optimizing the CNN Algorithm

There are several actions that could be taken to improve the CNN performance. An extension to backpropagation algorithm could be considered to improve the convergence speed of the algorithm, avoiding local minima, improve the generalization of neural network and finally improves the recognition rate/accuracy. A character database consisting of 33 classes representing alphanumeric characters was created. The system sensitivity to angles and distortions was reduced by taking 10 samples for each class. There are several factors that affect the system performance as discussed below. The parameter value were taken from [1]. Among these techniques, weight decay is not implemented by the writer.

52


Training Mode. There are two principle training modes which determine the way the weights are updated. The first mode is online training (stochastic gradient). This mode represents a single example that is chosen randomly from the training set at each iteration t and the error is calculated before the weights are updated accordingly. Second mode is the offline training (batch training). The whole training example is fed into the network and the accumulated error is calculated before updating the weights. Between these two modes, online learning is much faster than batch learning and results in better generalization solution for large datasets [1, 3]. Learning Rate. Learning rate is used during the weight update of such architecture. This parameter is crucial in determining the successful of convergence and generalization of such neural network. A too small learning rate leads to slow convergence and oppositely leads to divergence. For this architecture, the values of the global learning rate η is adapted from [1]. The value was decreased using the following schedule: 0.0005 for the first two passes; 0.0002 for the next three; 0.0001 for the next three; 0.00005 for the next four; and 0.00001 thereafter. Activation Function. The activation function is pre-conditioned for faster convergence. The squashing function used in this convolutional network is f ( a) = A tanh( Sa) . In [1], A is chosen as 1.7159 and S=2/3. With this choice of parameters, the equalitites f (1) = 1 and f (−1) = −1 are satisfied. Symmetric functions are believed to yield faster convergence, although the learning can become extremely slow if the weights are too small. Second Order Backpropagation. Second order methods has the greatest impact on speeding convergence of the neural network, in that it dramatically reduced the number of epochs needed for convergence of the weights. All second order techniques aims to increase the speed with which backpropagation converges to optimal weights. However, most second order techniques are designed for offline mode which is useless with neural network training. Neural network training works considerably faster with online mode (stochastic gradient) where the parameters are updated after every training sample. Hence, [1] has proposed a stochastic version of the Levenberg-Marquardt algorithm with a diagonal approximation of the Hessian. The Diagonal Hessian (square matrix of second-order partial derivatives of a function) in neural network was shown to be very easy to compute with backpropagation. During the simulation process, the number of subsamples chosen for diagonal hessian estimation has different result as shown in Table 2. Size of Training Set. The size of training set also affects the system performance in term of accuracy [7]. The training set should be as large as possible by adding a new form of distorted data. By this, the network could learn different training patterns that will result to accuracy increment. Momentum. A momentum is added to improve the convergence speed. It controls the previous weight change on the current weight change [1, 3] from oscillating. The following shows the weight update formula.

Δwk (n) = −λ

∂E p ∂wk (n)

+ αΔwk (n − 1)

(4)


53

Where α is the momentum rate (0 ≤ α < 1) and λ is the learning rate. Table 3 represents the misclassification rate for different momentum value. Weight Decay. This is a regularization technique where the term

α

2

2

x

w x is added

to the error function as shown in the equation below: Ep =

1 α || o p − t p || 2 + 2 2

w x

2 x

(5)

This term avoids large weights; reduce the flexibility of the neural network and avoid overfitting the data. By performing gradient descent to this function may lead to the following update formula: ∂E p − λαwx Δw x = − λ (6) ∂wn Where wx refers to all the weights and biases and α refers to a small positive constant. Cross Validation. This technique separates the data into two disjoint parts representing training and validation set. The purpose of this technique is to improve the generalization capacity of a neural network. Theoretically, the error on training and validation set should decrease during the training process. However, at some point the validation error remain constant or even increases. In this case, the increase shows that the neural network might have stop learning the common pattern in train and validation sets, and started to learn noise contained in the training set [3].

4 Results and Discussions This section discusses the results obtained from 4 hours duration of simulation. This network has gone through 30 iteration of simulation. 4.1 Misclassification Rate

In order to analyze the results obtained, the performances of the architecture are measured by misclassification rate. Misclassification rate refers to the number of samples being misrecognized. Thirty iterations through the entire training data were performed for each session. The values of the global learning rate µ was decreased using the following schedule: 0.0005 for the first two passes; 0.0002 for the next three; 0.0001 for the next three; 0.00005 for the next four; and 0.00001 thereafter. There is a vast value of momentums available for CNN. In order to select the optimum value for momentum, several experiments were conducted resulting to the table below. From Table 2, it can be seen that the optimum momentum is approximately 0.5. According to Fig. 8, the produced result is 1.21% of misclassification rate for 434 test data and 750 training data, and for momentum = 0.5. This means the neural network correctly recognized 428 patterns and misrecognized 6 patterns.

54

S.A. Radzi and M. Khalil-Hani Table 2. The reading of misclassification rate according to different momentum values Momentum 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Misclassification rate (percent) Train (percent) 0.9091 0.9091 0.6061 0.3030 0.9091 0.0000 1.2121 0.3030 0.0000 0.3030

Test (percent) 2.4242 4.8485 3.0303 1.8182 3.6370 1.2121 3.0303 1.8182 3.0303 3.0303

As shown from the diagram, the train and test error stabilize after 20 iterations. In certain case, the graph increases after meeting a stable condition. It is assumed that at this point the neural network starts to overtrain the data and generalization decreases. This is due to the fact that neural network have learned the noise specific to that dataset. It may also due to overfitting problem of the classifier and the sample of training set is not large enough to improve the classification accuracy. In order to avoid this situation, “early stopping” technique could be implemented once the graph starts to stabilize [3].

Fig. 8. Misclassification rate graph for momentum=0.5

5 Conclusion Several factors that affect the system performance have been discussed. The system performance could be increased by implementing online mode, choosing appropriate learning rate and activation function, using second order backpropagation (stochastic version of the Levenberg-Marquardt algorithm), expanding the size of training set with different form of distorted images, applying momentum and weight decay, and ultimately implementing cross validation. The number of training and test images used


55

in this research is 750 and 434, respectively with 1.21% misclassification rate or 98.79% accuracy. In comparison to [4-6, 8], the approach is similar except for some, they applied geometrical rule to extract the characters. The numbers of training and test data sets are more than 2000 samples which are much higher than the one proposed in this work. Moreover, the accuracy is more or less the same with this research work. As a conclusion, this research work is better in terms of the reduced number of samples. Acknowledgments. This work is supported by the Ministry of Science, Technology & Innovation of Malaysia (MOSTI) under TECHNOFUND Grant TF0106C313, UTM Vote No. 79900.

References 1. 2.

3. 4.

5.

6.

7.

8.

LeCun, Y., et al.: Gradient-Based Learning Applied to Document Recognition. In: Intelligent Signal Processing, pp. 306–351. IEEE Press (2001) Strigl, D., Kofler, K., Podlipnig, S.: Performance and Scalability of GPU-Based Convolutional Neural Networks. In: 18th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP 2010), pp. 317–324 (2010) Duffner, S.: Face Image Analysis with Convolutional Neural Networks. [Dissertation], [cited Doctoral Thesis / Dissertation; 192 ] (2007) Zhihong, Z., Shaopu, Y., Xinna, M.: Chinese License Plate Recognition Using a Convolutional Neural Network. In: Pacific-Asia Workshop on Computational Intelligence and Industrial Application, PACIIA (2008) Chen, Y.-N., et al.: The Application of a Convolution Neural Network on Face and License Plate Detection. In: 18th International Conference on Pattern Recognition, pp. 552–555 (2006) Han, C.-C., et al.: License Plate Detection and Recognition Using a Dual-Camera Module in a Large Space. In: 41st Annual IEEE International Carnahan Conference on Security Technology, pp. 307–312 (2007) Simard, P.Y., Steinkraus, D., Platt, J.C.: Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. In: Seventh International Conference on Document Analysis and Recognition. Institute of Electrical and Electronics Engineers, Inc. (2003) Johnson, M.: A Unified Architecture for the Detection and Classification of License Plates. In: 2008 10th International Conference on Control, Automation, Robotics and Vision, Hanoi, Vietnam (2008)

Simulation Strategy of Membrane Computing to Characterize the Structure and Non-deterministic Behavior of Biological Systems: A Case Study with Ligand-Receptor Network of Protein TGF-β Muniyandi Ravie Chandren1 and Mohd. Zin Abdullah2 1

School of Computer Science School of Information Technology, Faculty of Information Science and Technology, University Kebangsaan Malaysia, 43600 Bangi, Selangor, Malaysia {ravie,amz}@ftsm.ukm.my 2

Abstract. The processes in biological systems evolve in discrete and nondeterministic ways. Simulation of conventional models such as ordinary differential equations with continuous and deterministic evolution strategy has disregarded those behaviors in biological systems. Membrane computing which has been applied in a nondeterministic and maximally parallel way to capture the structure and behaviors of biological systems could be used to address the limitations in ordinary differential equations. The stochastic simulation strategy based on Gillespie’s algorithm has been used to simulate membrane computing model. This study was carried out to demonstrate the capability of membrane computing model in characterizing the structure and behaviors of biological systems in comparison to the model of ordinary differential equations. The results demonstrated that the simulation of membrane computing model preserves the structure and non-deterministic behaviors of biological systems that ignored in the ordinary differential equations model. Keywords: Membrane computing, stochastic simulation, biological systems.

1 Introduction The biological systems are assumed to be continuous and deterministic in the modeling and simulation with conventional mathematical approaches such as Ordinary Differential Equations (ODE) contrary to its discrete and non-deterministic behaviors. The concentration of chemical substances in the ODE is measured according to the decrease and the increase of the chemical substances by corresponding to kinetic constants of the processes in the system. Although ODE allows the processes to be described in detail, a number of implicit assumptions underlying ODE are no longer applicable on the molecular level [1]. Jong [1] explains the limitations of the continuity and deterministic aspects of ODE at the molecular level. Firstly, the small number of molecules in biological systems such as a few tens of molecules of a transcription factor in the cell nucleus could H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 56–66, 2011. © Springer-Verlag Berlin Heidelberg 2011

Simulation Strategy of Membrane Computing

57

compromise the continuity assumption of ODE. Secondly, deterministic change assumed by using differential operator may be questionable due to fluctuations in the timing of cellular events, such as the delay between start and finish of transcription. This could lead to a situation where two regulatory systems having the same initial conditions may ultimately settle into different states, a phenomenon strengthened by the small numbers of molecules involved [1]. Probability theory [2] has been used to understand stochastic behavior of biological systems, and the mathematical analysis based on this theory provides complete description of properties for simple random systems. Nevertheless, stochastic model of biological systems such as biochemical network showed the limitations of mathematical analysis and such system is described as analytically intractable [3]. In recent years, the ways to capture the role of stochasticity in biological systems ‘insilico’ has been the subject of increasing interest among biologist. With the development of computer system, the simulation of time-evolution of biological system becomes possible. Stochastic simulation is the way to simulate the dynamics of a system by capturing the random phenomena to understand the model and to extract the many realizations from it in order to study them [3]. In recent years, numbers of algorithms have been devised to deal with stochastic character of biological systems [4]. Some of them are, stochastic reaction-diffusion simulation with MesoRD [5]; Stochastic simulation of chemical reactions with spatial resolution and single molecule [6]; and, Monte Carlo simulation methods for biological reaction-diffusion systems [7]. However, such attempts focus more on the general behaviors of biological systems without taking into the structure of the system where the behaviors are taking place. Membrane computing [8] is an area of computer science that conceptualized the ideas and models of computation from the structure and behavior of living cell. It provides a mechanism for biological systems to preserve its structural characters as well as the behaviors of stochastic processes in the systems [9]. This is possible because membrane computing is a computing model with distributed and parallel processing, in which the processes evolve in parallel and non-deterministic or stochastic way in which all evolution rules are simultaneously applied to all the objects. The computation halts to produce output when no rule is applied. The membranes also can be arranged in a hierarchical structure as in a cell, or in an internetworking of different membranes as in a tissue. This conception provides the opportunity of a selective communication between the two regions in the biological systems. The structure and the stochastic behaviors of biology systems modeled with membrane computing has been verified by using stochastic simulation strategy based on Gillespie algorithm [10]. Gillespie algorithm [11] provides a method for the stochastic simulation of systems of bio-chemical reactions and proposed that at each time step, the chemical system is in exactly one state and to directly simulate the time evolution of the system. Basically, the algorithm determines the nature and occurrence of the next reaction, given that the system is in state s at time t. This paper analyzes the simulation of the membrane computing model of ligandreceptor network of protein TGF-β [12] to demonstrate the capability of membrane computing to preserve the structure and behaviors of biological systems.

58

M.R. Chandren and M.Z. Abdullah

1.1 Case Study: Ligand-Receptor Network of TGF-β The Ligand-Receptor Network of TGF-β is two compartments system with five chemical substances and fourteen different reactions. The compartments are Plasma membrane and Endosome as illustrated in Figure 1. Plasma membrane Endosome

Fig. 1. Membrane Structure of Ligand-Receptor Network of TGF-β

In signal transduction some cells secrete TGF-β and also generate receptors for TGF-β. The TGF-β signal transduction pathway plays a central role in tissue homeostasis and morphogenesis. It transduces a variety of extracellular signals into intracellular transcriptional responses that control the excesses of cellular processes such as cell growth, migration, adhesion, apoptosis and differentiation. At the molecular level, complex signal transduction machinery integrates signals from the 42 known ligands of the TGF-β superfamily of proteins. The elements of this machinery incorporate the members of the two main receptor families called type I and type II receptors. Each ligand induces the formation of a receptor complex with type I and type II receptors, which then signal through the channels. The capacity of most ligands to bind several type I and type II receptors lead to a complex ligand-receptor interaction network. TGF-β has particular interest in cancer research. For instance, in epithelial cells, it suppresses cellular growth and its inactivation contributes to tumourigenesis. The versatility of the pathway in eliciting different types of behaviour is perhaps best epitomized by the pervasive, rather paradoxical ability of TGF-β to change its function from suppressor to promoter of growth in epithelial cells during tumour progression. It has been suggested that TGF-β can suppress the growth of cells around the tumour, that it can shut down the immune system locally, and that it can promote angiogenesis. All these paracrine effects would help the growth of the tumour in vivo, where it has to compete with neighbouring cells. In this biological system, ligand concentration is constant to relate the consistent availability of ligand in the system. However there will be a trigger in the system at time steps 2500 to change the ligand concentration from 3x10-5 to 0.01 to reach a steady state. In this case ligand is considered as the event object that triggers the state changes in the system when a specific event or condition is invoked.

2 Membrane Computing Model of Ligand-Receptor Network of TGF-β The structure and behaviors of Ligand-Receptor Network of TGF-β (LRN) are extracted from Villar [12]. The gathered information is represented in membrane computing formalism [13] as follow.


59

LRN = (V , μ , ω P , ω E , RP , RE ) The system contains two compartments that are Plasma membrane (P) and Endosome (E). Therefore its membrane structure can be represented as,

μ = [[

]E ]P

The combination of right and left square brackets refers to a membrane. The structure refers to a compartmentalized membrane system in which membrane E is encapsulated within membrane P. The objects are receptors and complex of ligand and receptors. l is the ligand which is assumed to be always available in P. RI and RII are receptors, and lRIRII is ligand-receptors complex which are manipulated by the rules in P and E. The objects are listed as follow:

V = {l , RI , RII , lRIRII } The combination of objects in a compartment at a time step t will form a multiset in which an object of a multiset can have multiple instances. The multisets at time step t=0 are,

ωP = {l , RI , RII } and ω E = {lRIRII } . k The evolution rule has the form: R j : u[v ] j ⎯ ⎯→ u ' [v ' ] j , where u , v, u

'

, v ' are

multisets and j is the label of compartment. k is a real number representing the kinetic constant. A rule of this form is interpreted as follows: based on the value of k, the multiset u outside membrane j and the multiset v inside membrane j react together to produce u ' outside a and v ' inside j. If there is more than one rule in compartment j, the rules are labeled as R j 1, R j 2,..., R j n to denote each of the n rules. The two common evolution rules in biological systems are: (a) Transformation rule: It is a rewriting rule that occur between objects inside a compartment. For example the rule [X,X]j→[X,Y], means that, when there are 2 objects of X available in compartment j, they can interact with each other to generate an object X and an object Y. (b) Communication rule: It is a transportation of object from a compartment to another compartment. It can be either an object in a compartment moving out or an object moving into a compartment. Y[X] j→ [X,Y] j : In this communication rule, object Y which is outside compartment j is moving into that compartment. [Y,X] j→Y[X]j: In this communication rule, object Y, which is inside compartment j is moving out of the compartment. The rules of ligand receptor network of TGF-β are extracted from the ODE model given by Villar [12]. The ODE model is converted into membrane computing by using rewriting rules [14]. The rules of ligand receptor network of TGF-β are represented in membrane computing as follow. 1) Ligand receptor complex formation: This rule is activated by ka in which l , RI , RII combine to evolve into lRIRII in the plasma membrane. In this rule an

60


event is accommodated in which the concentration of ligand (l) would change from 3x10-5 to 0.01 at time steps 2500.

RP1: (if t >=2500 l = 0.01 else l = 3×10−5 ) >> k [l , RI , RII ]P ⎯⎯→ [lRIRII , l ]P

(1)

a

2) Ligand receptor complex constitutive degradation: This rule is activated by kcd in which lRIRII in the plasma membrane is depleted. k cd [ RP 2 : [lRIRII ]P ⎯⎯→

]P

(2)

3) Ligand independent complex degradation: This rule is activated by klid in which lRIRII in the plasma membrane is depleted. lid [ RP 3 : [lRIRII ]P ⎯k⎯→

]P

(3)

4) Ligand receptor complex internalization: This rule is activated by ki in equation which lRIRII in the plasma membrane is transported into endosome.

RP 4 : lRIRII [

k ]E ⎯⎯→ [lRIRII ]E i

(4)

5) RI synthesis: This rule activated by pRI in which RI is generated in the plasma membrane.

RP 5 : [

p ]P ⎯⎯→ [RI ]P

(5)

RI

6) RI constitutive degradation: This rule is activated by kcd in equation in which RI in the plasma membrane is depleted. k cd [ RP 6 : [RI ]P ⎯⎯→

]P

(6)

7) RI internalization: This rule activated by ki in which RI in the plasma membrane is transported into endosome.

RP 7 : RI [

k ]E ⎯⎯→ [RI ]E i

(7)

8) RI recycling: This rule activated by kr in which RI in the endosome is transported into plasma membrane. kr RE1 : [RI ]E ⎯⎯→ RI [

]E

(8)

9) Ligand Receptor complex recycling: The first rule is activated by kr in which lRIRII in the endosome is transported into plasma membrane. The second rule is activated by the product of kr and α in which the produced lRIRII is transformed into RI , RII in plasma membrane. kr RE 2 : [lRIRII ]E ⎯⎯→ lRIRII [

]E

r .α RP 8 : [lRIRII ]P ⎯k⎯ ⎯ →[RI , RII ]P

(9) (10)


61

10) RII synthesis: This rule activated by pRII in which RII is generated in the plasma membrane.

RP 9 : [

p ]P ⎯⎯→ ⎯ [RII ]P RII

(11)

11) RII constitutive degradation: This rule activated by kcd in which RII in the plasma membrane is depleted. k cd [ RP10 : [RII ]P ⎯⎯→

]P

(12)

12) RII internalization: This rule activated by ki in which RII in the plasma membrane is transported into endosome.

RP11 : RII [

k ]E ⎯⎯→ [RII ]E i

(13)

13) RII recycling: This rule activated by kr in which RII in the endosome is transported into plasma membrane. kr (14) RE 3 : [RII ]E ⎯⎯→ RII [ ] E

3 Simulation of LRN System The LRN system modeled in membrane computing is simulated with the Gillespie algorithm. This experiment intended to analyze how different parameters affect the behavior of the LRN system by using Gillespie simulator [15]. The signaling activity will peak or stop rising at certain steps according to the changes in the parameters. The parameters are extracted from the mathematical model build by Villar [12]. The initial multisets are, ω P = {1130 RI ,1130 RII } and ω E = {40lRIRII} .

Fig. 2. Behavior of LRN system for typical trafficking rates

62


Figure 2 shows the behavior of the model for typical trafficking rates. In this model, internalization rate (ki) represented by k4, k7 and k13, is 1/3; recycling rate (kr ) represented by k8, k9, k10 and k14 is 1/30; constitutive degradation rate (kcd) represented by k2, k6 and k12 is 1/36; ligand-induced degradation rate (klid) represented by k3 is 1/4; complex formation rate (ka) represented by k1 is 0.01; synthesis rate (pRI and pRII) for k5 and k11 is 8 and 4, respectively. The efficiency of recycling of active receptors rate, α is 1. The results show that signalling activity is peaking when the concentration of lRIRII in Endosome is around 500.

Fig. 3. Behavior of LRN system when rate constants for internalization and recycling are decreased

Fig. 4. Behavior of LRN system when rate constants for internalization and recycling are increased


63

When the rate constants for internalization and recycling are decreased to 1/10 and 1/100 respectively, the peak of signaling activity is also decreased as shown in Figure 3. The concentration of lRIRII in Endosome is around 300. Meanwhile, when rate constants for internalization and recycling are increased to 1 and 1/10, respectively, the peak of signaling activity is also increased as shown in Figure 4. The concentration of lRIRII in Endosome is around 600.

Fig. 5. Behavior of LRN system when rate constants for ligand-induced degradation and efficiency of recycling of active receptors are decreased

Fig. 6. Behavior of LRN system when recycling of active receptors is decreased

The simulation in Figure 5 and Figure 6 show the behaviour of the model when the efficiency of recycling of active receptors rate is decreased to 0.5. Figure 5 shows that

64


when ligand-induced degradation rate is decreased to 0.01, the signaling activity is peaking when the concentration of lRIRII in Endosome is around 800. Meanwhile, with the same ligand-induced degradation rate as in Figure 2, the signaling activity is peaking when the concentration of lRIRII in Endosome is around 500. 3.1 Results and Discussions The membrane computing generated Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6 are compared to the ODE generated figures (A), (B), (C), (D) and (E) in Figure 7, respectively. The simulation of membrane computing model of LRN system shows that the general behaviors of the system as indicated in the ODE model [12] in Figure 7 are also obtainable in membrane computing as demonstrated by the peak reached in the membrane computing model which is almost similar to the ODE model.

(A)

(B)

(C)

(D)

(E)

Fig. 7. Simulation of ODE model of TGF-β [12]

Moreover, the membrane computing model also captures the communication of objects between compartment P and compartment E based on the rules (4), (7), (8), (9), (13) and (14). Since the objects in the system are specified according to compartment, and each of these objects evolves in discrete, the number of each type of objects in the specific compartment can be measured at each time step. At the same time, the combinations of behaviors of processes or objects in the system could


65

generate the emergent behavior as shown by the peak of lRIRII in Endosome at certain time step. The simulation results also show that the discrete and nondeterministic characters of membrane computing could not only sustain the general behavior of the system but also the specific stochastic elements at each time step by executing the rules non-deterministically based on the specific behavior of the system. Moreover, the model checking approach carried out by Muniandy et al. [16] reinforced that the properties of LRN system have been preserved in membrane computing model. However, since objects are modeled as a global entity in the ODE model, the concentration of objects in a specific compartment could not be verified. Therefore the ODE model is merely concentrating on the global behavior and disregards the structure and specific behavior in the system. Meanwhile, the membrane computing simulation strategy is taking more time compared to ODE. For instance, the ODE simulation takes around 200 time units to reach a peak [12] and the membrane computing simulation takes around 2500 steps to reach the similar peak with same initial concentration of lRIRII in Endosome. This observation demonstrates that more processing time is needed to select a reaction in the stochastic approach at each time step compared to deterministic approach of ODE. In the simulation of membrane computing model the numbers of simulation steps are almost similar to each of the investigation and it could not be adjusted as in the ODE model due to the limitations of Gillespie Simulator which could not accommodate event object. Event object triggers the state changes in model when a specific event or condition is invoked. In this model, ligand can be considered as the event object in which the event at time step 2500 should change the ligand concentration from 3x10-5 to 0.01 to reach a steady state. However this element is not accommodated in the Gillespie Simulator and due to this limitation the ligand concentration is fixed at 0.01 at all the time.

4 Conclusions The evolution processes in LRN system with two compartments show that the membrane computing model preserves the structure and behaviors of hierarchical systems, and this has been proven with the model checking approach [16] that certified that the properties of the system has been preserved in the model. The advances in memory capacity and processing time in computer system could reduce the simulation time of membrane computing compared to the ODE. At the same time, the Gillespie Simulator should be enhanced to speed up the processing time of membrane computing as well as to manage specific biological behavior such as event object as in the LRN system. Nevertheless, the investigation reinforce that the elements and structure of biological system can be characterized in a better way by using membrane computing model in order to be interpreted straightforwardly by biologist compared to the conventional mathematical models. Acknowledgements. This work supported by Exploratory Research Grant Scheme (ERGS), Ministry of Higher Education (Malaysia). Grant code: ERGS/1/2011/STG/ UKM/03/7.

66


References 1. Jong, H.D.: Modeling and Simulation of Genetic Regulatory Systems: A Literature Review. Journal of Computational Biology 9(1), 67–103 (2002) 2. Jaynes, E.T., Bretthorst, G.L.: Probability Theory: The Logic of Science. Cambridge University Press, London (2003) 3. Wilkinson, D.J.: Stochastic Modeling for Systems Biology. CRC Press, London (2006) 4. Modchang, C., Nadkarni, S., Bartol, T.M., Triampo, W., Sejnowski, T.J., Levine, H., Rappel, W.: A Comparison of Deterministic and Stochastic Simulations of Neuronal Vesicle Release Models. Phys. Biol. 7(2), 26008 (2010) 5. Hattne, J., Fange, D., Elf, J.: Stochastic Reaction-diffusion Simulation with MesoRD. Bioinformatics 21, 2923–2924 (2005) 6. Andrews, S.S., Bray, D.: Stochastic Simulation of Chemical Reactions with Spatial Resolution and Single Molecule Detail. Phys. Biol. 1(3), 137–151 (2004) 7. Kerr, R.A., Bartol, T.M., Kaminsky, B., Dittrich, M., Chang, J.J., Baden, S.B., Sejnowski, T.J., Stiles, J.R.: Fast Monte Carlo Simulation Methods for Biological Reaction-diffusion Systems in Solution and on Surfaces. SIAM J. Sci. Comput. 30, 3126 (2008) 8. Paun, G.: Computing with Membranes. Journal of Computer and System Sciences 61(1), 108–143 (1998) 9. Muniyandi, R., Mohd.Zin, A.: Modeling A Multi Compartments Biological System with Membrane Computing. J. Comput. Sci. 6, 1148–1155 (2010) 10. Muniyandi, R., Mohd.Zin, A.: Experimenting the Simulation Strategy of Membrane Computing with Gillespie Algorithm by Using Two Biological Case Studies. J. Comput. Sci. 6, 525–535 (2010) 11. Gillespie, D.T.: Approximate Accelerated Stochastic Simulation of Chemically Reacting Systems. Journal of Chemical Physics 115(4), 1716–1733 (2001) 12. Villar, J.M., Jansen, R., Sander, C.: Signal Processing in the TGF-β Superfamily LigandReceptor Network. PLoS Comput. Biol. 2(1), e3 (2006) 13. Muniyandi, R., Mohd.Zin, A.: Modeling of Biological Processes by Using Membrane Computing Formalism. Am. J. Applied Sci. 6, 1961–1969 (2009) 14. Bezem, M., Klop, J.W., Vrijer, R.D.: Term rewriting systems. Cambridge University Press, London (2003) 15. Romero-Campero, F., Gheorghe, M., Auld, J.: Multicompartment Gillespie Simulator in C (2004), http://www.dcs.shef.ac.uk/~marian/PSimulatorWeb/ PSystemMF.htm 16. Muniyandi, R., Mohd.Zin, A., Shukor, Z.: Model checking the biological model of membrane computing with probabilistic symbolic model checker by using two biological systems. J. Comput. Sci. 6, 669–678 (2010)

Development of 3D Tawaf Simulation for Hajj Training Application Using Virtual Environment Mohd Shafry Mohd Rahim, Ahmad Zakwan Azizul Fata, Ahmad Hoirul Basori, Arief Salleh Rosman, Tamar Jaya Nizar, and Farah Wahida Mohd Yusof Faculty of Computer Science & Information Systems, Universiti Teknologi Malaysia 81310 UTM Skudai Johor, Malaysia {shafry,aswar,tamar,farahw}@utm.my, {uchiha.hoirul,zakwanfata}@gmail.com

Abstract. 3D application has become very popular among people due to its capability to imitate real world environment and its interactivity to entertain users. This paper proposed an interactive training method for Hajj education in Malaysia by providing user a scene of Tawaf (one of Hajj pilgrimage rituals). Participants are provided with a flexible user control which is connected wirelessly to the system. The Tawaf simulation is created based on crowd technology and in this case we use a new method to control the movements of human character around Kaaba. Based on the user testing feedbacks, users are thrilled with the system, particularly because it is user-friendly and flexible. Keywords: 3D application, interactive training method, crowd, Visual Informatics.

1 Introduction Virtual reality has been used as a fundamental application in most of engineering fields (medical systems like surgery simulation), government defense system (military simulation training, flight simulation), and other engineering fields that related to real life (L.Greitzer et al., 2007, Michael and Chen, 2006, Geiger et al., 2008). There are various approaches to develop 3D simulation especially that utilize game engines and haptic devices, visualization, animation and feedback by sense of touch. In addition, virtual environment offers more interactivity on making simulation, because it has visual interpretation, text information, audio, and feedback based on sense of touch. Recently, games and simulation has evolved into serious game which comprises tasks that teach the users with specific knowledge (L.Greitzer et al., 2007). On the subject of Hajj pilgrimage, previous researcher has made virtual walkthrough over jamarat area to solve the issue of safest path during the annual pilgrimage which is involved million of people (Widyarto and Latiff, 2007). Researchers like Widyarto and Latiff (2007) focus on providing users a tour of jamarat area and show them the safe route to escape when an emergency occurs. H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 67–76, 2011. © Springer-Verlag Berlin Heidelberg 2011

68

M.S.M. Rahim et al.

Millions of pilgrims from around the world gather each year to perform Hajj. Based on a survey, in 2009 the number of pilgrims reached 2.5 million. To simulate this huge population, we have used crowd simulation technique to imitate the real situation inside virtual environment. The crowd simulation of hajj requires efficient algorithm and hardware specification to achieve the best performance. In this paper we focus on one of the annual Hajj pilgrimage ritual which is called Tawaf. Tawaf means moving in circular pattern and refer to Kaaba as the epicentre of the movement. This paper will discuss the experience on developing Tawaf application using crowd simulation technique and the results are tested with real users.

2 Previous Work Collaborating virtual reality, game and simulation give the virtual reality game new paradigm in terms of entertainment aspect. Simulation can stimulate player or user through entertaining, contest, and immediate reaction in digital environment that can generate immersive feeling (L.Greitzer et al., 2007). However, the simulation still faces some problem like how to improve 3D relation in digital environment such as choosing, exploitation, direction-finding (navigation) and managing system (Mocholí et al., 2006). Solving those issues with special integration between human haptic emotion and artificial intelligence will give the simulation closer to reality. The complexity of 3D simulation has been classified into several elements(Capilla and Martinez, 2004): • • • • •

Using special hardware like: haptic devices or head-mounted display devices, etc. Complication of user interface and interaction 3D modeling Collision, physics and deformation Presence of user in virtual environment.

In order to produce good quality of 3D simulation application we can utilize game engines as development tools. Currently game engines have advanced technology that can provide and facilitate better visualization and communication between graphics hardware, sound hardware and application (Marks et al., 2007). The latest 3D application must be able to handle graphics and visualization computational process for instance of loading highly complex primitive, terrain, constructions of trees, stones, cloud, ocean, sky, and loading thousand textures. The other researchers have focused to solve the safety issue of hajj pilgrimage by analyzing the structure of terrain Mina (place near Mecca, see Fig.1) and also the building structure based on GIS method to increase the human safety during the pray session(Al-Kodmany, 2009, Shehata and Koshak, 2006). According to Shehata and Koshak (2006) the hydrology structures need to be investigated to see water drafting path pattern and their impact to the area near mina. It is potential to cause flood or not? In order to solve this issue Shehata and Koshak (2006) has created 3D GIS model to analyze the potential of natural hazard.

Development of 3D Tawaf Simulation for Hajj Training Application

69

Fig. 1. Map of Hajj pilgrimage places

Education of Hajj pilgrimage especially in Malaysia still depends on conventional teaching methods such as hajj training, and prime hajj course (see Figure.2).

Fig. 2. Hajj training and prime hajj course (picture source: http://sabangcenter.blogspot.com/2008/10/22-calon-jamaaf-haji-sabang-ikuti.html)

3 Environment Modeling 3.1 Architecture of System This section will explain the process of designing and creating 3D model of Haram Mosque using mesh lab tools. The Haram Mosque model is created using 3D max and viewed by using meshlab tools. The models of Kaaba and Haram Mosque are bundled in one file and saved into collada model which is supported by horde 3D game engine. The 3D Edu-Hajj system is divided into three main elements: input, main process and output. Wiimote and nunchuk are used as input media for controlling the movement of camera

70

M.S.M. Rahim et al.

inside Haram Mosque environment. The main process of the system is 3D model design, loading and rendering by using horde 3D game engine. Output of the system is displayed using head-mounted display (HMD) (see Figure.3).

Fig. 3. Architecture of virtual hajj

The input of the system is read from wiimote event which is triggered when user presses one of the wiimote buttons. These actions include: move to right, left, forward and backward, rotate the camera and etc. The rendering process is started by interpreting the 3D collada model then arranged all those models into proper position. The first object that rendered in this application is Haram Mosque model. The reason being is Haram Mosque has a huge number of polygons compared to a human character. After rendering process of the mosque, it is continued with human characters rendering. The human characters are placed in certain location inside the mosque and its movement follows a particular pattern. The output of Edu-Hajj system is a mass of crowd model that perform Tawaf ritual with Kaaba as the epicenter. Interactively, Edu-Hajj user will be able to walk around Kaaba as onde of the Tawaf prayers. The output can be viewed on a monitor screen or HMD to get a more realistic experience. In our testing, users are more excited in using HMD rather than normal monitor or screen projector, HMD provide user a better focus.

3.2 Creating Model for Haram Mosque and Kaabah Haram Mosque around Kaaba is divided into three main parts: Kaaba, mosque area and floor. The design of Kaaba, Haram Mosque and floor is started by putting a cube in the centre of the mosque. The texture of the 3D model is based on jpeg file. The design of the whole mosque in wireframe and full scene can be viewed in Figure 4. The model on Figure 4 needs to be exported into collada model in order to be loaded in horde 3D game engine. The conversion into collada model is utilizing 3D max.


71

Fig. 4. Model of Haram Mosque in Full scene rendering

4 Implementation and Testing 4.1 Experimental Setup This study is run on PC Pentium 4 with RAM 2 GB and VGA card 512 MB. The controllers used in this experiment are mouse and wiimote. Wiimote is preferred due to its flexibility to control object thorough wireless connection. The game engine is built from horde 3D and the human model is man.geo that taken from Chicago model of horde 3D. 4.2 Implementation To perform Tawaf, pilgrimages have to wear ihrams which consist of two piece of white cloth for men and any cloth that fulfills Islamic conditions for women. This ihram is yet implemented to our character. In our case, implementation is tested with 80 human characters and the destinations of each character are computed in real time. The human characters walk in circular pattern around Kaaba (black cube in the centre of Haram Mosque) starting from Hajar Aswad (black stone) to perform Tawaf, see Figure 5 for the illustration.

Fig. 5. Tawaf pray started from Hajar Aswad

72

M.S.M. Rahim et al.

The Kaaba has five important elements as follows: 1. 2. 3. 4. 5.

Hajar Aswad Maqam Ibrahim Kabaa doors Hijr Ismail Yamani corner.

Based on the testing records with 80 characters, our application has shown a good result on simulating Tawaf using crowd simulation. The frame rate (FPS) is relatively high and the CPU time is small, which indicates that the device will still able to handle a larger crowd number with high performance. The experiment is simulated using 80, 300 and 500 characters to see the performance of the Edu-Hajj. Subsequently, the system has been tested by users to see how they interact with EduHajj. The growth of character population on Edu-Hajj crowd system will reduce the rendering speed which is shown with low frame rate and large CPU times that needed to render the object. The complete comparison can be seen in Figure 6, Table 1, and Table 2.

Fig. 6. Performance evaluation of E-Hajj on various numbers of characters


73

Table 1. Result of Performance Test in FPS Period (second)

80 characters (FPS) in millisecond ms

500 characters (FPS) in millisecond ms 4.3937

1

24.0462

300 characters (FPS) in millisecond ms 9.6915

2

23.5271

9.9591

4.4042

3

23.2154

10.0231

4.4292

4

23.4321

9.6583

4.4269

5

27.9471

9.7491

4.4198

6

23.0999

9.8261

4.4528

7

23.1895

9.7902

4.4487

8

23.2122

9.3862

4.2859

9

24.0571

9.7311

4.3203

10

23.2693

9.3778

4.2863

Table 2. Result of Performance Test in CPU Time 300 characters (CPU Time) in millisecond ms

500 characters (CPU Time) in millisecond ms

Period (second) in millisecond ms 1

80 characters (CPU Time) in millisecond ms 42.5855

107.1105

227.0720

2

43.2354

106.1046

225.7757

3

43.0867

105.6572

223.8707

4

42.5274

108.7314

226.7975

5

42.8334

104.3172

226.8674

6

41.7027

105.5220

224.0514

7

45.0591

109.9290

227.1216

8

38.3828

109.1557

232.6574

9

43.3686

109.6607

234.2742

10

43.9918

108.1883

233.4143

5 Result and Discussion In this section we will analyze the Edu-Hajj testing result based on empirical testing and user testing to evaluate the performance and usability of Edu-Hajj. The first subsection will discuss the performance of Edu-Hajj when it was tested using a large number of characters.

74

M.S.M. Rahim et al.

The Edu-Hajj has made characters walk circularly around Kaaba by calculating four main destination points for seven times during Tawaf. This is to be in consistent with the rule of Tawaf which encourages user to walk circularly around Kaaba seven times for one Tawaf pray. The four destination points are illustrated in Figure 7.

Fig. 7. Walking pattern

The following formula is generated pattern to calculate the circulation computational complexity of crowd simulation in Edu-Hajj.

Cindex = index × 4 × 7

(1)

Based on Table 1 and Table 2, it has shown the character growth will reduce the performance significantly and it also made CPU takes long time to render the object. 500 characters only has FPS 4.3937 with CPU times 234.2742. It’s very slow, due the complexity of Cindex is very huge.

C500 = 500 × 4 × 7 =14000

(2)

C300 = 300 × 4 × 7 =8400

(3)

Cindex 500 characters have reduced the performance by half compared to Cindex for 300 characters. We are still working to solve this issue which will be carried out as our future work.

6 Future Work In the future, we are planning to enhance the sensory experience of a crowded situation by placing a haptic device on a user’s body. By the movements of a haptic device, user can feel the simulation and the pressure level depends on the number of characters in the system. Moreover, future research will also focus on increasing the rendering speed using In-Situ approach. Due to the current system limitation, the system only allows up 500 models for an accurate simulation. Therefore, by improving the rendering speed, theoretically, a larger number of models can be simulated. This will enhance the


75

simulation accuracy and also create a more realistic experience for the users, as the actual hajj pilgrimage normally draws millions of pilgrims from across the world . The future research will be based on hypothesis agents, where all old computers will be used. Each old computer central processing units (CPU) will be given a data in parallel, so that all data can be processed simultaneously. This will speed up the rendering process, as compared to using a single computer which is less efficient and severely slowing down the rendering process. When the rendering process is too slow, the simulation pattern become less natural and lagging behind the user’s movements. Consequently, the level of user acceptance will be low and the result of the simulation is inaccurate.

7 Conclusions Hajj pilgrimage is the largest people mass movement occurs annually. Therefore to dynamically simulate this scenario through a virtual environment is not an easy task. Each hajj rituals has its own rules and procedures, for instance, in performing Tawaf, there is a specific movement pattern and a number of rotations that must be achieved. The Edu-Hajj is a prototype, which is implemented according to these hajj rules. The current system is able to simulate the Tawaf method and its crowd, to a certain precision. Of course, there is room for improvement and our algorithm needs to be revised in the future. Nevertheless, the testing has shown a promising result on representing the crowd during hajj. Furthermore, during the walkthrough, users seem very excited about Edu-Hajj’s special feature, where they are able to control the camera perspective using the wiimote freely. Acknowledgement. The authors would like to thank Ministry of Science, Technology and Innovation Malaysia and Universiti Teknologi Malaysia (UTM) for their financial support and also CFIRST as main research centre for Edu-Hajj Project.

References 1. 2.

3.

4.

5. 6.

Al-Kodmany, K.: Planning for the Hajj: Political Power, Pragmatism, and Participatory GIS. Journal of Urban Technology (2009), doi:10.1080/10630730903090289 Capilla, R., Martinez, M.: Software Architectures for Designing Virtual Reality Applications. In: Oquendo, F., Warboys, B.C., Morrison, R. (eds.) EWSA 2004. LNCS, vol. 3047, pp. 135–147. Springer, Heidelberg (2004) Geiger, C., Fritze, R., Lehmann, A., Stocklein, J.: HYUI- A Visual Framework for prototyping Hybrid User Interfaces. In: Proceedings of the Second International Conference on Tangible Anda Embedded Interaction (TEI 2008). ACM, Bonn (2008) Greitzer, L., Kuchar, F., Huston, K.: Cognitive Science Implications for Enhancing Training Effectiveness in a Serious Gaming Context. Journal of Educational Resources in Computing-ACM 7(3) (2007) Marks, S., Windsor, J., Wunsche, B.: Evaluation of Game Engines for Simulated Surgical Training. In: GRAPHITE. ACM (2007) Micheal, D., Chen, S.: Serious Game, game that educate, train, and Inform. Thomson Johnson Technology (2006)

76 7.

8.

9.

M.S.M. Rahim et al. Mocholi, J.A., Esteve, J.M.J., Jaen, R.A., Xech, P.L.: An Emotional Path Finding Mechanism for Augmented Reality Applications. In: Harper, R., Rauterberg, M., Combetto, M. (eds.) ICEC 2006. LNCS, vol. 4161, pp. 13–24. Springer, Heidelberg (2006) Shehata, A.M., Koshak, N.A.: Using 3D GIS to Assess Environmental Hazards in Built Environments (A Case Study: Mina). The Journal of Al-Azhar University (2006); ISSN:1110-640 Widyarto, S., Latiff, M.S.A.: The use of virtual tours for cognitive preparation of visitors: a case study for VHE. Emerald Journal of Facilities 25, 271–285 (2007)

A Grammar-Based Process Modeling and Simulation Methodology for Supply Chain Management Mohsen Mohammadi1,*, Muriati Bt. Mukhtar1, and Hamid Reza Peikari2 1

Information Science and Technology, Universiti Kebangsaan Malaysia, Bangi, 43600, Malaysia [email protected] 2 Graduate School of Business, Universiti Kebangsaan Malaysia, Bangi, 43600, Malaysia

Abstract. In order to respond to customer demands, supply chains must be rapidly reconfigured and evaluated. This has given rise to supply chain simulation as an important tool to aid in the evaluation of supply chain configurations. Part of the reconfigurations involved designing and redesigning business processes. Hence, business process simulation is an integral part of supply chain simulation. Supply chain simulation models are usually large-scale complex models. It is thus usual for the simulation experts to get overwhelmed with the simulation process itself. This paper intends to propose a methodology, combining two approaches (i.e. grammar-based business process modeling and simulation) to facilitate process thinking (reengineering). This methodology bridges the gap between business process modeling and simulation by providing the grammatical approach. This paper presents a novel approach to design business processes based on grammar-based process modeling, eventdriven process chains (EPC) and discrete event simulation. This paper illustrates that the grammar-based process modeling approach is applicable to the simulation of dynamic systems such as supply chains as well as representing detailed description of the processes and events. More detailed and advanced analysis, and discussions will be reported in future papers. Keywords: Event-driven process chain, grammar-based modeling, process simulation, supply chain, Business Process Reengineering.

1 Introduction The uncertainties in business environment change business models in supply chain. Therefore, it is needed to create/change business process in the shortest possible time to respond to the uncertainties of environment. In the design of new business processes, simulation facilitates the validation of the processes to ensure that they will work as designed. Simulation is used to evaluate supply chain performance and is an integral part of the decision making process in supply chain management. Business process-based simulation provides a precise, visual method to analyze and compare the performance before and after business process engineering [1]. For a *

Corresponding author.

H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 77–86, 2011. © Springer-Verlag Berlin Heidelberg 2011

78

M. Mohammadi, M.B. Mukhtar, and H.R. Peikari

manufacturing supply chain, manufacturing patterns such as manufacturing process model, which focus on process modeling, process management, process analysis, and process reengineering [2] will form an important input in the supply chain simulation model. As such, in order to develop supply chain simulation models, the modeler is required to use several approaches and tools in order to capture the important facets of the supply chain. Tools such as the Business Process Modeling (BPM), Business Process Simulation (BPS) and discrete event simulation are usually used. However, these tools have limitations and they need to be complemented by other tools. For instance, Business Process Modeling (BPM) has a static structured approach to business process improvement. It provides a holistic approach on how the business operates by documenting the business processes. However, BPM does not provide any information about the dynamics of the business and how the changes in business process with the minimum risk is possible. To provide a dynamic approach to the business and to consider the impacts of change in such dynamics without risk, the concept of Business Process Simulation (BPS) was developed by researchers and practitioners [3]. However, the diagrams in Business Process Modeling and Business Process Simulation are not sufficient to describe the details of a complex system and its processes [4]. There are numerous studies about Business Process Reengineering (BPR), in which each have used different methodologies. For example, [5] have introduced a methodology based on five business process modeling languages to integrate business process. [6] have proposed a general framework to assess a reengineering project, from its strategic planning stage to post-implementation phases [7] on the other hand have proposed a conceptual model to demonstrate the links between organizational re-structuring and behavioral changes to reengineer business processes. All of such studies have focused on business processes while in order to link between process modeling and process simulation; it is needed to have descriptive details of events and processes. Therefore, it is needed to apply a Grammar-Based Process Modeling. Applying such an approach, can improve the simulation of complex systems such as supply chains. Moreover, using a grammar-based process modeling approach is essential to prevent repeating the sub-processes and events of the system. The objective of this paper is to apply a Grammar-Based Process Modeling and Simulation methodology for supply chain management to illustrate more details of the system and its processes. [8] developed a simulation model by proposing an analytical framework for integrated logistic chain and this paper by using Arena 10 applies a simulation model based on numerical example in [8]; however it does not show the key performance indicators (KPI) and optimization of the model but show the possibility of using the information in simulation and analyzing them to have the results.

2 Literature Review The grammatical approach to design and redesign supply chain processes has received some attention. It is because this approach can balance and integrate the use of the other process modeling approaches [9]. MIT process handbook recommends the use of grammar-based approach to model the organizational and system processes [10]. Using this approach, each of the processes of a supply chain can be further described, modeled and modularized in more detailed activities and processes. The representation of the supply chain processes using this approach is easy to maintain because the processes can be described in hierarchical chunks [11].Many process

A Grammar-Based Process Modeling and Simulation Methodology

79

simulation tools can be integrated into the manufacturing process, and can be used to reengineer and optimize the manufacturing process model which lead to the improvement of many performance indices. Process models can be developed and supported by different systems such as workflow management system which support the decision making of managers in the business environment [2]. Business Process Simulation is an important part of the design and redesign of business processes. It provides quantitative reports about the impacts of a process design on process performance. Moreover, it provides quantitative evidences to decide the best process design. Simulation of business processes overlap with the simulation of other event systems [12].Simulation has enabled companies to deal with the impacts of the changes in their sites’ parameters without any unwanted or uncalculated effects on their operations and outputs. Business processes and companies knowledge can become much more transparent and understandable with an explicit, standard documentation [13]. The EPC [14] has been developed in 1992 by the team of August-Wilhelm Scheer at the Saarbrücken University in a joint research project with SAP AG. for modelling business processes with the goal to be easily understood and used by business people. The basic idea of EPC is that events trigger functions, or the executed functions cause events. The activities of a business process can be model by functions, and events are created by processing functions[15]. Three different tools applicable for BPS are process execution, process modeling and simulation. Jansen-Vullers and Netjes [12] developed a framework to find strengths and weaknesses of simulation modeling and process modeling tools. As shown in the Table 1, these tools for each of the evaluation criteria ranges from very bad (– –) and bad (-) to neutral (+/–), good (+) and very good (++). Table 1. Modeling capabilities [4] Feature Ease of model building Formal semantics/verif. Workflow patterns Level of details Animation Scenarios

ARIS + – – ++ + +

FLOWer + –– + ++ –– ––

Arena + +/– + ++ ++ +

3 Model Description As shown in Figure 1, a supply chain model was considered with the following assumptions: Three suppliers: Supplier 1, Supplier 2, Supplier 3 Two Semi-finished manufacturers: Semi-finished manufacturer 1, Semi-finished manufacturer 2 :One Final manufacturer: Final manufacturer One Distributor: DC Two Warehouse: Warehouse1, Warehouse2 Three Retailers: Retailer1, Retailer2, Retailer3 and Ultimate Customers.

80


Fig. 1. The supply chain model

3.1 The Methodology An efficient business process is a critical factor for any business to succeed. Business process modeling facilitates the understanding and analysis of the business process, and business process simulation is an effective technique to evaluate and diagnose business processes in an organization [16]. Due to different issues and shortcomings in time, costs, and users’ skills, the advanced simulation and optimization techniques have not been widely applied [16].The methodology to design business process reengineering modeling is presented in the following flowchart(Figure 2)

Fig. 2. The Methodology of Business Process Reengineering Modeling


81

3.2 The Workflow The workflow of the supply chain model illustrated in Figure 3 has been considered without the retailers.

Fig. 3. Workflow diagram

3.3 Grammar-Based Modeling in Supply Chain Process Numerous manufacturing process modeling approaches can be used in process view such as workflow modeling approach and event-driven process chain. Many process simulation tools can be integrated to the manufacturing process model. This optimizes and reengineers the manufacturing process model which lead to the improvement of many performance indices [2]. The element for representing the Supply Chain Process with structural form can be defined [4]: Process: Pi= Event: Ei= Resource: Resource(Ei)={Rj,j=1,2,...•Rj Rj X,X=document or material} Rj= Dependency: Dependency (Di) = 3.4 Modeling the Process In order to represent the workflow of the model, we used Event-Driven Process (EPC) modeling language using ARIS Express 2.3. the events, processes and junctions in this software include and, or, xor. The model has been illustrated in Figure 4. The graph in a model is more intuitive than grammatical model. The above models can represent dynamic process with their representation logic and graphical symbols.

82


Fig. 4. Event-driven process chain diagram

3.5 Description of the Events and the Process There are a lot of constraints and relationships among the processes of a supply chain, which can be represented by the grammatical approach [9]. In a model, grammatical model is less intuitive than graph; but diagram cannot represent all information related to processes of a supply chain [9]. Then, in order to ensure the legibility of the model, the basic information of the processes of a supply chain can be represented by diagram as an accessorial tool. Table 2 illustrates the basic information of the grammatical model and provides detailed description of the events and the processes of the model. Table 2. Description of the events and the processes Process Name P1 Produce material 1

Agent Supplier1

P2

Produce material 2

Supplier2

P3

Produce material 3

Supplier3

P4

Produce semi-finished semi-finished product 1 product manufacturer 1 Produce semi-finished semi-finished product 2 product manufacturer 2 Produce final product final product manufacturer

P5

P6

Input Order bill of material 1 Order bill of material 2 Order bill of material 3 material 1 & material 2

Output Material 1

Pre Post D1 D2

Material 2

D3

D4

Material 3

D5

D6

Produce form manufacturer 1

D7

D9

material 2 & Produce form material 3 manufacturer 2

D8

D10

material 1,2,3

D11 D12

Produce final manufacturer


83

Table 2. (continued) P7 P8 P9 Event E1 E2,E3 E4 ….. D D1 …. D7 D14 ….

Distribution

Distribution Final productPrepare for D13 center warehouses Storing in Warehouse Warehouse 1 Prepare for Store in D15 1 warehouse 1 warhouse1 Storing in Warehouse Warehouse 2 Prepare for Store in warehouse D16 2 warehouse 2 2 Description Resource Pre. Receiving bill of order material 1 form Null semi-finished product manufacturer 1 Are similar to E1 … Material 1 is produce by manufacturer 1 <material 1 , received by D2 manufacturer 1 > Input E1

Output P1

Rule Flow

E4,E5 P7

P4 E10,E11

and xor

D14 D17 D18 Post. D1

D7

3.6 Arena Model and Simulation Results Based on the supply chain model shown in Figure 5 and information from Table 2 with a more details, a sub-model was simulated in Arena 10 by eliminating the retailers. As shown in the proposed sub-model illustrated in Figure 2, each of the

Fig. 5. The simulated sub-model

84


manufacturers can procure their materials from the suppliers 1, 2 and 3 and after producing the final product, send them to the distribution centers. Distribution centers decide about the distribution of the products to the warehouses based on a 50% 2-way by chance decision type. The simulation was designed run the replication for 500 times within 720 days before giving the results. Some of the results of the simulation related to entities (time), and process have been illustrated in Figure 6,7. Figures illustrate the process related to suppliers, manufacturers, distributors and warehouses. Our objective is not to show the key performance indicators (KPI) and optimization of the model but to show the possibility of using the information in simulation and analyzing them to have the results.

Fig. 6. The process result-time per entity

Fig. 7. The accumulated time result

4 Conclusion Business process modeling (BPM) provides management with a static structured approach to business improvement, providing a means of documenting the business processes, and business-process simulation (BPS) allows management to study the dynamics of the business. In the context of business process change BPS can be helpful. Thus BPM and BPS help to facilitate process thinking.This paper provides a


85

preliminary analysis only and more findings with details will be presented in future reports. However, despite the preliminary analysis provided in this paper, it has numerous findings and contributions. First, it illustrated that grammar-based process modeling can be used with business process simulation to improve the transparency of the system description and its process details. Moreover, since the modeling and simulating the processes include modeling the inputs, outputs, resources, workflow, etc, using a grammar-based modeling can bring about more flexibility of the supply chain dynamic simulation. Besides, a dynamic environment like supply chain includes its specific mechanisms and rules. Because these rules can be represented grammatically, therefore, in order to simulate a rule-based supply chain, the approach presented in this paper can be useful.

References 1. Cui, L., Chai, Y., Liu, Y.: Business Process based Simulation: A Powerful tool for demand analysis of business process reengineering and information system implementation. In: Proceedings of the 2008 Winter Simulation Conference (2008) 2. Feng, Y.: Research on Manufacturing Process Modeling and Simulation Based on Resource Optimization (2010), http://www.china-papers.com/?p=127962 (last accessed April 25, 2011 ) 3. Barber, K.D., Dewhurts, F.W., Burns, R.L.D.H., Rogers, J.B.B.: Business-Process Modeling, and simulation for manufacturing management: A Practical Way Forward. Business Process Management Journal 9, 527–542 (2003) 4. Yan, J., Qin, F., Yang, J., An, L., You, D.: A Grammar-Based Modeling of Supply Chain Process, Service Operations and Logistics, and Informatics. In: SOLI 2006, pp. 364–369 (2006) 5. Muthu, S., Whitman, L., Hossein Cheraghi, S.: Business Process Reengineering: A Consolidated Methodology. In: The 4th Annual International Conference on Industrial Engineering Theory, Applications and Practice, San Antonio, Texas, USA, November 1720 (1999) 6. Dodaro, G. L., Crowley, B. P.: BUSINESS Process Reengineering Assessment Guide, Accounting and Information Management Division (1997), http://www.gao.gov (August 16, 2011), http://www.gao.gov/special.pubs/bprag/bprag.pdf 7. Gunasekaran, A., Kobu, B.: Modelling and analysis of business process reengineering. International Journal of Production Research 40(11), 2521–2546 (2002) 8. Dong, M., Frank Chen, F.: Performance modeling and analysis of integrated logistic chains: An analytic framework. European Journal of Operational Research, 83–98 (2005) 9. Lee, J., Pentland, B.T.: Exploring the Process Space: A Grammatical Approach to Process Design/ Redesign (2002), http:Hccs.mit.edu/papers/pdfwp2l5.pdf (last accessed April 1, 2011) 10. Malone, T.W., Crowston, K., Herman, G.A.: Organizing Business Knowledge: The MIT Process Handbook. MIT Press, Cambridge (2003) 11. Zeng, Y., Pentland, B., Kwon, P.: A Grammar-Based Approach to Capturing and Managing Processes: An Industrial Case Study. Transactions of NAMRIIISME 31, 555– 562 (2003) 12. Jansen-Vullers, M.-H., Netjes, M.: Business Process Simulation - A Tool Survey, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.87. 8291&rep=rep1&type=pdf (last accessed May 2, 2011)

86


13. Ponis, S.T., Hadzilias, E.A., Panayiotou, N.A.: Business Process Engineering with ARIS Simulation – A Case Study of a Call Centre Operation Assessment. WSEAS Transactions on Business and Economics 1(4), 338–343 (2004) 14. Scheer, A.-W.: ARIS – Business Process Modeling. Springer, Heidelberg (1999) 15. Kruczynski, K.: Business Process Modelling in the context of SOA – An empiric study of the acceptance between EPC and BPMN. In: Proceedings of the 5th International Conference on Information Technology and Applications, Cairns, Queensland, Australia, June 23-26 (2008) 16. Ren, C., Wang, W., Dong, J., Ding, H., Shao, B.: Towards A Flexible Business Process Modelling and simulation Environment. In: Proceedings of the 2008 Winter Simulation Conference (2008)

A Parallel Coordinates Visualization for the Uncapaciated Examination Timetabling Problem J. Joshua Thomas1, Ahamad Tajudin Khader2, and Bahari Belaton2 1

School of Computer Sciences, University Sains Malaysia 2 KDU College (PG) Sdn Bhd [email protected], {tajudin,[email protected]}

Abstract. The paper describes a new approach, based on visualization, to the uncapacitated examination timetabling problem. This approach involves with a naturally inspired evolutionary technique with parallel coordinates that facilitates exploration of data. By explicitly representing the visualization parameter space, the interface provides an overview of rendering options and enables user to easily explore different processes namely preprocessing, during the processing and post processing of examination schedules. The paper discusses the performance of this method on the Carter set of benchmark problems. This data set is comprised of real-world timetabling problems. Keywords: Visual Informatics, visualization, time-tabling, parallel coordinates.

1 Introduction Parallel coordinates is a well-established visualization technique first proposed by Inselberg [1, 2]. It is a scalable framework in the sense that the increase of the dimensions corresponds to the addition of extra axes. The visual clutter caused by an excessive overlaying of polylines limits the effectiveness of parallel coordinates in visualizing a dense data set. However parallel coordinates visualization method has adopted to visualize the timetabling input matrices and output matrices in its multivariable visualization. Visual clustering, filtering, and axis reordering are the common methods to reduce clutters in parallel coordinates. Some filtering techniques remove part of data in a pre-processing. Axis reordering computes the best axis order with minimal clutters, but it is very likely that the best axis order still leads to an unsatisfactory and cluttered display. This paper introduces the ParExaViz system, a software tool developed as a research aid in the field of examination scheduling. ParExaViz contains a set of visualizations designed to be used for analyzing the complexity of examination scheduling problems, helping to solve these problems and visualizing the heuristic strategies. Before discussing the motivations behind building ParExaViz, it is useful to look at the reasons for investigating this topic, present the university examination scheduling problem. The examination scheduling problem is interesting as the subject of these visualization as the basic structure of the examination scheduling problem can be clearly defined; however when scaled up to real life situations the problems become H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 87–98, 2011. © Springer-Verlag Berlin Heidelberg 2011

88

J.J. Thomas, A.T. Khader, and B. Belaton

highly complex making valid solutions are difficult. It is believed that the visualization presented here can be used to help successfully reduce this complexity. The goal of visualization software is to facilitate preprocess, during the process, and the post processing stages from raw data to feasible examination timetable generation in an intuitive way. Information visualization focuses on continuous data, often from acquisition process or computational simulation.Visualization tools often fail to reflect this fact both in functionality and in their user interfaces, which typically focus on graphics and programming concepts rather than more meaningful to end-user, although advances have been made, interfaces that support exploration and insight have substantial room for improvement, at least for assignment data in examination timetabling. We describe a prototype interface that utilizes parallel coordinates, processing of examination data exploration. The outline of this paper is as follows: Section 2 summarizes examination timetabling and the data. In Section 3, describes the design in more detail. Implementation stages are in Section 4 and conclusion and future work in Section 5.

2 Examination Timetabling Examination scheduling problem involves assigning resources, rooms and times, to exam events whilst ensuring that the number of violations, such as clashes where a student is scheduled to one or more exams simultaneously, is minimized. Other violations considered here are: room and time overflows, where exams are scheduled in locations without the required capacity or duration; and, consecutive violations, where a student has one or more exams in immediate succession. Throughout this discussion ‘cluster’ refers to a set of exams that do not share any common students (conflict) and can therefore all be scheduled in the same timeslot without causing any clash violations. There are many variants of examination timetabling problems due to the fact that each educational institution has their own rules, regulations and expectations resulting with various constraints. There has been much research into finding solutions to the uncapacitated examination timetabling problem and various techniques such as tabu search, simulated annealing, constraint programming, evolutionary algorithms, ant colonization, variations of the great deluge algorithm and the variable neighborhood search algorithm have been investigated for this purpose [4]. 2.1 Model of Examination Timetabling The model is based on constraints, as in the literature. Each of the constraints is formulated as integer programming. Examination timetabling problem (ETP) is known to be highly constrained and complex assignment problem. To reduce the complexity, it is normally divided into two sub-problems. ─ Exam-timeslot assignment ─ Exam-room assignment.

A Parallel Coordinates Visualization

89

In exam – timeslot assignment exams are scheduled into a fixed number of timeslots, and in exam-room assignment exams are assigned to rooms. Hence, the ETP, and assignment is ordered 3-tuple (a,b,c), where as , , and has the straightforward general interpretation. For example, exam a starts at timeslot b in room c. However, we focus on “exam-timeslot” assignment. There matrices are required to show the interrelationships between these set of variables. There are two types of matrices, input matrices and output matrices. The input matrices are the matrices where the values are known earlier (timetabling data), and have been allocated or pre-assigned. The output matrices are the assignment matrices where the values need to be determined by solving ETP. 2.2 Hard Constraints and Soft Constraints Hard constraints must be satisfied in order to produce a feasible timetable. Any timetable which fails to satisfy all these constraints is considered infeasible. Soft constraints are those which are desirable to be satisfied, but which in general cannot all since soft constraints are numerous and varied on the needs of the individual problem. 2.2.1 Exam-Timeslot Assignment Hard Constraints ─ Exam- clashing: Every exam must be assigned to exactly one timeslot of the timetable. ─ Student-clashing: No student should be scheduled in two different places at once. ─ Group together: Certain exams with questions in common must be grouped and scheduled together in the same timeslot. 2.2.2 Exam –Timeslot Assignment Soft Constraints ─ Timeslot-capacity: the total number of students in all exams in the same timeslot must be less than the total capacity for that timeslot. ─ Exam related (1) : no student should have two exams in adjacent timeslots on the same day. ─ Exam related (2) : no student should have two exams in s timeslots or in adjacent days. Quality measures of an examination timetabling are derived from soft constraints, most frequently from student restrictions. If several quality measures are used the objective function is a linear combination of these measures. 2.3 Dataset The Parallel coordinate’s visualization has tested with the set of Carter benchmarks listed in Table 1 below. Note that for some of the data sets more than one version exists, thus the version is also indicated, e.g. car-s-91I. The density of the conflict matrix is an estimate of the difficulty of the problem and is the ratio of the number of examinations involved in clashes and the total number of examinations. The hard

90


constraint for the set of benchmarks is that there are no clashes, i.e. each student must not be scheduled to sit more than one exam in a given timeslot. Thus, the hard constraint cost for this problem is the number of clashes. A feasible timetable is one in which the hard constraint cost is zero, i.e. there are no clashes. Table 1. Carter benchmark dataset Data car-f-92 I car-s-91 I ear-f-83 I

hec-s-92 I kfu-s-93

lse-f-91

rye-s-93 sta-f-83 I

Institution Carleton University,Ottawa Carleton University, Ottawa Earl Haig Collegiate Institute, Toronto Ecole des Hautes Etudes Commerciales, Montreal King Fahd University of Petroleum and Minerals, Dharan London School of Economics Ryerson University, Toronto St Andrew’s Junior High School, Toronto

Periods 32

No. of Exams 543

No. of Students 18419

35

682

16925

24

190

1125

18

81

2823

20

461

5349

18

381

2726

23

486

11483

13

611 139

tre-s-92

Trent University, Peterborough, Ontario

23

261

4360

uta-s-92 I

Faculty of Arts and Sciences, University of Toronto Faculty of Engineering, University of Toronto

35

622

21266

10

184

2749

21

181

941

ute-s-92

yor-f-83 I

York Mills Collegiate Institute, Toronto

Table 1. Illustrates the carter’s version –I of dataset. For each data starting from the first row (car-f-92-I, 32, 543, and 18419) has shown the institution periods, number of exams and the total number of students enrolled. The subsequent rows are with different institutions provided as benchmarking data for the examination timetabling problem.


91

3 Parallel Coordinates the Design 3.1 Analyze the Problem There are few questions before we start on explaining. “How hard is this problem to solve?” This may be in relation to other similar problems, or instances of the same problem. Mathematics helps answer the first query by providing a means of classifying different problems, for example the examination scheduling problem is known to be NP-Hard. In contrast the visualization discussed here attempts to provide a visual solution to the second question: how hard are examinations scheduling problems to solve relative to each other? Further how do different parts of the same problem, for example different departments, relate to each other in terms of difficulty to schedule? For using this visualization for analyzing the complexity of an examination scheduling problem instance is to see which parts of the problem look like they are going to be the hardest to solve. Many heuristics involve trying to solve the hardest parts of problems first, for example by initially scheduling exams with the highest number of clashes. Although this is relatively easy for an automated system it is not so easy for a human scheduler, especially when the criteria for selecting exams become more complicated. Another feature of the problem, which does appear to be a good indicator of the difficulty of solving an examination scheduling problem, is the number of potential clashes (or exam intersections) that exist. This value is not dependent on any algorithm, and is used here as a measurement of the problems complexity. We designed our interface to support the visualization tasks described by [3]. Note that we apply this visualization seeking mantra to processing stage representation rather than data representation. Our design organizes all visualization processes together in one space so that users can quickly gain an overview of the possibilities. The effects of changing parameter settings can be studied by making changes with a simple mouse action and viewing consecutive results. 3.2 Relationship to Parallel Coordinates Our interface is based on the concept of parallel coordinates. Many of the axes do not display one-dimensional variables. For example, a transfer function is a complex concept consisting of variables such as data intensities, opacities, and sometimes intensity derivatives [10]. Furthermore, axes can be merged to produce even more complex nodes. Sorting and interpolating nodes therefore requires slightly more complex techniques than ordinary parallel coordinates [5]. In addition, a major strength of parallel coordinates is their ability to make high-level trends in data sets (by drawing many lines).We display lines without colors and relate one other variables. There is no history view displayed. Our proposed interface is novel in that it is the first interface that would allow such analysis on the examination timetabling problem. In Figure 1 has depicted the general approach of the parallel coordinates versus Cartesian coordinate graphs.

92


Fig. 1. Multiple points in Cartesian coordinate graph versus Parallel Coordinate graph

4 Implementation Visualization systems combine visualization and interaction techniques, ranging from simple slider and drag & drop interfaces to more advanced techniques such as Fish Eye Views. These techniques have in common that the visualizations transform data from an internal representation into an easily recognizable visual representation. This visual representation has contained more information about the data preprocessing or problem instance as input matrices as population the traditional genetic algorithm has the engine to do the assignment or schedule and these during the process has been visualized. This visualization has exposed the clashes between exams to student or student group, exams to timeslot etc. we want to use the medium of visualization in combination with interaction to capture additional facts about the data and the problem, which are hard or costly to capture and communicate otherwise. The constraints are usually not represented by the formal problem description (see unspecied portion of the problem in figure 2). Because of the complexity the problem and the optimization goals, the formal speciation of the problem is usually simplified, and captures only the necessary requirements but not the sufficient ones. Our interface is based on the concept of parallel coordinates, it has substantial differences from ordinary parallel coordinates displays [9]. Many of the axes do not display onedimensional variables. The next sections briefly explain the processes involved in visualization. Our proposed interface is novel in the sense it is the first interface that would allow such analysis.


93

Fig. 2. Visual complexity of the examination timetabling problem

There are five set of variables that should be taken into account in a university examination timetabling. ─ Exam: the exam to be timetabled the domain of this variable E is the set of all 1….. has student exams to be scheduled. For each exam ei , enrollments, department, length and type. ─ Timeslot: the time used by the exam represent as domain of this variable T, is the set of all timeslots in the timetable preparation. For each timeslot tj, 1….. , has start and finish times and type. ─ Room: the room where the exam to be held the variable domain R, is the set of all rooms available for exams. ri 1….. has department, size and type. ─ Student: enrolled for the exams, the domain is S, is the set of all students enrolled for the exams. For each si 1….. has department and a list of exams. ─ Department: represents a faculty with the domain variable D, is the set of all 1….. has a list of departments at an institution. Each department di rooms, students, and the list of invigilators. 4.1 Preprocessing Data preprocessing can significantly improve the quality of data mining results, no matter whether an algorithmic or a visual approach is considered. There are four different aspects of data preprocessing in literature data cleaning, data integration, data transformation and data reduction [7]. However, these techniques are not quite suitable for the examination timetabling problem. In the pre-processing step, we first detect clusters in the given data set. Numerous clustering algorithms can be applied to

94


detect clusters. The clustering process provides an abstraction of data without filtering out potentially important information such as outliers. Throughout the experiments in this paper, the K-means clustering algorithm [8] is adopted.

Fig. 3. Clustering algorithms applied on the car-s-91-I dataset

Although the algorithm is not guaranteed to return a global optimum, it is extremely fast and has been widely adopted in many research areas. In figure 3 a clustering method and the brushing system are used accordingly on the variables. Each column represents the constraint parameters of the problem. The starting column has the car-s-91-I input matrices, second and subsequent columns are timeslot, exams, students and rooms. Here the focus is only between exam and timeslot assignment and the clustered data variables are highlighted with simple brushing. We adopted the input matrices a pool of data as the primary input for the next processing phase. 4.2 During the Processing During the processing must have an algorithmic engine to produce the schedules or assignment according to the given input matrices. For example, figure 3 is one of the input timetabling data these files are of two types pertaining course data and student data. The course file has the parameter of course, enrollment and with the student file the parameters are courses in ascending order. As we mentioned in section 2.2 the assignment is between the exam and timeslot. This process has been assisted with visual metaphors to witness the clashes between exam, timeslot and student. Parallel coordinates visualization is more suitable if the cluster clashes between, exam with student and student group, timeslot with room, time with exam and student etc. the overview of the pattern as explained. Every student must do one exam. There are group exams and individual exams. The maximum number of students for group exam is 3 per group. The group size is predetermined by the student enrollment or with administrator. An exam can 35 many time slots. The maximum of 10 exams per timeslot, and with the minimum of 5 exams per slot will be considered and each student must enrolled 3 subjects.


95

4.2.1 Data Set Structure and Constraints Given No of exams= 543, No of students = 18413. For ease of representation, we will adopt numerical data representation instead of binary in each locus in a chromosome. There are 2 types of data sets (two pools of chromosomes): 4.2.2 1st Pool of Phenotypes 1st pool contains chromosomes in the form of the 1st data set, which is a matrix of numbers, X [i, j] where X is a (N x 3) matrix (where N= total number of students available); ─ For j=1, each locus represents a exam_id ─ For j=2, each locus represents a timeslot, 1 ≤ i ≤N, where N= total number of timeslot available ─ For j=3, each locus represents no of students allocated to the exams Table 2. Input matrices of examination to timeslot and the student group

E1 E2 E3 E4 E5 : E543

T1 T1 T1 T2 T2 : T32

2 3 1 3 3 : :

In table 2, The size of X[i,j] is limited to X[1,1]……X[N,3], where N= total number of exams to be schedule ─ ─ ─ ─

X[i,1] = Tm , where 1≤ Tm ≤ 35 X[i,2] = En, where 1≤ Pn ≤ N, where N= total number of exams X[i,3] = G, where 1≤ G ≤ 3, where G = groups of students For every similar value of Tm , 5 ≤ ∑ of each corresponding value of G ≤ 10

4.2.3 2nd Pool of Phenotypes 2nd pool contains chromosomes in the form of the 2nd data set, which is a matrix of numbers, Y [i,j] where Y is a (16925 x 5) matrix; ─ For j=1, each locus represents a student id, ─ For j=2, each locus represents the 1st course of the courseid1, Cn Cn = X[i,2], where

1≤ C ≤ N, where N= total number of (courses) available

─ For j=3, each locus represents the 2nd choice of the student (a courseid2),) Cn Cn = X[i,2], where

1≤ C ≤ N, where N= total number of (course2) available

─ For j=4, each locus represents the 3rd choice of the student (courseid3,Cn Cn = X[i,2], where

1≤ C ≤ N, where N= total number of (course3)) available

96

J.J. Thomas, A.T. Khader, and B. Belaton Table 3. Input Matrices of students with courses

S1 S2 : S16925

Ca Ce : Cj

Cb Cf : Ck

Cc Cg : Cj

Cd Ch Cm

In table 3, The size of X[i,j] is limited to Y[1,1]……Y[16925,5] ─ Y[i,1] = Sc , where 1≤ Sc ≤ 16925 Cn = X[i,2], where 1≤ C ≤ N, where N= total number of ─ Y[i,2] = Cn, courses available Cn = X[i,2], where 1≤C ≤ N, where N= total number of ─ Y[i,3] = Cn, courses available ─ For every value of Y[i,2], Y[i,3], Y[i,4], Y[i,5], every value of Cn must not appear more than 3 times in the whole matrix (excluding values in Y[i,1]). Figure 4, explains the graphically drawn simple understanding of the parallel coordinates visualization of the examination timetabling parameters in during the processing. The visualization handles five variables which supports the individual or group types of data values. In the timeslot column and exam column few clashes are identified. For example, look at the time, exam and student column there are clashes identified. In the time column, T1 and T2 are the clash identified upon exam column. The exam column has identified the clashes of E1, E2 and E3 that clashes with individual or group of students.

Fig. 4. Parallel coordinates representation of examination timetabling parameters

4.3 Post Processing Figure 5 shows a screen shot of our system. The two squares on the left side shows conflict summaries of different time tables. The upper square contains the parent population (size 16). The lower square contains the result of a crossover operation of


97

two time tables of the parent population. The bar on the right side contains a list of the best time tables seen so far. The large area in the middle shows the processing of the examination schedules. On right hand side the column boxes shows the conflicts. In here, it shows the time versus exams and exam versus subject. The post processing phase deals with that resultant timetable with minimum clashes. The slider symbolizes the examination threshold from the range of 0 to 2184.

Fig. 5. ParExaViz: A tool to solve examination timetabling problem

5 Conclusion and Future Work We presented a parallel coordinates user interface that facilitates examination time tabling processes data exploration. All parameters are clearly organized in terms of vertical lines in the visualization and visible so that users can see and remember what options are available and what settings generated based on the clashes. A history bar will be implemented as in future work however; previous and next buttons are useful in the interface to allow users to easily backtrack to previous states and quickly scroll to see which options have been previously tried. However, visual clutter in parallel coordinates significantly limits its usage for visualizing large data sets. Inspired by some previous graph based visual analytic tools, our approaches do not alter the data in the parallel coordinates. Instead, they are developed on top of the parallel coordinates to visualize the core examination timetabling processes.

98


References [1] [2] [3] [4]

[5]

[6]

[7]

[8] [9] [10]

Parallel Coordinates: VISUAL Multidimensional Geometry and its Applications, Textbook, 554 pages. Springer, New York (2009) Inselberg, A., Dimsdale, B.: Parallel Coordinates: A Tool forVisualizing Multidimensional Geometry. In: Proc. IEEE Visualization, pp. 361–378 (1990) Shneiderman, B.: The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations. In: Proc. IEEE Symp. Visual Languages, pp. 336–343 (1996) Qu, R., Burke, E.K., McCollum, B., Merlot, L.T.G., Lee, S.Y.: A Survey of Search Methodologies and Automated System Development for Examination Timetabling. Journal of Scheduling 12(1), 55–89 (2009) Ericson, D., Johansson, J., Cooper, M.: Visual data analysis using tracked statistical measures within parallel coordinate representations. In: CMV 2005: Proceedings of the Coordinated and Multiple Views in Exploratory Visualization, Washington, DC, USA, pp. 42–53. IEEE Computer Society, Los Alamitos (2005) Joshua Thomas, J., Khader, A.T., Belaton, B.: Exploration of Rough Sets Analysis in Real-World Examination Timetabling Problem Instances. In: Tan, Y., Shi, Y., Chai, Y., Wang, G. (eds.) ICSI 2011, Part II. LNCS, vol. 6729, pp. 173–182. Springer, Heidelberg (2011) Ward, M.O.: Xmdvtool: integrating multiple methods for visualizing multivariate data. In: VIS 1994: Proceedings of the Conference on Visualization 1994, pp. 326–333. IEEE Computer Society Press, Los Alamitos (1994) Hartigan, J.A., Wong, M.A.: A K-means clustering algorithm. Applied Statistics 28, 100–108 (1979) Zhou, H., Yuan, X., Qu, H., Cui, W., Chen, B.: Visual clustering in parallel coordinates. Computer Graphics Forum 27 (2008) McDonnell, K., Mueller, K.: Illustrative parallel coordinates. Computer Graphics Forum (Special Issue Eurovis 2008) 27, 1027–1031 (2008)

A Modified Edge-Based Region Growing Segmentation of Geometric Objects Nursuriati Jamil1, Hazwani Che Soh1, Tengku Mohd Tengku Sembok2, and Zainab Abu Bakar1 1

Faculty of Computer and Mathematical Sciences, Universiti Teknologi MARA, 40450, Shah Alam, Selangor, Malaysia [email protected], [email protected], [email protected] 2 Faculty of Technology and Information Sciences, Universiti Kebangsaan Malaysia, 43600, Bangi, Selangor, Malaysia [email protected]

Abstract. Region growing and edge detection are two popular and common techniques used for image segmentation. Region growing is preferred over edge detection methods because it is more robust against low contrast problems and effectively addresses the connectivity issues faced by edge detectors. Edgebased techniques, on the other hand, can significantly reduce useless information while preserving the important structural properties in an image. Recent studies have shown that combining region growing and edge methods for segmentation will produce much better results. This paper proposed using edge information to automatically select seed pixels and guide the process of region growing in segmenting geometric objects from an image. The geometric objects are songket motifs from songket patterns. Songket motifs are the main elements that decorate songket pattern. The beauty of songket lies in the elaborate design of the patterns and combination of motifs that are intricately woven on the cloth. After experimenting on thirty songket pattern images, the proposed method achieved promising extraction of the songket motifs. Keywords: Image segmentation, region growing, edge detection, geometric objects, songket, Visual Informatics.

1 Introduction Region growing and edge detection are two popular and common techniques used for image segmentation. While region growing segment images by grouping similar pixels into regions based on seeds, edge techniques segregate pixels into different regions based on gradient discontinuities. Region growing works by appending neighbouring pixels of starting seed pixel to form a region based on predefined criteria [1] [2]. The region grows by appending to each seed pixel those neighbouring pixels that have the similar properties to the seed point such as specific range of gray level values or colour. Region growing is preferred over edge detection methods because it is more robust against low contrast problems and effectively addresses the H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 99–112, 2011. © Springer-Verlag Berlin Heidelberg 2011

100

N. Jamil et al.

connectivity issues faced by edge detectors [3]. Region growing, however, requires prior knowledge about the image. This is necessary to select the seed points [3] and the threshold values [4]. In recent years, numerous work on automating the seed points and threshold value selections have been attempted [5] [6] [7][8][9][10] [11] and the results are encouraging. Over and under segmentation are also common problems in region growing [12]. Edge detectors work by segmenting the image based on the abrupt changes in gray levels. These abrupt changes or discontinuities among pixels are detected by finding the maximum and minimum in the first derivative of the image and by searching the zero crossings in the second derivative of the image. The derivative nature of edge detectors, however, makes them extremely sensitive to image noise level. Noise will affectively affect the performance of edge detectors, thus significant noise removal is required to remove the noisy edges. Edge-based techniques are also subject to global variation in an image, but do reliably identify strong boundaries [13]. Edge methods are popularly employed for image segmentation [14][15][16][17][18][19][20] because it can significantly reduce useless information while preserving the important structural properties in an image. 1.1 Integration of Region Growing and Edge Detection Both region growing and edge-based segmentation have their own strengths and limitations [17]. Pavlidis and Liow [21] stated that combining region growing and edge methods for segmentation will produce much better results. [22] suggested that in region growing, region joining decisions may be based not only on pixel or neighborhood similarity but also on already-extracted edges and completion of these edges. Once two adjacent regions that are candidates for merging are found, strength of the boundary should be examined first. If the boundary is of sufficient strength, they should be kept separated. If the boundary between them is weak, they may be merged. Since 1980s, substantial researches on integrating region growing and edgedetection techniques for image segmentation have been conducted. Grinaker [23] used edge detection in areas around the borders of regions found by region-based segmentation. Fua and Hanson [24] used edge–based techniques and area-based geometric constraints to pick out the best segmentation from a series of region-based segmented images. Xiahon et al. [25] combined region growing, edge detection and a new edge preserving smoothing algorithm to avoid characteristic segmentation errors which occur when using region growing or edge detection separately. In [26], Yi-Wei and Jung-Hua proposed an edge detector criterion to determine whether the region growing process should continue or terminates when a region grew to such extent that it touched an edge pixel. Al-Hujazi and Sood [27] performed segmentation by using residual analysis to detect edges, and then a region-growing technique was used to obtain the final segmented image. The integration of region growing and edge detection techniques has been increasingly popular in recent studies. A hybrid method using region growing, edge detection and mathematical morphology was introduced by [28] in 2002. The threshold controlling the process of region growing was automatically determined by a fuzzy technique. Experimental results indicated that the proposed method for 3D segmentation of MRI brain images provided much better results than the traditional

A Modified Edge-Based Region Growing Segmentation of Geometric Objects

101

method using a single technique in the segmentation of a human brain MRI data set. In [13], region growing initially was used to produce over segmented image. The image was then modified using edge strength, edge smoothness, edge straightness and edge continuity. A reasonable segmentation results were produced by [29] using region growing and region merging of color images. Then, multi-threshold concept to generate local entropies was used for reasonable edge detection. In [30], adaptive thresholding based on the gradients and variances along and inside of the boundary curve was proposed to overcome the difficulty of manual threshold selection and sensitivity to noise. In clinical MRI image segmentation this method produced very satisfactorily results. The region distribution and global edge information are further employed to identify region with texture characterization to obtain segmentation results.

2 Data Collections In this paper, edge information is used to automatically select seed pixels and guide the process of region growing. Experiments are conducted on thirty images of scanned songket patterns. An example of a songket pattern is illustrated in Figure 1. Songket is an exquisite hand woven cloth of the Malays made by hand weaving silver, gold and silk threads on a handloom. The beauty of songket lies in the elaborate design of the patterns and combination of motifs that are intricately woven on the cloth. Motif is the main element of designing songket pattern and preservation of these motifs in digitized form is essential for cultural heritage purposes. Thus, the main purpose of this study is to automatically segment these motifs from the songket pattern and archived them.

Fig. 1. A songket pattern

The basis of this study is inspired by Fan et al. in [30]. Fan et al. employed edge detection techniques to define two kinds of seed pixels, one called edge region seeds (hot seeds) and the other homogenous seeds (cold seeds). Region growing is then applied based on the fact that the gray levels of the hot seeds are lower than the pixels not far away from the edge region in the hot object and the gray levels of the cold seeds are higher than the pixels not far away from the edge region of the cold background. The region growing process is highly computational and timeconsuming. Thus, this study proposed employing morphological operations to

102

N. Jamil et al.

complete the segmentation process. Further discussions of each step are presented in the next section.

3 Methodology In this section, each step of the experiments is detail out. Figure 2 demonstrates the flowchart of the proposed methodology. Start Songket pattern Pre-processing Edge detection Edge region detection Modified Sobel detection Threshold to edge region

Seeds definition Morphological operations Segmented Motifs End Fig. 2. Flowchart of the proposed methodology

3.1 Pre-processing In pre-processing stage, the scanned songket pattern color image is cropped and converted to grayscale. Noises captured during image acquisition process are then reduced by smoothing the image using a 9 x 9 weighted average filter as shown in


103

Figure 3. Smoothing process also help enlarged sharp edges which later will be identified as edge region. An example of image smoothing is shown in Figure 4. 1 2 3 4 6 4 3 2 1

2 3 4 6 10 6 4 3 2

3 4 6 10 12 10 6 4 3

4 6 10 12 14 12 10 6 4

6 10 12 14 16 14 12 10 6

4 6 10 12 14 12 10 6 4

3 4 6 10 12 10 6 4 3

2 3 4 6 10 6 4 3 2

1 2 2 4 6 4 2 2 1

Fig. 3. Weighted average filter used for noise removal

(a)

(b)

Fig. 4. (a) Grayscale image. (b) Smoothed image.

3.2 Edge Detection Canny edge detector [32] provides the most possible compromise between noise reduction and edge localization and is observed to give good results. Its edge detection algorithm utilizes these steps: 1.

Noise smoothing - First of all, the image is smoothed by Gaussian convolution using a Gaussian approximation: ,

1 2

(1)

where σ is the standard deviation of the distribution. 2. Edge enhancement – The gradient magnitude and orientation are computed using finite-difference approximations for the partial derivatives. This can be done by applying a simple 2-D first derivative operator such as the Sobel vertical (Gx) and horizontal (Gy) filters to the smoothed image to highlight regions of the image with high first spatial derivatives.

104

N. Jamil et al.

Gx=

-1 -2 -1

0 0 0

1 2 1

Gy=

1 0 -1

2 0 -2

1 0 -1

The gradient magnitude and direction are then computed using Equation 1 and 2, respectively. |

|

|

|

(2)

 3.

4.

(3)

Edge localization - Edges give rise to ridges in the gradient magnitude image. Nonmaxima suppresion is applied to the gradient magnitude to thin the wide ridges around local maxima in gradient magnitude down to edges that are only one pixel wide. Edge linking - To remove noise in the nonmaxima suppressed magnitude image, two threshold values (hysteresis thresholding algorithm) are applied to it. With these threshold values, two thresholded edge images T1[i, j] and T2[i, j] are produced. The image T2 has gaps in the contours but contains fewer false edges. With the double thresholding algorithm, the edges in T2 are linked into contours. When it reaches the end of a contour, the algorithm looks in T1 at the locations of the 8-neighbours for edges that can be linked to the contour. This algorithm continues until the gap has been bridged to an edge in T2. The algorithm performs edge linking as a by-product of thresholding and resolves some of the problems with choosing a single threshold value.

The results of applying Canny edge detection on the smoothed image in Figure 4b is shown in Figure 5.

(a)

(b)

Fig. 5. (a) Smoothed image (b) Canny edge detected image

3.3 Edge Region Detection The edge region is defined as pixels surrounding the single pixel detected by Canny operator (Figure 5b). Four Sobel operators are applied to the smoothed image in


105

Figure 4b to identify all possible pixels that are to be included in the edge region. The four Sobel operators are shown as follows [31]: S1=

2 1 0

1 0 -1

S2=

-2 -1 0

S3=

0 1 2

S4=

0 -1 -2

0 -1 -2

+

1 0 -1

2 0 -2

1 0 -1

=

3 1 -1

3 0 -3

1 -1 -3

-1 0 1

0 1 2

+

-1 0 1

-2 0 2

-1 0 1

=

-3 -1 1

-3 0 3

-1 1 3

-1 0 1

-2 -1 0

+

1 2 1

0 0 0

-1 -2 -1

=

1 3 3

2 1 0

+

-1 -2 -1

1 2 1

=

-1 -3 -3

1 0 -1

0 0 0

-1 0 1

-3 -3 -1

1 0 -1

3 3 1

The general equations of edge region detection are as follows: ,

,

,

, ∑

1. .4

(4)

, 4

(5)

where fn(i,j) are the images obtained from the four Sobel operators and h(i,j) is the final edge region image. Figure 6 illustrates the final edge region image applied on the smoothed image.

(a)

(b)

Fig. 6. (a) Smoothed image. (b) Final edge region image.

After the edge region image is obtained, it is converted to binary using Otsu’s global threshold method. Otsu’s method uses a criterion function as a measure of

106

N. Jamil et al.

statistical separation between classes of foreground and background intensity values. Given these two groups of pixels, the overall or total variance of the gray values in 2 the image can be easily calculated, denoted by σ T . For any given threshold t, variance of the foreground and background pixels can be separately computed to 2 represent the within-class variance, σ W . Finally, the variation of the mean values for each class from the overall mean of all pixels defines a between-class variance, 2 denoted by σ B . An optimal threshold can then be calculated by minimizing the ratio of the between-class variance to the total variance [33]:

η (t ) =

σ B2 σ T2

(6)

The value t that gives the smallest value for η is the best threshold. Figure 7 shows the result of applying Otsu’s threshold algorithm on the edge region image.

(a)

(b)

Fig. 7. (a) Edge region image. (b) Thresholded image.

(a)

(b)

Fig. 8. (a) Binary edge region image. (b) Enhanced edge region image.


107

Thresholding process sometimes produced small erroneous regions considered as noises. These small-sized regions are eliminated by performing median filtering as shown in Figure 8. Only then hot and cold seeds are computed. 3.4 Seeds Definition Seeds are defined into two types, that is hot and cold seeds. Seeds selections are based on the Canny edge image (Figure 5b) and the edge region image (Figure 8b). Hot seeds are pixels in edge region image not far away from foreground objects (i.e. songket motifs) and cold seeds are pixels in edge region image not far away from background. The hotness degree of the pixels in the edge region image is then estimated using the following pseudocode: For each pixel in the grayscale image If the current pixel belongs to the edge image && edge region image Calculate the hotness degree of the pixel. Hotness degree of each pixel is dependent on a threshold value determined by the median of the pixel’s 5x5 neighbours. If the pixel’s gray value is greater than the threshold, hotness degree of the pixel is increased by 1, otherwise the hotness degree is decreased by 1. After the hotness degree image is estimated, each pixel is in the image is defined based on the criteria listed in Table 1. Table 1. Seeds definition criteria Seed category Hot Cold 0(not a seed)

Criteria Hotness degree > 0 and pixel belongs to edge region image Hotness degree < 0 and pixel belongs to edge region image Hotness degree = 0

After the seeds are defined, the seeds image is shown in Figure 9.

Fig. 9. Seeds image

108

N. Jamil et al.

3.5 Morphological Operations The seeds are then grown using morphological reconstruction by connecting the pixels in 8-directions beginning from the holes of the binary image. A hole is a set of background pixels that cannot be reached by filling in the background from the edge of the image. Enhancement of the image is further done by erosion using a square structuring element of size 5 x 5. The final segmented image is as shown in Figure 10.

Fig. 10. Segmented image

4 Results and Discussions Thirty songket patterns are experimented using the proposed method and the results are summarized in Table 2. Seventy percent of the songket patterns are completely correctly segmented into the corresponding motifs. Only a small percentage of the patterns (6 out of 30) have less than 50% correct segmentation results. Songket patterns that have high percentage of correctly segmented motifs are mostly isolated, complete motifs such as shown in Figure 11. Patterns that have partial motifs at the border such as in Figure 12 tend to produce incorrectly segmented motifs. The morphological reconstruction seems to fill up the intricate details of the motifs at the border of the pattern image. Table 2. Summarized results of segmentation Songket Pattern No. Correctly segmented motifs 11, 13, 19, 20, 29, 30, 33, 35, 45, 47, 51, 54, 100% 55, 60, 72, 73, 76, 81, 85, 86, 89 74 90% 75, 98 50% 31, 43, 95 33% 49, 97, 94 10%


109

Fig. 11. Isolated, complete motifs

(a)

(b)

Fig. 12. (a) Songket pattern. (b) Poorly segmented image

Partial Motifs

Another type of problematic patterns is those of chained motifs as in Figure 13. Hot and cold seeds are used to differentiate foreground and background pixels in the edge region. As long as the edge regions are connected, the seeds grew from one motif to another producing incorrectly segmented motif.

Fig. 13. Chained motifs

110

N. Jamil et al.

5 Conclusion This paper reports a method of image segmentation by using edge information as the basis of automated seed selection of region growing. There are two main contributions of this paper: (1) The selection of seed pixels are done automatically based on a Canny edge-detected image and edge region image defined by Sobel operators. (2) Region growing is done using morphological reconstruction methods, thus it is much faster and less computational. Further work need to be done to improve the segmentation result especially for the chained and partial motifs. Other performance measurements such as area overlap, false positive and negative rate should also be taken into consideration as only minimal measurement is used in this paper. As reported in the experiments, seventy percent of the songket patterns achieved total correctly segmented motifs.

References 1. Hojjatoleslami, S.A., Kittler, J.: Region Growing: A New Approach. IEEE Transactions on Image Processing 7(7), 1079–1084 (1998) 2. Gonzales, R.C., Woods, R.E.: Digital Image Processing, 2nd edn. Prentice Hall (2002) 3. Park, J.G., Lee, C.: Skull Stripping Based On Region Growing For Magnetic Resonance Brain Images. NeuroImage 47, 1394–1407 (2009) 4. Adams, R., Bischof, L.: Seeded Region Growing. IEEE Transactions on Pattern Analysis and Machine Intelligence 16(6), 641–646 (1994) 5. Poonguzhali, S., Ravindran, G.: A Complete Automatic Region Growing Method for Segmentation of Masses on Ultrasound Images. In: International Conference on Biomedical and Pharmaceutical Engineering, December 11-14, pp. 88–92 (2006) 6. Ghelich Oghli, M., Fallahi, A., Pooyan, M.: Automatic Region Growing Method using GSmap and Spatial Information on Ultrasound Images. In: 18th Iranian Conference on Electrical Engineering, May 11-13, pp. 35–38 (2010) 7. Yang, J.-H., Liu, J., Zhong, J.-C.: Anisotropic Diffusion with Morphological Reconstruction and Automatic Seeded Region Growing for Color Image Segmentation. In: International Symposium on Information Science and Engineering, December 20-22, vol. 2, pp. 591–595 (2008) 8. Tatanun, C., Ritthipravat, P., Bhongmakapat, T., Tuntiyatorn, L.: Automatic Segmentation of Nasopharyngeal Carcinoma from CT Images: Region Growing Based Technique. In: International Conference on Signal Processing Systems (ICSPS), July 5-7, vol. 2, pp. V2537–V2-541 (2010) 9. Jianping, F., Yau, D.K.Y., Elmagarmid, A.K., Aref, W.G.: Automatic Image Segmentation by Integrating Color-edge Extraction and Seeded Region Growing. IEEE Transactions on Image Processing 10(10), 1454–1466 (2001) 10. Siddique, I., Bajwa, I.S., Naveed, M.S., Choudhary, M.A.: Automatic Functional Brain MR Image Segmentation using Region Growing and Seed Pixel. In: International Conference on Information & Communications Technology, December 10-12, pp. 1–2 (2006) 11. Roslan, R., Jamil, N., Mahmud, R.: Skull Stripping of MRI Brain Images using Mathematical Morphology. In: IEEE EMBS Conference on Biomedical Engineering and Sciences, November 30-December 2, pp. 26–31 (2010)


111

12. Sharma, N., Aggarwal, L.: Automated Medical Image Segmentation Techniques. J. Med. Phys. 35, 3–14 (2010) 13. Chowdhury, M.I., Robinson, J.A.: Improving Image Segmentation Using Edge Information. In: Canadian Conference on Electrical and Computer Engineering, vol. 1, pp. 312–316 (2000) 14. Law, M.W.K., Chung, A.C.S.: Weighted Local Variance-Based Edge Detection and Its Application to Vascular Segmentation in Magnetic Resonance Angiography. IEEE Transactions on Medical Imaging 26(9), 1224–1241 (2007) 15. Bellon, O.R.P., Direne, A.I., Silva, L.: Edge Detection to Guide Range Image Segmentation by Clustering Techniques. In: International Conference on Image Processing, vol. 2, pp. 725–729 (1999) 16. Morris, O., Lee, M., Constantinides, A.: A Unified Method for Segmentation and Edge Detection using Graph Theory. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 11, pp. 2051–2054 (April 1986) 17. Kaganami, H.G., Beiji, Z.: Region-Based Segmentation versus Edge Detection. In: Fifth International Conference Intelligent on Information Hiding and Multimedia Signal Processing, IIH-MSP 2009, September 12-14, pp. 1217–1221 (2009) 18. Bourjandi, M.: Image Segmentation Using Thresholding by Local Fuzzy Entropy-Based Competitive Fuzzy Edge Detection. In: Second International Conference on Computer and Electrical Engineering, December 28-30, vol. 2, pp. 298–301 (2009) 19. Hsiao, Y.-T., Chuang, C.-L., Jiang, J.-A., Chien, C.-C.: A Contour Based Image Segmentation Algorithm using Morphological Edge Detection. In: IEEE International Conference on Systems, Man and Cybernetics, October 10-12, vol. 3, pp. 2962–2967 (2005) 20. Yishao, L., Jiajun, B., Chun, C., Mingli, S.: A novel image segmentation algorithm based on convex polygon edge detection. In: TENCON 2004: IEEE Region 10 Conference, November 21-24, vol. B, 2, pp. 108–111 (2004) 21. Pavlidis, T., Liow, Y.-T.: Integrating Region Growing and Edge Detection. IEEE Trans. on Pattern Analysis and Machine Intelligence PAMI-12, 225–233 (1990) 22. Morse, B.S.: Bringham Young University, http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/ MORSE/region.pdf 23. Grinaker, S.: Edge Based Segmentation and Texture Separation. In: 5th Int. Conf. Pattern Recognition, Miami Beach, FL, December 1-4, pp. 776–780 (1980) 24. Fua, P., Hanson, A.J.: Using Generic Geometric Models for Intelligent Shape Extraction. In: AAAI 6th National Conference on Artificial Intelligence, pp. 706–711. MorganKaufmann, Los Altos (1987) 25. Xiaohan, Y., Yla-Jaaski, J., Huttunen, O., Vehkomaki, T., Sipila, O., Katila, T.: Image Segmentation Combining Region Growing and Edge Detection. In: 11th IAPR International Conference on Image, Speech and Signal Analysis, August 30-September 3, pp. 481–484 (1992) 26. Yu, Y.-W., Wang, J.-H.: Image Segmentation Based on Region Growing and Edge Detection. In: IEEE International Conference on Systems, Man, and Cybernetics, vol. 6, pp. 798–803 (1999) 27. Al-Hujazi, E., Sood, A.: Range Image Segmentation Combining Edge-Detection and Region-Growing Techniques with Applications to Robot Bin-Picking using Vacuum Gripper. IEEE Transactions on Systems, Man and Cybernetic 20(6), 1313–1325 (1990)

112

N. Jamil et al.

28. Xiang, Z., Dazhi, Z., Jinwen, T., Jian, L.: A Hybrid Method for 3D Segmentation of MRI Brain Images. In: 6th International Conference on Signal Processing, August 26-30, vol. 1, pp. 608–611 (2002) 29. Huang, Y.-R., Kuo, C.-M.: Image Segmentation using Edge Detection and Region Distribution. In: 2010 3rd International Congress on Image and Signal Processing (CISP), October 16-18, vol. 3, pp. 1410–1414 (2010) 30. Deng, W., Xiao, W., Deng, H., Liu, J.: MRI Brain Tumor Segmentation with Region Growing Method Based on the Gradients and Variances Along and Inside of the Boundary Curve. In: 3rd International Conference on Biomedical Engineering and Informatics (BMEI), October 16-18, vol. 1, pp. 393–396 (2010) 31. Fan, S.L.X., Man, Z., Samur, R.: Edge Based Region Growing-A New Image Segmentation Method. Journal of the ACM, 302–305 (2004) 32. Canny, J.: A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 8(6), 679–698 (1986) 33. Otsu, N.: A Threshold Selection Method from Gray-Level Histograms. IEEE Trans. Sys. Man, Cyber. 9(1), 62–66 (1979)

Comparison on Performance of Radial Basis Function Neural Network and Discriminant Function in Classification of CSEM Data Muhammad Abdulkarim, Afza Shafie, Radzuan Razali, Wan Fatimah Wan Ahmad, and Agus Arif Universiti Teknologi PETRONAS, Seri Iskandar, 31750 Tronoh, Perak, Malaysia {mmmhammmad,agusarif}@gmail.com, {afza,radzuan_razali,fatimhd}@petronas.com.my

Abstract. Classification of Controlled Source Electro-Magnetic data into dichotomous groups based on the observed resistivity contrast measures is presented. These classifications may indicate the possible presence of hydrocarbon reservoir. Performance of Radial Basis Function of Neural network and Discriminant Function models were analyzed in this study. Both model's classification accuracy, Sensitivity and Specificity are compared and reported. Gaussian basis function was used for the hidden units in the RBF neural network, while quadratic form is used for the discriminant functions. The Controlled Source Electro-Magnetic data used for this study were obtained from simulating two known categories of data with and without hydrocarbon using COMSOL Multiphysics simulation software. The preliminary result indicates that the radial basis function neural network display superior accuracy, sensitivity and specificity in classifying CSEM data when compared to discriminant functions model. Keywords: Discriminant Function Analysis, Controlled Source ElectroMagnetic, hydrocarbon reservoir, Radial Basis Function, Resistivity Contrast, Visual Informatics.

1 Introduction Control source electromagnetic (CSEM) is one of the techniques used to detect the presence of hydrocarbon layer beneath the sea bed. This method utilizes man made electric and magnetic field to excite the earth beneath the sea floor [1, 2, 3]. In this method, a powerful Horizontal Electric Dipole (HED [4] is used to transmit ultra-low frequency (~ 0.1 ─ 5Hz) electromagnetic waves (EM) waves which is being towed approximately 30 ─ 40m above the seabed to detect resistivity contrasts in the subsurface. Classification accuracy of CSEM data based on the logged EM fields play an important role for oil and gas industry since it can provide an indication presence of H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 113–124, 2011. © Springer-Verlag Berlin Heidelberg 2011

114

M. Abdulkarim et al.

hydrocarbon. The objective of this study is to compare the performance of Radial Basis Functions (RBF) of artificial neural network and discriminant functions analysis techniques for the classification of CSEM data. In order to achieve this objective, CSEM simulations are carried out for without hydrocarbon and with hydrocarbon. Simulation data from both categories are extracted to analyze the classification capability between RBF neural network and Discriminant Functions Analysis (DFA) in discriminating and categorizing the data into one of the two groups labeled as; (Group 1 = Yes Hydrocarbon) and (Group 0 = No Hydrocarbon) based on the measured electric and the corresponding magnetic fields data. The paper is organized as follows; next section briefly explains the radial basis functions neural network and discriminant function analysis, followed by the research methodology and simulation set-ups in section 3. The results and discussion will be given in section 4 and the conclusions in section 5.

2 Radial Basis Function Network Model The radial basis function neural network is a multilayer feed forward neural network consisting of an input layer of source nodes, a layer of nonlinear hidden j locally tuned units which are fully interconnected to an output layer of L linear units. All hidden units simultaneously receive the n-dimensional real valued input vector X (Figure 1). In response to an input vector, the outputs of the hidden layer are linearly combined to form the network response that is processed with a desired response to the output layer.

Fig. 1. Schematic diagram for radial basis neural network architecture

The weights are trained in a supervised fashion using an appropriate linear method [5]. The response of RBF neural network is related to the distance between the input and the centroid associated with the basis function. The RBF hidden-unit output, Zn is obtained by closeness of the input X to an n-dimensional parameter vector Mj associated with the nth hidden unit [6, 7]. The response characteristics of the nth hidden unit (n = 1, 2, 3, …, N) is assumed as:

Comparison on Performance of Radial Basis Function Neural Network

 X − μn Z n = ϕ  σ2 n 

   

115

(1)

Where φ is a strictly positive radially symmetric function (kernel) with a unique maximum at its ‘Centre’ μn and which drops off rapidly to zero away from the centre. The parameter σn is the width of the receptive field in the input space from unit n. This implies that Zn has an appreciable value only when the distance ||X – μn|| is smaller than the width σn. Given an input vector X, the output of the RBF network is the M-dimensional activity vector Y, whose mth component (m = 1, 2, 3, …, M) is given by: Ym ( X ) =

N

W

mn Z n

(X )

(2)

n =1

For m = 1, mapping of eq. (1) is similar to a polynomial threshold gate. However, in the RBF network, radially symmetric kernels are used as ‘hidden units’. The degree of accuracy of these networks is controlled by three parameters: the number of basis functions used, their location and their width [6, 7, 8, 9, 10]. In the present work we have assumed a Gaussian basis function for the hidden units given as Zn for n = 1, 2, 3, …, N, where;  X − μn Z n = exp −  2σ n2 

   

(3)

and μn and σn are mean and the standard deviation respectively, of the nth unit receptive field and the norm is the Euclidean. 2.1 RBF Model Training A training set is an n labeled pair {Xi, di} that represents associations of a given mapping or samples of a continuous multivariate function. The training method used here will adaptively update the free parameters of the RBF network while minimizing the error function E. These parameters are the receptive field centers μn of the hidden layer Gaussian units, the receptive field widths σn, and the output layer weights (Wmn). The training methods considered in the study was a fully supervised gradientdescent method over E [11]. In particular, μn, σn, and Wmn are updated as follows:

Δμ n = −ε μ ∇ μn E

(4)

∂E ∂σ n

(5)

∂E ∂ W mn

(6)

Δσ

n

= −εσ

Δ W mn = − ε w

116


where εμ, εσ and εw are small positive constants. This training model will select the centers and dimensions of the functions and calculates the weights of the output neuron. The center, distance scale and precise shape of the radial function are parameters of the model, all fixed if it is linear. Selection of the centers can be understood as defining the optimal number of basis functions and choosing the elements of the training set used in the solution. It was done according to the method of forward selection [12]. The Artificial Neural Network (ANN) used in this work was implemented using MATLAB 2009a neural network toolbox. 2.2 Discriminant Function Analysis (DFA) The aim of DFA is to obtain a discriminant function that maximizes the distance between the categories, i.e. come up with an equation that has strong discriminatory power between groups. The form of the linear and quadratic discriminant functions of an n-dimensional feature vector is given in equation (7) and (8) respectively by [13] as follows: Dc ( X ) = 2 μ cT σ −1 X − μ cT σ −1 μ c − 2 log P(c)

(7)

Dc ( X ) = ( X − μ c ) T σ c−1 ( X − μ c ) + log σ c − 2 log P (c )

(8)

where for each class c, the parameters µ c and σc denote the mean vector and the covariance matrix for X in the class c, respectively, and are estimated by their sample analogs [see equations (9) and (10)]. P(c) is the a priori probability for the class c.

∧

μc

= X

∧

σc =

c

1 = Nc

Sc 1 = Nc Nc

       

N



i =1 N



X

n1

 X

nm

i =1

(X − X

c )( X

   x1     =   x   m  

    

− X c )T

(9)

(10)

c

The aim of this statistical analysis DFA is to combine (weight) the variable scores in some way so that a single new composite variable, the discriminant score, is produced. In this study, predicted membership is calculated by first producing a score for D for each case using the discriminate function. Then cases with D values smaller than the cut-off value are classified as belonging to one group while those with values larger are classified into the other group. This technique for classification of CSEM data based on phase versus offset has been reported by [14]. 2.3 Terminology and Derivations from a Confusion Matrix A confusion matrix contains information about actual and predicted classifications done by a classification system [15] is given in Table 1. Note that, in this work,


117

classification of No Hydrocarbon (0) data as Yes Hydrocarbon (1) is considered as False Positive (FP) and classification of Yes Hydrocarbon (1) data as No Hydrocarbon (0) is considered False Negative (FN). True Positive (TP) and True Negative (TN) are the cases where the Yes Hydrocarbon (1) is classified as Yes Hydrocarbon (1) and No Hydrocarbon (0) classified as No Hydrocarbon (0) respectively. Table 1. Illustration of Confusion Matrix

Predicted Actual

Negative θ1 θ3

Negative Positive

Positive θ2 θ4

The entries in the confusion matrix have the following meaning in the context of our study:    

θ1 is the number of correct predictions that an instance is negative, θ2 is the number of incorrect predictions that an instance is positive, θ3 is the number of incorrect of predictions that an instance negative, and θ4 is the number of correct predictions that an instance is positive.

The accuracy, sensitivity, specificity and adjusted accuracy were estimated using the following relations [16]: The Accuracy (AC) is the proportion of the total number of predictions that were correct. It is determined using the equation: AC =

θ1 + θ 4 θ1 + θ 2 + θ 3 + θ 4

(13)

The True Positive rate (TP) also known as “Sensitivity” or the “recall” is the proportion of positive cases that were correctly identified, as calculated using the equation: TP =

θ4 θ3 + θ 4

(14)

The False Positive rate (FP) is the proportion of negatives cases that were incorrectly classified as positive, as calculated using the equation: FP =

θ2 θ1 + θ 2

(15)

The True Negative rate (TN) also known as “Specificity” is defined as the proportion of negatives cases that were classified correctly, as calculated using the equation: TN =

θ1 θ1 + θ 2

(16)

The False Negative rate (FN) is the proportion of positives cases that were incorrectly classified as negative, as calculated using the equation:

118


FN =

θ3 θ3 + θ 4

(17)

The Negative Predictive Value (NPV) is the proportion of the predicted negative cases that were correct, as calculated using the equations: NPV =

θ1 θ1 + θ 3

(18)

The Precision (P) also known as “Positive Predictive Value (PPV)” is the proportion of the predicted positive cases that were correct, as calculated using the equations: P=

θ4 θ2 +θ4

(19)

Finally, the Adjusted Accuracy (AA) is the average value of sensitivity and specificity, as calculated using the equation:

AA =

Sensitivit y + Specificit y 2

(20)

Accuracy is the representation of classifier performance in global sense. Sensitivity and Specificity are the proportions of Yes Hydrocarbon (1) data classified as Yes Hydrocarbon (1), No Hydrocarbon (0) data classified as No Hydrocarbon (0) respectively. The adjusted accuracy is a measure that accounts for unbalanced sample data of No Hydrocarbon (0) and Yes Hydrocarbon (1) events. The adjusted accuracy combines sensitivity and specificity into a single measurable value [15].

3 Methodology 3.1 The Simulation Set-Up COMSOL Multi-Physics software is used to model this simulation. The set-up is shown in Fig. 2. The Computational domain is a sphere of 10km diameter. The top region of this sphere represents air with resistivity of 1000Ω. A 500m deep ocean water domain with a conductivity of 3 Sm–1 and relative permittivity of 80 is specified above the mid-plane of the sphere. Below the mid-plane, a conductivity of 1.5 Sm–1 and relative permittivity of 30 is specified for the rock. Embedded in the rock at an average depth of 1000m, there is a block-shaped 100m deep and 7km-by-7km wide hydrocarbon reservoir. The resistivity of the hydrocarbon layer is 1000Ω and the permittivity is 4. The transmitter is modeled as a short 1250A AC line current segment of length 150m of frequency 0.125 Hz is located 35m above the mid-plane. At the external spherical boundaries, a scattering boundary condition absorbs outgoing spherical waves. The Maxwell’s electromagnetic field wave equation in vacuum in the absence of electric or magnetic sources is solved for the electric field vector E inside the computational domain.


119

Fig. 2. Snapshot of the CSEM Simulation set up

3.2 Data and Analysis Procedure The training data were the sets of E-field values extracted from the post processing results of the two CSEM simulations. The RBF neural network architecture considered for this application was a single hidden layer with Gaussian RBF. The basis function φ is a real function of the distance (radius) r from the origin, and the center is μ. Gaussian-type RBF was chosen here due to its similarity with the Euclidean distance and also since it gives better smoothing and interpolation properties [17]. Training of the RBF neural network involved two critical processes. First, the centers of each of the N Gaussian basis functions were fixed to represent the density function of the input space using a dynamic ‘k means clustering algorithm’. The algorithm is given by [18, 19]. This was accomplished by first initializing the set of Gaussian centers μn to random values. Then, for any arbitrary input vector X(t) in the training set, the closest Gaussian center, μn, is modified as:

(

μ nnew = μ nold + η X (t ) − μ nold

)

(21)

where η is a learning rate that decreases over time. This phase of RBF network training places the weights of the radial basis function units in only those regions of the input space where significant data are present. The parameter σn is set for each Gaussian unit to equal the average distance to the two closest neighbouring Gaussian basis units. If μ1 and μ2 represent the two closest weight centers to Gaussian unit n, the intention was to size this parameter so that there were no gaps between basis functions and only minimal overlap between adjacent basis functions were allowed. After the Gaussian basis centers were fixed, the second step of the RBF network training process was to determine the weight vector W which would best approximate the limited sample data X, thus leading to a linear optimization problem that could be solved by ordinary least squares method. Least mean square (LMS) algorithm [9] has been used to estimate these weights. This avoids the problem of gradient descent methods and local minima characteristic of back propagation algorithm [15].

120


For the DFA model analysis, an equal prior was used because all of the populations sizes are equal (i.e. the extracted E-field and corresponding B-field data). Bartlett’s test was carried out to determine the variance-covariance matrices are homogenous for the two population involved, that is the E-field and the B-field data. SPSS statistical software and MATLAB Statistics toolbox were used to carry out the discriminant analysis [13].

4 Results and Discussion 4.1 RBF Neural Network A random sample of 1212 (60%) observed values was used as training, 242 (20%) for validation, 242 (20%) for testing. Training data were used to train the application; validation data were used to monitor the neural network performance during training and the test data were used to measure the performance of the trained application. The network performance plot and plotconfusion matrix in figures 3 & 4 respectively, provides an index of overall RBF model classification performance. The RBF neural network architecture used in this work had two input variables, E-field and its corresponding B-field. It has one hidden layer with five hidden nodes and one output node. The RBF performed best at seven centers and maximum number of centers tried was 11. Mean square error using the best centers was 0.115.

Fig. 3. Snapshot of Neural Network performance plot

Figure 4 shows a snapshot of the Matlab neural network confusion matrix for the training, testing, validation and the overall. The Diagonal cells in each table show the number of cases that were correctly classified, and the off-diagonal cell show the misclassified cases. The cell located at row 3, column 3 (i.e. in the bottom right) of each of the confusion matrices shows the total percent of correctly classified cases and the total percent of misclassified cases just below it within the same cell. The results for all the three data sets (training, validation and testing) show very good recognition. The overall sensitivity of the RBF neural network model was 97.49%, specificity was 98.07% and percentage correct prediction was 97.7%.


121

Fig. 4. Snapshot of the Neural Network Confusion Matrix

4.2 Discrminant Analysis The mean differences between B-field and E-field data together with the standard deviations are depicted in Table 2 for the DFA. The values suggest that these may be good discriminators as the separations are large. Table 2. Group Statistics

Hydrocarbon

Mean

Std. Deviation

0

1

Total

Valid N (listwise) Unweighted

Weighted

Bfield

0.000

0.000

1212

1212

Efield

–1.7E2

10.55

1212

1212

Bfield

0.000

0.000

1212

1212

Efield

–1.4E2

6.47

1212

1212

Bfield

0.000

0.000

2424

2424.000

Efield

–1.57E2

17.52

2424

2424.000

Table 3. Test of Equality of Group Means

Wilks' Lambda

F

df1

df2

Sig.

Bfield

0.248

605.908

1

2422

.000

Efield

0.248

606.040

1

2422

.000

Table 3 provides strong statistical evidence of significant differences between means of Yes Hydrocarbon (1) and No Hydrocarbon (0) groups with E-field having a slightly higher value of F. Since for both variables p < 0.05 and the multivariate test—Wilks‘ lambda of 0.248 each for E and B fields we can say that the model is a good fit for the data.

122

M. Abdulkarim et al. Table 4. Performance of DFA Model

Actual Value DFA

Yes Hydrocarbon (1)

No Hydrocarbon (0)

Total

Prediction

TP = 735

FP = 43

778

False Negative

FN = 59

TN = 375

434

Total

794

418

1212

Table 4 presents a classification results of the DFA with simple summary of numbers classified correctly and incorrectly. This gives information about actual group membership versus predicted group membership. Table 5. Performance of RBF Neural Network

Actual Value RBF

Yes Hydrocarbon (1)

No Hydrocarbon (0)

Total

Prediction

TP = 778

FP = 8

786

False Negative

FN = 20

TN = 406

426

Total

798

414

1212

Table 5 shows the classification results of the RBF with simple summary of numbers classified correctly and incorrectly. This gives information about actual group membership versus predicted group membership. Table 6. Comparison of DFA and RBF

Indices Accuracy Sensitivity Specificity False Positive Rate = 1 – Specificity Positive Predictive Value (PPV) Negative Predictive Value (NPV) False Negative Rate Adjusted Accuracy

DFA 91.58% 92.57% 89.71% 10.29% 94.47% 86.41% 33.24% 91.14%

RBF 97.69% 97.49% 98.07% 1.93% 98.98% 95.31% 34.39% 97.78%

The comparative results of DFA and RBF models are presented in Table 6. Judging by all the indices, the results obtained indicate that the RBF network has a better performance than DFA models. The sensitivity of 97.49% by RBF which is higher than that of DFA 92.57% shows the ability of RBF neural network to correctly identify data indicating presence of hydrocarbon (true positive rate), whereas the specificity of 98.07% for the RBF as the ability of the network to correctly identify data indicating no hydrocarbon (true negative rate) is also higher than 89.71% for the DFA. The False


123

Positive Rate of 10.29% for DFA which is higher than 1.93% for RBF explains the superiority of RBF to have less error of incorrectly identifying data without hydrocarbon as data with hydrocarbon. Figure 5 below shows a MATLAB Graphical Group Classification of the data.

Fig. 5. MATLAB Graphical Group Classification

5 Conclusion Accurate classification of CSEM data based on the logged EM fields can plays an important role for oil and gas industry. This can give an indication of reservoir presence during exploration. In this work, CSEM data are classified into two categories, Yes Hydrocarbon (1) and No Hydrocarbon (0) using artificial neural networks and classical statistical approach. The performance comparisons of the two methods are assessed. It is observed that radial basis function networks have better accuracy when compared to discriminant function analysis. The value of specificity shows that RBF network classifies No Hydrocarbon data more accurately than DFA. The positive predictive value suggests that the classification of CSEM data into Yes or No Hydrocarbon is higher in the RBF than that of the DFA model. This is evident through obtaining higher value false positive rate from DFA than RBF. It appears that ANN could be a valuable alternative to traditional statistical methods and may be particularly useful when the primary goal is classification. The next step in the study is to carry out this analysis with experimental data to establish the full potential of the RBF.

124


Acknowledgment. This research is carried out under the Fundamental and Research Grant Scheme. The authors would like to acknowledge Universiti Teknologi PETRONAS for giving the opportunity to carry out this research work.

References 1. Eidesmo, et al.: Sea Bed Logging (SBL): A new method for remote and direct identification of hydrocarbon filled layers in deep water areas (2002) 2. Constable, S., Srnka, L.J.: An introduction to marine controlled-source electromagnetic methods for hydrocarbon exploration Geophysics 72, WA3 (2007) 3. Hesthammer, J., Boulaenko, M.: The potential of CSEM technology. GeoExpro. and HGS Bulletin 4, 52–58 (2007) 4. Demuth, H., Beale, M., Hagan, M.: Neural Network ToolboxTM User’s Guide 1992-2008, vol. 3, pp. 01760–2098. The MathWorks, Inc., Apple Hill Drive (2008) 5. Raiche, A.: A pattern recognition approach to geophysical inversion usingneural networks. Geophysical Journal International 105, 629–648 (1991) 6. Poulton, M., Sternberg, K., Glass, C.: Neural network pattern recognition of subsurface EM images. Journal of Applied Geophysics 29, 21–36 (1992) 7. Roth, G., Tarantola, A.: Neural networks and inversion of seismic data. Journal of Geophysical Research 99, 6753–6768 (1994) 8. El-Qady, G., Ushijima, K.: Inversion of DC resistivity data using neural networks. Geophysical Prospectin 49, 417–430 (2001) 9. Haykin, S.: Neural Networks a Comprehensive Foundation, 2nd edn. Prentice-Hall Inc. (1999) 10. Singh, U.K., Somvanshi, V.K., Tiwari, R.K., Singh, S.B.: Inversion of DC resistivity data using neural network approach. In: Proceedings of the International Groundwater Conference, IGC 2002, Dindigul, India, pp. 57–64 (2002) 11. Rummelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, pp. 318–362. MIT Press, Cambridge (1986a) 12. Rummelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by backpropagating errors. Nature 323(9), 533–535 (1986b) 13. Pardoe, I., Xiangrong, Y., Dennis, R.C.: Discriminant Analysis and Statistical Pattern Recognition: Graphical tools for quadratic discriminant analysis. John Wiley Sons, New York (2006) 14. Muhammad, A., Afza, S., Radzuan, R.: Wan Fatimah Wan Ahmad: Application of Discriminant Analysis to Phase versus Offset Data in Detection of Resistivity Contrast. Paper will be Presented at NPC Symposium (2011) 15. Kohavi, R., Provost, F.: Machine Learning, vol. 30, pp. 271–274. Kluwer Academic Publishers, Boston (1998) 16. Abboud, S., Barnea, O., Guber, A., Narkiss, N., Bruderman, I.: Maximum expiratory flowvolume curve: mathematical model and experimental results. Med. Eng. Phys. 17(5), 332– 336 (1995) 17. Hartman, E.J., Keeler, J.D., Kowalski, J.M.: Layered neural networks with Gaussian hidden units as universal approximators. Neural Comput. 2, 210–215 (1990) 18. Orr, M.J.: Regularization in the selection of radial basis function centers. Neural Comput. 7, 606–623 (1995) 19. Sing, J.K., Basu, D.K., Nasipuri, M., Kundu, M.: Center selection of RBF neural network based on modified k-means algorithm with point symmetry distance measure. Found. Comput. Decis. Sci. 29, 247–266 (2004)

Simulation for Laparoscopy Surgery with Haptic Element for Medical Students in HUKM: A Preliminary Analysis A.R. Norkhairani1, Halimah Badioze Zaman2, and Azlina Ahmad2 1

Faculty of Technology and Information Science 2 Institute of Visual Informatics, Universiti Kebangsaan Malaysia, 43600 Bangi, Selangor [email protected],{hbz,aa}@ftsm.ukm.my

Abstract. Minimal invasive surgery or laparoscopy has taken been applied to especially pediatrics. Traditionally, medical students learn surgical skills through the mentorship model during their residency training. As technology evolves, more and more applications, simulations and systems are built as aids to medical students in acquiring surgical skills. This study was conducted to investigate how haptic technology can play a part in shaping students’ skills at the Department of Surgery in HUKM. This paper will discuss the first phase of the research: the Preliminary Analysis. Previous research were reviewed; interviews were conducted with experts and related staff of the department; and observations were made during actual training sessions. Results of the preliminary analysis show that current training approaches have some weaknesses that can be replaced by applying simulations with haptic devices. The results will be used to design a framework for Laparoscopy Simulation Surgery with Haptic Elements or Simulasi Pembedahan Laparoskopi dengan Elemen Haptik (SPLasH). This innovation will not only enrich the field of visual informatics, but also improve medical training in the field of Laparoscopy Surgery. Keywords: Simulation, Laparoscopy Surgery, Visual Informatics, Haptics.

1 Introduction In medical field, especially in surgery, various research have been conducted to reduce incisions in surgery that led to the implementation of MAS (minimal access surgery) which is also known as laparoscopy. This procedure needs highly skilled surgeons to operate, and is a difficult skill for the surgical trainees to master [1],[2]. Therefore, surgical trainees needed to spend more time supervised by experts. Some issues were raised such as limitation of time, social and financial aspects that restricted experts to spend more time in training the trainees [2]. These issues led to the use of technology as an effort to help trainees acquire the special skills required. Robotics, simulation and virtual technologies are currently available to fulfil this gap [3]. Besides that, the use of dummy has helped in the acquisition of such surgical H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 125–138, 2011. © Springer-Verlag Berlin Heidelberg 2011

126

A.R. Norkhairani, H. Badioze Zaman, and A. Ahmad

skills [3]. Some of the skills were acquired through supervision by experts together with consistent practices [1],[2]. Simulation is an approach that seems to be accepted as a tool for surgery skill training with the integration of haptic devices that provide force and tactile feedback to increase realism [4]. This multidisciplinary research has attracted researchers in both the medical and computer science field as part of visual informatics[5]. Thus, this preliminary analysis was conducted to investigate how simulation for laparoscopic surgery training with haptics elements or SPLasH can be implemented in medical student training at the department of surgery in HUKM (Hospital Universiti Kebangsaan Malaysia). Specifically, this paper focuses on the current issues in laparoscopy surgery, current research to the date that is carried out to overcome the issues and surgery training method at HUKM. The results obtained will be used as an input for the second phase of the research. 1.1 Literature Review Laparoscopy is a minimally invasive procedure that is used either to diagnose diseases or as a method of surgery [6],[7],[8]. It normally applies to those procedures that cannot be detected through diagnostic imaging such as pelvic pain by gynecologists, and is also used in normal procedure to examine abdominal organs. Laparoscopy starts with a small incision to accommodate the insertion of laparoscope instruments through cannula or trocar and is normally under the navel. Other incisions are also made to allow other insertions of laparoscopy additional instruments and are normally done on both left and right abdomen. Once the procedure is completed, the instruments will be removed and the incision will be sutured and bandaged. This incision will take a shorter time to recover and patients will not need to stay long at the hospital. Sometimes they are treated as daily patients [6],[7],[8]. Currently, there are four common approaches of surgery as summarized in Table 1 [9]. Open Surgery and Robot assisted Laparoscopy are two approaches that share same values in terms of vision and hand freedom which are three Dimensional and 6 DOF (degree of freedom). Degree of freedom refers to momentum of the body that can move independently in three directions of space. However, tactile sensation is absent in Robot assisted Laparoscopy. While endoscopic and laparoscopy share the same level of tactile sensation and hand freedom, they differ in vision criteria where endoscopic is monocular and laparoscopy is two Dimensional. Table 1. Differences between current surgical techniques Surgery Approach Open Endoscopic Laparoscopy Robot assisted laparoscopy

Vision Three Dimensional Monocular Two Dimensional Three Dimensional

Tactile Sensation Fully Present Reduced Reduced Absent

Hand Freedom 6 DOF 4 DOF 4 DOF 6 DOF

The differences, in terms of vision, tactile sensation and hand freedom, between Open surgery and Laparoscopy surgery are not too wide. It is clearly shown in Table 1, that in terms of vision, hand freedom and tactile sensation, laparoscopy shows less degree of freedom and vision compared to open surgery. This lack of properties in

Simulation for Laparoscopy Surgery with Haptic Element for Medical Students

127

laparoscopy can be replaced by using technology. This gap can be fulfilled by the application of simulations that will provide a higher degree of freedom and threeDimensional vision. The absence of tactile sensation can be filled by using appropriate haptic devices that are aligned to the need of the simulations. The approach in surgery training seems controversial among the surgeons [10].Some of the surgeons focus more on observations in the operation theatre and others promote the usage of animals and simulation model or combination of tasks related to surgery. It has been identified that there are a few levels of training in surgery, in order to acquire skills which involve cognitive and motor skills [1]. Traditionally, surgeons go through apprenticeship, which is based on residency training system that was introduced by Sir William Halsted in late 1800s [1]. Currently, training curriculum in medical faculties or institution is multilevel as depicted in Figure 1. Students start with first level of training which is learning through learning aids such as courseware, video, dummies and simulations with special devices. After the basic knowledge gained, they then will move to the next level where observation in operation room and fully simulated environment take place. The higher level involves consultation with experts and lastly, they are ready to practice in hospitals. Each of the levels will increase the skills acquired by the students through different approaches of training method. The higher the students go through, the approaches become closer to the real world.

Fig. 1. Training curriculum of Minimally Invasive Surgery Centre of Caceres (Spain). Simulator or virtual reality system can be implemented at the lowest level of skill training [11].

Surgical training needs to prepare surgeons in mastering a different set of psychomotor and visuo-spatial skills [12]. There are a few skills acquisition theories such as Kopta’s, Schmidt’s Schema, Traditional Apprenticeship, Cognitive apprenticeship and Ericsson’s Rationale for acquisition of expertise as shown in Table 2. Each of the theories has its own phase which is divided into three to six phases. Each phase focuses on each level of the whole process in skill acquisition but overall, it can be concluded that skills can be acquired through three main processes [1].

128


a) Demonstration – surgeons observe how the procedures are being carried out either through video, operation theatre or simulation b) Coaching – surgeons then carry out the procedures with the tutelage of the mentor c) Practice – the surgeons do the procedure on their own in order to be assessed in terms of skills acquired. Table 2. Theories of skill acquisition and their phases Theory Kopta’s

Schmidt’s Schema

Traditional Apprenticeship

Cognitive Apprenticeship

Ericsson’s Rationale for acquisition of expertise

• • • • • • • • • • • • • • • • • • •

Phase Cognitive Integration Autonomous Initial condition Action initiation Sensory feedback Outcome Knowledge Observation Coaching Practice Modelling Coaching Scaffolding Articulation Reflection Exploration Deliberate practice Focused attention Time of day and duration of practice

Currently, there are many applications that have been developed to be used as training aids for minimal invasive surgery (MIS) or laparoscopy. SINERGIA laparoscopic virtual reality simulator is an application that provides exercises for specific skill acquisitions – coordination, speedy coordination, navigation, navigation and touch, accurate grasping, grasp and transfer, coordinate pulling, force sensitivity and accurate cutting [11]. The MISTELS system (McGill Inanimate System for Training and Evaluation of Laparoscopic Skills) was designed to provide a series of tasks that is performed under video guidance in a box using standard laparoscopic instruments to objectively assess basic laparoscopic skills [13]. Meanwhile, Laparoscopic Training Simulator (LTS) 2000 which is based on MISTELS and improved by combining computer-based electronic scoring, interactive user feedbacks and digital capture of performance data. Another part is, it is portable. Tasks provided in MISTELS and LTS are summarized in Table 3.These two training aids basically have the same function but LTS provides more feedbacks to user. Tasks provided in both training aids were dedicated to train basic skills in laparoscopy which involved: grasping (task 1, 6, 8 and 9), pressure (task 4, 5 10) and knotting (task 7, 8 and 9). Results showed that users were more satisfied with LTS.


129

Table 3. Task provided in MISTELS and LTS 2000

1 2 3 4 5 6 7 8 9 10

Task Peg transfer Ring manipulation (right) Ring manipulation (left) Cannulation (right) Cannulation (left) Roeder-type knot Application of knot on vertical tube Extracorporeal knot on horizontal tube Intracorporeal knot and tension test Cutting circle

MISTELS √

√ √ √ √

LTS √ √ √ √ √ √ √ √ √ √

Sansregret et al.[13], used both simulators to a group of medical students or interns, fellows, and attending surgeons with classified experience in laparoscopic surgery (novice, intermediate, competent, expert). Results showed that there was a progressive improvement according to academic level as well as experience. This means that simulators do help in skills acquisition. Another virtual reality simulator is the computer-based developed system called MIST-VR (Mentice AB, Gothenburg, Sweden) that includes interfaces with two laparoscopic instruments that is passed through a frame containing a tracking device [12]. MIST-VR provides basic skills training (core skills 1 and 2) under module 1 and module 2 for intracorporal suture training. The benefits of using haptic devices in surgical simulation have been realized and noticed by many parties including researchers and companies. Few companies that really put effort in this particular area, for example, Imersion Medical, Mentice, Surgical Science and Reachin [14],[15],16]. This has led to the growth of research in haptics and simulations. Over the last ten years, Basdogan, et al.[14] and the team at MIT Touch Lab have put an effort to explore the roles of haptics in Minimal Invasive Surgery Simulation and Training (MISST) as shown in Figure 2. MISST has integrated haptic devices to offer more realistic simulator. They have identified four (4) main areas of haptic roles as follows:

(i) Haptic interfaces— the integration of available haptic devices that have been commercialized with a simulation of MIS procedures. (ii) Haptic rendering—the development of computerized model of surgical instruments to enable collision detection, the geometric model of organs to be visualized, responses of forces to the users and models that are able to respond towards user interaction in real time. (iii) Haptic recording—the use of haptic devices in order to measure the properties of material during recording. (iv) Haptic playback—the development of techniques for user interaction which is based on force feedback to provide guidelines to the users.

130


Fig. 2. The MISST – the handler is connected to Phantom I and Phantom II [12]

MISST divides the instrumentation of laparoscopic into two categories. One category is for instruments that are featured as long, thin and straight which are normally used for puncturing and another group consists of instruments that are articulated which are suitable for pulling, clamping, gripping and cutting. A few challenges were faced during the recording and playback, object deformation and rendering process. Findings of this project stated that the integration haptic into simulator for MIS seems essential since the MIS is a procedure that involves touching, feeling and manipulating organs through instruments. Beside MISST, THUMP that stands for Two-Handed Universal Master Project has been developed to improvise tele-robotic interfaces [17]. This is due to the loss of some classic hand-eye coordination challenges and other traditional limitations of MIS in tele-robotic systems. THUMP incorporate stereoscopic goggles with two eight-axis da VinciTM master mechanisms provided by Intuitive Surgical® [17] . All these systems development incorporate haptic element as tools to provide tactile and force feedback in order to give more realistic experience to the users. Haptics, according to Fisher et al.[18] is to be touched, as visual to be seen and as auditory to be heard while Oakley et al.[19] defines haptics as something that is related to the sense of touch [18-20]. Haptics is possible to be implemented in four areas which are human haptics, computer haptics, multimedia haptics and machine haptics [20],[21],[22]. Researches related to haptics have moved to multidisciplinary research that involves the four main areas[21],[22] as shown in Figure 3. Each of the areas focuses on different aspects of haptics technology but is interrelated to each other. Human haptics is an area that focuses on human sensory motor loop and all related aspects with human perception towards sense of touch. Machine haptic focuses on mechanical devices that replace or augment the human touch. Human haptics involve design and development of mechanical devices. The mechanical devices need a life to be felt by the users where this is the part played by computer haptics. It focuses on the algorithm development and software to produce sense of touch to the virtual object through haptic visual and rendering. While multimedia haptics is referred as the integration and synchronization haptic interfaces data presentation, with other media, in multimedia application that utilized gesture recognition, tactile sense and force feedback [21],[22]. The integration of these four focuses, enable the visualization of information that enhances features in visual informatics.


131

Human haptics • Perception • Cognition • neurophysiology Computer haptics • Modelling • Rendering • Stability

Machine haptics • Device design • Sensors • Communications Multimedia haptics • Collaborative environment

Fig. 3. Multidisciplinary research in haptics [21]

From the above discussions, it can be concluded that skills acquisition in medical field is a critical issue. Various research have been carried out for many years to study the best approach in medical training which for students and surgeons, then led to the combination of learning aids in residency systems. Various training aids have been developed and implemented in the area of medical training, specifically in laparoscopy surgery. Even though these training aids do help in improving surgery skills but there are still weaknesses, such as less sense of grasp and touch. This gap can be improved by implementing haptics technology that offers tactile sensation. Another criterion is, most of the application developed focused on task but none of them provide a single and complete surgery procedure such as hernia repair or cyst remove. This lack of criteria will be fulfilled by SPLasH through this current study. SPLasH will be developed as a simulation that focuses on a complete single procedure that stresses upon important aspects of laparoscopy surgery with consideration to blend the theory of skill acquisition in each of simulation part.

2 Methodology For the initial analysis, two methods were used to gather the required information. This information was used to identify the possibility of implementing haptic technology as additional or alternative tools for medical students at HUKM to acquire the skills required for laparoscopy surgery. All the results obtained were analysed qualitatively. 2.1 Interview with Experts To support the information gathered through paper review, an interview was conducted. The instruments comprised of open ended questions focused according to the different levels of respondents. Table 4 shows detail of the interviews conducted.

132

A.R. Norkhairani, H. Badioze Zaman, and A. Ahmad Table 4. Interviews details

Respondent Experts

Number 1

Role Leader surgery

Senior Fellow

2

Assist Expert in surgery

Skedul Temu Jurubedah (STBJ)

Sister

1

Skedul Temu Bual Ketua Jururawat (STBKJ)

Nurse

3

General Staff

1

Coordinates nurse and equipment Prepares the equipment for surgery Prepare the material for lab training

in

Instruments Skedul Temu Bual Pakar (STBP)

Bual

Nota Jururawat (NJ)

Nota Latihan (NL)

Focus • Surgery approach • Main elements in laparoscopy surgery • Current issues in laparoscopy surgery training tools • Surgery department planning • Current approach in laparoscopy surgery training at HUKM • Surgery preparation • Staff involvement • Instruments used • Identify instrument • Dummies preparation • Material preparation

The information gathered through the interviews complemented with the observations done in operation theatre and the training laboratories. Cross check was then made to ensure that they were accurate and reliable. 2.2 Observations Observations took place after the interviews. The observations were needed to get the real pictures of laparoscopy surgery procedure. The observation done during the training session of laparoscopy surgery, was also needed to complement the data gathered from previous method. Table 5 portrays the whole observation process. Table 5. Observation details

Venue Operation HUKM

Process observed theatre,

Training Lab, Surgery Department, HUKM

Hernia repair (left and right) procedure for 2 month old boy Equipment preparation

Instruments for data capture • Video Camera • Photographs • AIDA workstation • Photographs • Observational notes

Observations were done in both situations, the real laparoscopic surgery procedure in actual operating room and setting for training environment. For the real laparoscopic procedure surgery, observation involved with hernia repair procedure on a 2-month-baby boy. The procedure was recorded using Sony Video CamCorder and AIDA workstation.


133

3 Findings This section will describe results from both methods used. Results were summarized in Table 6. These results were concluded from the interviews conducted based on the instruments used in both methods. Table 6. Summary of findings Methods Interviews

Category Experts

Senior Fellow

Sister

Nurse General Staff Observation

Operation theatre

Training lab

Result • Laparoscopy is taking place as major approach especially for paediatrics • Laparoscopy is a method that need small incision for instruments insertion • Pressure and grasp is main elements in laparoscopy approach • Current learning tools are lacks in accurate pressure and grasp • HUKM is planning to develop a surgical training centre • Currently, dummies are used as training aid for laparoscopy surgery • The same dummies used for every sessions • Animal parts used as human organs • Experts skill is need to guide students • Only few master student is allowed to be in operation theatre • At least 2 – 3 nurses will assist for the process of preoperation, operation and post operation • Each procedure has its’ own set of instruments • Animal’s parts used to replace human organs (hearts, limb and etc.) • Few masters students witness the operation • Anaesthesia experts, assistant surgeons and nurses surround the bed beside the experts • No recording done for pre-operation, outside look of operation and post operation • Operation used set of instruments that specially for paediatrics • No trocar used • Usage of dummy box as training aid • Dummy box consist of upper part and lower part • Upper part is made from rubber and having many pin hole made • Lower part is dedicated for animal’s organ. No stability of organs.

Next section will analyse the results and discuss the significance of the results on the study conducted.

134


4 Discussions As a result of the interviews,it was found that there were no computerized tools used as aids for laparoscopy surgery training. According to the experts, who is known as pediatric experts’ surgeon, haptics application for laparoscopy surgery training that is available in the market today, lacks accuracy in pressure and grasp. It is supported with findings that showed only few virtual reality simulators available today included haptic feedback as a functionality and they were not acknowledged as natural enough [12]. These two main elements are very important in laparoscopy procedure. Currently, at the department of Surgery in HUKM, medical students use dummies as learning aids. This dummy is in a form of box that has a surface that represents human abdomen which is made from rubber on the upper part as shown in Figure 4 and space for putting animal parts (limb, heart and etc.) in lower part as shown in Figure 5. Each time a student creates a small pin on the abdomen, it will remain on the dummies. This leads to the loss of elasticity that happens when a student pushes the instruments at the same spot that should be experienced if the student carries out the experiments on real human abdomen [24]. The loss of elasticity will influence the pressure that the students should be giving to the instruments. Findings by Storm et al.[23], indicate that haptic feedback is important at the early stage of skills acquisition in surgical simulations, while Webster, R. et al.[24] and Cotin, et al.[25] said that users must feel the contact forces in haptics surgical simulation.

Fig. 4. Upper part of the dummy

Fig. 5. Lower part of the dummy

Observations showed that during procedure of the surgery, only a few masters students were allowed to witness inside the operation room. This is due to the limited spaces (surrounding the bed) available, where other important staff needed to be there too, such as anesthesia experts, assistant surgeons and assistant nurses. Even though there was a recording going on, once the instruments were inserted into the abdomen, the pre process of determining where the instruments should be inserted was not recorded at all. The knowledge can only be obtained if the students or interns were present in the operation theatre. Figure 6 to Figure 11 shows some snapshots during the procedure. In Figure 8 it clearly shows there is a need for careful and accurate pressure because there are organs under the abdomen that, if pressure is used more than is needed, it will harm the intestine while insufficient pressure will result to incomplete incision. Figure 10


Fig. 6. Incision made under navel

135

Fig. 7. Laparoscope inserted through the incision

Fig. 8. Cutter is pressed to create incision for Fig. 9. Grasper inserted through incision grasper insertion made in Fig. 8.

Fig. 10. Grasper holds needle and grasps Fig. 11. After the procedure, incision is muscle for suturing almost invisible

shows that accurate pressure hat needs to be given to both graspers so that the appropriate grasping of muscle and needle will ease the suturing process. Figure 11 was recorded after the procedure has been completely done where the incision under

136


the navel and incision made by instruments did not require any bandage as it was too small. These are all the reasons why laparoscopy is much more preferred as an approach in certain cases since it is less risky compared to open surgery approach. This leads to the need for the surgeons to acquire laparoscopy surgery skills in order to practice in this surgical approach, hence the need for the training tools that can provide accurate skills exercises. Haptics technology can fulfill these requirements. From the explanation above, it is very important for students to acquire the same skills set that is demonstrated by the experts through experts’ guide, but issues such as time, cost and social have been obstacles to make it a reality. Simulations are one of the ways to mimicking the expert’s skills. It can be concluded that surgical simulators, dummy box or virtual reality systems allow the exercising of basic laparoscopy tasks repeatedly in a controlled environment, which is free from pressure as if the procedure is carried out on real patients, although they have been criticized in terms of its realisticness [3]. The future plan of the Department of Surgery in HUKM is to develop a surgical training centre that can also be a motivational factor to develop a simulation that can be used as additional or alternative training aids. Preliminary Analysis of SPLasH Initial Analysis: a) Document review (i) Laparoscopy surgery (ii) Current training method in surgery (iii) Skill Acquisition Theory (iv) Adult Learning Theory (v) Past and current haptics implementation in surgery training b) Interview (i) Various level of staff involves with surgery training and procedure (ii) Different aspects of task c) Observation (i) Operation theater – involvement, complete procedure (ii) Training Lab – equipment, procedure, weaknesses -

Theories and methods in Simulation development ID models for simulation development Software Life cycle

Design and Development of SPLasH

ID model for SPLasH; • Kopta’s skill acquisition theories • Multimedia and Haptics • Cognitive and Psychomotor • Hands-on • Force and Tactile feedback Modules: • Guided Learning • Assisted Practices • Self-Practices

SPLasH Prototype

-

SPLash Development Model

User Acceptance Test of SPLasH

Usability test of SPLasH: a) • • •

Students Ability to learn Easy to remember Error rate

b) Medical lecturer / experts • Efficiency • Satisfaction Instruments: (i) Survey • Usability Test For Medical Students (UTFMS) • Usability Test For Medical Lecturers (UTFML) (ii) Observation • Observation note for students (ONFS) • Observation note for lecturers (ONFL)

Fig. 12. Simplified Research Process of SPLasH


137

5 Conclusion Surgery training evolves and attracts attention of researchers in recent years. It has been evolved from various angles of perspectives. Dunkin, B., et al. [13], reported that currently, many companies are actively involved with the implementation of haptics in the simulations, which can be used in laparoscopy skills training, portrayed that researches in this area are highly in demand [16]. He also highlighted issues that need to be covered for the future direction of surgical simulations. For example, the need to analyze the fiscal realities in providing training to the residents and the considerations of adult learning theories that need to be adapted into simulations. Besides that, the competency and proficiency provided by the simulators should be studied carefully in order to provide a surgical simulation that allows the skills acquisition as well as reduce fiscal issues and socials issues in medical training. Next part of the research will use the information gathered in this initial study to propose a design framework to develop a simulation integrating haptic devices as a learning aid for medical students at HUKM. Figure 12 simplified the whole research process. The simulation will use hernia repair as a procedure to gain basic laparoscopy skills. The integration with haptic devices will help students to experience the realism of laparoscopy surgery procedure with the ability of these devices to provide force and tactile feedbacks. This study will enrich the literature for future research conducted in the area of Visual Informatics which is growing rapidly. Acknowledgement. This research is funded by a grant under the Projek Arus Perdana (UKM-AP-ICT-16-2009) entitled ‘Multi-Display Interactive Visualization Environment on Haptic Horizontal Surface’ at Institute of Visual Informatics, UKM.

References 1. Wong, J.A., Matsumoto, E.D.: Primer: cognitive motor learning for teaching surgical skill—how are surgical skills taught and assessed? Nature Clinical Practice Urology 5(1), 47–54 (2008) 2. Hamdorf, J.M., Hall, J.C.: Acquiring surgical skills. British Journal of Surgery 87(1), 28– 37 (2000) 3. Najmaldin, A.: Skills training in pediatric minimal access surgery. Journal of Pediatric Surgery 42(2), 284–289 (2007) 4. Panait, L., Akkary, E., et al.: The role of haptic feedback in laparoscopic simulation training. Journal of Surgical Research 156(2), 312–316 (2009) 5. Zaman, H.B., et al.: Visual Informatics: Bridging Research and Practice. In: Badioze Zaman, H., Robinson, P., Petrou, M., Olivier, P., Schröder, H., Shih, T.K. (eds.) IVIC 2009. LNCS, vol. 5857, pp. 868–876. Springer, Heidelberg (2009) 6. Sándor, J., Lengyel, B., et al.: Minimally invasive surgical technologies: Challenges in education and training. Asian Journal of Endoscopic Surgery 3(3), 101–108 (2010) 7. Chamberlain, R.S., Sakpal, S.V.: A comprehensive review of single-incision laparoscopic surgery (SILS) and natural orifice transluminal endoscopic surgery (NOTES) techniques for cholecystectomy. Journal of Gastrointestinal Surgery 13(9), 1733–1740 (2009) 8. Bax, N.M.A.: Ten years of maturation of endoscopic surgery in children. Is the wine good? Journal of Pediatric Surgery 39(2), 146–151 (2004)

138


9. Laguna, M., Wijkstra, H., et al.: Training in Laparoscopy. Laparoscopic Urologic Surgery in Malignancies, 253–269 (2005) 10. Rosen, J., Hannaford, B., et al.: Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Transactions on Biomedical Engineering 48(5), 579–591 (2001) 11. Lamata, P., Gómez, E.J., et al.: SINERGIA laparoscopic virtual reality simulator: Didactic design and technical development. Computer Methods and Programs in Biomedicine 85(3), 273–283 (2007) 12. Debes, A.J., Aggarwal, R., et al.: A tale of two trainers: virtual reality versus a video trainer for acquisition of basic laparoscopic skills. The American Journal of Surgery 199(6), 840–845 13. Sansregret, A., Fried, G.M., et al.: Choosing the right physical laparoscopic simulator? comparison of LTS2000-ISM60 with MISTELS: validation, correlation, and user satisfaction. The American Journal of Surgery 197(2), 258–265 (2009) 14. Basdogan, C., De, S., et al.: Haptics in minimally invasive surgical simulation and training. IEEE Computer Graphics and Applications 24(2), 56–64 (2004) 15. Basdogan, C., Ho, C.H., et al.: Virtual environments for medical training: Graphical and haptic simulation of laparoscopic common bile duct exploration. IEEE/ASME Transactions on Mechatronics 6(3), 269–285 (2001) 16. Dunkin, B., Adrales, G.L., et al.: Surgical simulation: a current review. Surgical Endoscopy 21(3), 357–366 (2007) 17. Niemeyer, G., Kuchenbecker, K.J., et al.: THUMP: an immersive haptic console for surgical simulation and training. Studies in Health Technology and Informatics, 272–274 (2004) 18. Fisher, B., Fels, S., et al.: Seeing, hearing, and touching: putting it all together. ACM (2004) 19. Oakley, I., McGee, M.R., et al.: Putting the feel in’look and feel. ACM (2000) 20. Eid, M., Andrews, S., et al.: HAMLAT: A HAML-based authoring tool for haptic application development. Haptics: Perception, Devices and Scenarios, 857–866 (2008) 21. Eid, M., Orozco, M., et al.: A guided tour in haptic audio visual environments and applications. International Journal of Advanced Media and Communication 1(3), 265–297 (2007) 22. El Saddik, A.: The potential of haptics technologies. IEEE Instrumentation and Measurement Magazine 10(1), 10 (2007) 23. Ström, P., Hedman, L., et al.: Early exposure to haptic feedback enhances performance in surgical simulator training: a prospective randomized crossover study in surgical residents. Surgical Endoscopy 20(9), 1383–1388 (2006) 24. Webster, R., Haluck, R., et al.: Elastically deformable 3D organs for haptic surgical simulation. Studies in Health Technology and Informatics, 570–572 (2002) 25. Cotin, S., Delingette, H., et al.: A hybrid elastic model for real-time cutting, deformations, and force feedback for surgery training and simulation. The Visual Computer 16(8), 437– 452 (2000)

Detection and Classification of Granulation Tissue in Chronic Ulcers Ahmad Fadzil M. Hani1, Leena Arshad1, Aamir Saeed Malik1, Adawiyah Jamil2, and Felix Yap Boon Bin2 1 Centre for Intelligent Signal & Imaging Research, Department of Electrical & Electronics Engineering, Universiti Teknologi PETRONAS, 31750 Tronoh, Perak, Malaysia 2 Department of Dermatology, General Hospital, Kuala Lumpur, 50586 Kuala Lumpur, Malaysia

Abstract. The ability to measure objectively wound healing is important for an effective wound management. Describing wound tissues in terms of percentages of each tissue colour is an approved clinical method of wound assessment. Wound healing is indicated by the growth of the red granulation tissue, which is rich in small blood capillaries that contain haemoglobin pigment reflecting the red colour of the tissue. A novel approach based on utilizing haemoglobin pigment content in chronic ulcers as an image marker to detect the growth of granulation tissue is investigated in this study. Independent Component Analysis is employed to convert colour images of chronic ulcers into images due to haemoglobin pigment only. K-means clustering is implemented to classify and segment regions of granulation tissue from the extracted haemoglobin images. Results obtained indicate an overall accuracy of 96.88% of the algorithm performance when compared to the manual segmentation. Keywords: Chronic Wounds, Ulcers, Granulation Tissue, Haemoglobin, Independent Component Analysis, Visual Informatics.

1 Introduction 1.1 Chronic Ulcers and Healing Assessment In pathology, wounds that fail to follow a normal specified course of healing are categorized as chronic wounds [1]. Ulcers are one common type of chronic wounds. Ulcers are generally divided into three main categories based on their etiological causes: vascular, pressure, and diabetic ulcers [2]. Ulcers are most commonly found on the lower extremity below the knee and affect around 1% of adult population and 3.6% of people older than 65 years [3]. Chronic ulcers introduce a major problem in dermatology and a huge economic dilemma especially in western countries. Chronic wounds affect three million to six million patients in the United States and treating these wounds costs an estimated $5 billion to $10 billion each year [2]. The annual cost associated with the care of ulcers is approximated to be £400 million in the H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 139–150, 2011. © Springer-Verlag Berlin Heidelberg 2011

140

A.F.M. Hani et al.

United Kingdom [4]. Non-healing ulcers can cause patients to suffer severe pain and discomfort, and may subject patients to the risk of limb amputation. Figure 1 shows two examples of chronic leg ulcers.

Fig. 1. Chronic Leg Ulcers* *Images acquired at Dermatology, Hospital Kuala Lumpur

Different types of tissue exist on ulcers surface as they progress throughout the healing process: black necrosis, yellow slough, red granulation and pink epithelial tissue. At any one time, all the four tissue types can be present on the ulcer surface. Recognizing and measuring the amount of each tissue is an approved method of wound assessment and understanding of wound progression. In daily clinical practice, physicians inspect the healing status of ulcers by describing the tissues inside the ulcer in terms of percentages of each tissue colour based on visual inspection utilizing the Black-Yellow-Red scheme or the Wound Healing Continuum [5], [6]. This method is widely used in clinical settings, but has the disadvantage of being simple, subjective and more importantly difficult to qualify and quantify. Moreover, chronic wounds heal gradually which makes detection of small changes with visual inspection challenging. Hence, imaging techniques utilizing colour digital images are developed to provide precise, objective and reliable data that aid physicians to evaluate healing status of ulcers. Some developed techniques in the area of wound tissue classification and segmentation are based on quantifying colour content in images using conventional colour models such as RGB and HIS [7],[8]. Other methods utilize the information obtained from RGB colour histogram to develop clustering techniques for automatic classification and segmentation of different types of tissues within the wound site [9-12]. Most recently, as part of the ESCALE project devoted to the design of a complete 3D and colour wound assessment tool, an unsupervised wound tissue segmentation method was proposed [13], [14]. The method utilizes three selected unsupervised segmentation methods, J-SEG, Mean Shift and CSC, to segment colour wound images into different regions. It then extracts both colour and texture descriptors from coloured images as inputs to an SVM based classifier for automatic classification and labelling of these regions. The work developed in the field of wound assessment thus far is based on processing and analyzing the colour content as the major component in digital imaging. However, the interpretation of colour content in the image is always compromised by the unavoidable differences in acquisition conditions such as the camera and flash used and the lighting in the room. Therefore, an alternative

Detection and Classification of Granulation Tissue in Chronic Ulcers

141

approach based on the optical characteristics of wound components is explored in this study. 1.2 Hypothesis and Research Objective One of the major changes during wound healing is the colour of the tissues, which results from human vision perception of the light reflected from the skin. Human skin is a highly heterogeneous media with multi-layered structure. Most of the incident light penetrates into the skin and follows a complex path until it exits back out of the skin or gets attenuated by skin pigments such as melanin and haemoglobin [15], [16]. Figure 2 shows the absorbance spectra of the main pigments in human skin, melanin, haemoglobin (deoxy-haemoglobin and oxy-haemoglobin) and bilirubin [16].

Fig. 2. Absorption Spectra of Main Skin Pigments, Melanin, Deoxy-haemoglobin (Hb), Oxyhaemoglobin (HbO2), and Bilirubin* * Reproduced from R.R. Anderson and J. A. Parrish (1981)

The first indication of ulcer healing is the growth of the mew healthy red granulation tissue on ulcer surface. Granulation tissue appears red in colour due to the presence of small blood vessels that contain haemoglobin pigment. Studies show that haemoglobin contains certain optical characters that can be detected from colour images and used to show their content within human skin [17-21]. Haemoglobin pigment exhibits its high absorbance at wavelengths around 420-430 nm with no absorbance (total reflectance) for wavelengths below 620nm. The portion of the electromagnetic spectrum that is visible to the human eye extends from violet at about 380 nm to red at about 750 nm as shown in figure 3. Different colour component exhibits different reflectance ranges in the visible spectrum. Red colour component exists at wavelengths between 620-740 nm from the visible light. This explains why the granulation tissue that contains haemoglobin pigment appears red in colour when viewed under visible light.

142

A.F.M. Hani et al.

Fig. 3. Reflectance Spectrum of the Visible Light

Because haemoglobin pigment causes the red colour of the granulation tissue it is hypothesized that image regions due to haemoglobin pigment, could be extracted from colour images of chronic ulcers. Extracted image regions due to haemoglobin pigment represent areas of haemoglobin distribution over the ulcer surface, which in return indicates regions of granulation tissue. Hence, the objective of this research work is to study haemoglobin pigment content in chronic ulcers as an image marker to detect the newly growing healthy granulation tissue.

2 Approach Independent Component Analysis (ICA) is a technique that extracts the original signals from mixtures of many independent sources without a priori knowledge on the sources and the process of the mixture [22], [23]. In digital imaging, colour is produced by combination of three different spectral bands, Red, Green and Blue (RGB). In this work, colour RGB images of chronic ulcers are considered the observed mixtures from which independent source of haemoglobin pigment content is extracted. The focus of this work is particularly on extracting image regions due to haemoglobin pigment, which represent detected areas of granulation tissue. 2.1 Colour Images of Chronic Ulcers Colour images of chronic ulcers that contain a mixture of each type of tissues are acquired at Hospital Kuala Lumpur, Malaysia. This is very crucial to the research work as it ensures working on ulcers images taken under controlled acquisition conditions. Each time before data acquisition session, the ulcers have to be examined and cleaned by the nurses at the hospital. Images of ulcers are then taken before the new dressing is applied to the ulcers. A total of 110 ulcer images are acquired from 69 patients. A Digital Single Lens Reflector (DSLR) camera with resolution of 12.3 megapixels is used to acquire colour images of chronic ulcers. A flash light is used to provide adequate reproducible lighting for image acquisition. To avoid spectral reflections and shadows, a diffuser dome was mounted on the flash light. Furthermore, small reference sticker of size 9x13 mm is placed next to the ulcer to provide a size reference. The images that provide the best view of the ulcer wounds under optimum lighting conditions are used in the analysis. 2.2 Detection of Granulation Tissue Each RGB colour ulcer image comprises of three grey-level images representing the spectral bands of Red, Green and Blue channels. For each ulcer image, these bands


143

are used to create row vectors of data comprising an observation dataset, which represent the mixtures upon which Independent Component Analysis (ICA) is implemented. In this work, ICA is applied using the FastICA algorithm developed by Hyvärinen and Oja [22]. FastICA is based on a fixed-point iteration that uses maximization of non-gaussianity as a measure of independence to estimate the independent components. Figure 4 is a flow chart of the algorithm applied in this work. The dataset is first normalized to centre on zero point by extracting the mean value from each spectral band. Data whitening was then applied by employing Principle Component Analysis (PCA) to transform the dataset linearly so that its components are uncorrelated and their variances equal unity. These steps are performed to simplify the ICA algorithms and reduce the number of parameters to be estimated [22]. ICA was then implemented to extract grey level images representing areas of haemoglobin distribution. The algorithm is developed using MATLAB version 9.0 and implemented on the images with JPEG file format.

Fig. 4. Flow Chart of the Algorithm Implemented

2.3 Segmentation of Granulation Tissue Segmentation is an image processing technique that is used to subdivide an image into constitutes regions [24]. Data clustering is an unsupervised learning approach where similar data items are grouped together into clusters. K-means clustering is a method of hard clustering which aims to partition N observations into K clusters in which each observation belongs to the cluster with the nearest mean [25]. The number of

144

A.F.M. Hani et al.

clusters K is an important input parameter to the algorithm and hence an inappropriate choice of K may yield poor results. Hence, it is important to run diagnostic checks for determining the number of clusters in the data set. K-means clustering is applied in this study to classify the extracted haemoglobin images into distinctive clusters based on the Euclidean distance of each pixel value to the mean of the cluster centre. The algorithm is applied iteratively with number of cluster (K) ranging from K=2 till K=8. The parameter K is initially chosen as K=2 because the algorithm is expected to segment at least two distinctive regions, the granulation tissue and the rest of the ulcer. Some extracted haemoglobin images represent three distinctive intensity value regions and hence for segmenting the granulation tissue in these images, K is initially chosen as K=3 to avoid misclassifying other wounds regions as part of the granulation tissue. The classified image is then converted into a binary image based on the intensity values of the clustered granulation tissue. Results obtained are shown and discussed in the next section.

3 Results and Analysis In clinical practice, ulcers are assessed based on their appearance and tissue condition. Physicians evaluate the type and condition of the granulation tissue according to its colour which is determined by the concentration of haemoglobin pigment. There are three main types of granulation tissue categorized by physicians: bright beefy red, dusky pink and pale pink. The type of granulation tissue determines the level of the ulcer healing from newly grown bright beefy red till the healed pale pink. Acquired ulcer images are assessed by the dermatologists at Hospital Kuala Lumpur to determine the type and amount of granulation tissue existing on the surface of these ulcers. 3.1 Detection of Granulation Tissue The proposed algorithm is applied on the colour images of ulcers to extract images that represent distribution of haemoglobin pigment. The extracted grey-level images indicate the distribution of haemoglobin pigment which reflects areas of granulation tissue on the surface of these ulcers. Figures 5 shows the results obtained. Figure 5(a) shows an image extracted representing bright beefy red granulation. Figure 5(b) shows an image extracted representing dusky pink granulation and figure 5(c) shows image extracted representing pale pink granulation. By examining the extracted images due to haemoglobin, the regions, which appear with distinctive range of intensity values, represent areas of haemoglobin pigment distribution, highlighting areas of granulation tissue. This shows that the novel approach upon which the granulation tissue detection algorithms are developed successfully detects areas of granulation tissue on ulcer surface regardless of its type.


145

(a)

(b)

(c) Fig. 5. Extracted Images Representing Granulation Tissue Regions Detected

3.2 Segmentation of Granulation Tissue Clustering based segmentation is employed to segment regions of granulation tissue from extracted images due to haemoglobin pigment. K-means clustering is utilized to classify the extracted images due to haemoglobin pigment into clusters of different regions. Figure 6 shows the results obtained from classifying an extracted grey level haemoglobin image into clusters.

146

A.F.M. Hani et al.

Fig. 6. Classified Image obtained from Haemoglobin Image

The classified image is then converted into a binary image based on the intensity values of the clustered granulation tissue region. Figure 7 shows the results of converting the classified image in figure 6 into a binary image with segmented regions of granulation tissue.

Fig. 7. Binary Image obtained from Classified Image

3.3 Validation of Results Twenty-two colour images of chronic ulcers are selected from the collected images based on different ulcer wounds with varying severity and healing status to validate the developed algorithm. The segmentation algorithm using k-means clustering is applied iteratively on each extracted haemoglobin image with parameter K ranging from K=2 or K=3 till K=8. The algorithm performance is validated by comparing the segmented granulation regions obtained with the developed algorithm for each number of clusters (K) against the ones obtained from manual segmentation. The manual segmentation is performed by the operator based on the dermatologist’s assessment of granulation tissue type and amount of each ulcer image. The difference between the algorithm segmentation and the manual segmentation is obtained by calculating the number of mis-segmented pixels as follows:


147

n

Error =

 xi 1

where n= Number of wrongly segmented pixels

N

N= Image size The accuracy of the algorithm performance is calculated as follows: Accuracy = |1-Error| x100 Table 1 below shows the calculated highest accuracy achieved of the developed segmentation algorithm performance, with the corresponding number of cluster (K), compared with the results obtained from the manual segmentation for each ulcer image. The table also shows the average accuracy achieved along with the corresponding standard deviation for each ulcer image when applying the algorithm iteratively for all range of K=2 or K=3 till K=8. Table 1. Accuracy of the Proposed Segmentation Algorithm Compared with Manual Segmentation Ulcer

Highest Accuracy

K

Average Accuracy

Standard Deviation

Ulcer_1 Ulcer_2 Ulcer_3 Ulcer_4 Ulcer_5 Ulcer_6 Ulcer_7 Ulcer_8 Ulcer_9 Ulcer_10 Ulcer_11 Ulcer_12 Ulcer_13 Ulcer_14 Ulcer_15 Ulcer_16 Ulcer_17 Ulcer_18 Ulcer_19 Ulcer_20 Ulcer_21 Ulcer_22 Overall

93.76 92.99 95.86 97.50 95.42 94.20 98.41 97.33 98.00 98.05 95.62 98.01 97.78 94.05 97.17 96.08 98.79 99.42 97.71 98.84 96.99 99.40 96.88

4 5 6 6 4 3 4 4 5 4 5 4 5 3 3 3 8 7 5 3 4 8

92.18 91.87 91.43 95.00 88.94 91.95 97.31 95.31 96.94 96.45 92.58 96.60 95.01 91.55 90.01 94.08 91.27 95.79 96.59 96.02 94.58 97.74 93.87

1.67 1.38 8.22 3.91 4.18 2.33 0.96 3.13 1.63 1.08 3.16 1.39 6.21 4.58 6.52 1.65 14.57 5.74 1.12 2.34 3.83 2.47 3.89

From Table 1, it is noted that the developed algorithm performs fairly well with an overall highest frequency of 96.88% and average accuracy of 93.87% with corresponding standard deviation of 3.89. The algorithm successfully segments regions of granulation tissue with K mostly ranging between K=3 and K=4. The algorithm compares well particularly with manual segmentation for ulcers, which

148

A.F.M. Hani et al.

contain regions of granulation tissue with well defined edges such as ulcers 7, 12 and 20. Such ulcers are shown in figure 8. However, some ulcer wounds are filled with exudates and infected blood, such as ulcers 17 and 22 shown in figure 9. In these ulcers, it is difficult to classify the little granulation tissue on the ulcer edge with small number of K. Hence, best performance results for these ulcers are achieved when K is selected to be a higher integer such as K= 7 or K=8.

Fig. 8. Ulcers Containing Single Granulation Region with Well- Defined Boundary

Fig. 9. Ulcers Filled with Exudates and Infected Blood

Fig. 10. Ulcers Containing Granulation Tissue Mixed with Slough

On the other hand, the algorithm does not compare well with manual segmentation for ulcers, which contains several scattered regions of granulation tissue such as ulcers1, 2, 6 and 14. This is because scattered granulation regions that are normally


149

mixed with slough or exudates are hard to detect and segment manually. Examples of these ulcers are shown in figure 10. It is noted that the number of clusters (K) is an important parameter that has to be determined to achieve a successful segmentation of granulation tissue with high accuracy when compared to the manual segmentation. Results obtained thus far are to be further analyzed to enhance the developed algorithm. The main goal is to achieve a fully automated segmentation with high accuracy and reliable reproducible results. Priory knowledge on the number and types of tissues contained within the ulcer may be required as an input to the system to determine the optimum number of clusters needed to achieve accurate results. Further review is required to relate the findings obtained so far with the current techniques available on image segmentation to make the system automated and more robust.

4 Conclusion A novel approach to investigate the healing status of chronic ulcers by detecting the healthy granulation tissue is investigated in this study. Most of the work developed in the field of wound assessment utilize colour in images for image analysis. However, the interpretation of colour is compromised by the unavoidable differences in acquisition conditions and settings. In chronic ulcers, granulation tissue appears red in colour due to the presence of blood vessels that contain haemoglobin pigment. Hence, a new approach based on analysis of the optical characteristics of haemoglobin pigment for the detection of granulation tissue is investigated. Independent Component Analysis is employed to convert colour images of chronic ulcers into images due to haemoglobin pigment only. K-means clustering is implemented to classify and segment regions of granulation tissue from the extracted haemoglobin images. Results obtained indicate the proposed algorithm detects and segments regions of granulation tissue successfully regardless of their type and amount. The algorithm performs with an overall accuracy of 96.88% when compared to manual segmentation. This introduces a new objective non-invasive technique to assess the healing status of chronic wounds in a more precise and reliable way. Acknowledgment. This is a collaborative work with the Department of Dermatology, General Hospital Kuala Lumpur, Malaysia. The authors would like to thank the hospital staff in assisting them to acquire coloured images of chronic ulcers.

References [1] Keast, D., Orsted, H.: The Basic Principles of Wound Healing (2002) [2] Werdin, F., Tennenhaus, M., Schaller, H.-E., Rennekampff, H.-O.: Evidence-based Management Strategies for Treatment of Chronic Wounds (June 4, 2009) [3] London, N.J.M., Donnelly, R.: ABC of Arterial and Venous Disease: Ulcerated Lower Limb. BMJ 320, 1589–1591 (2000) [4] Margolis, D.J., Bilker, W., Santanna, J., Baumgarten, M.: Venous Leg Ulcer: Incidence and Prevalence in the Elderly. J. Am. Acad. Dermatol. 46(3), 381–386 (2002) [5] Goldman, R.J., Salcid, R.: More than One Way to Measure a Wound: An Overview of Tools and Techniques. Advances in Skin and Wound Care 15(5) (2002)

150

A.F.M. Hani et al.

[6] Gray, D., White, R., Cooper, P., Kingsley, A.: The Wound Healing Continuum- An Aid To Clinical Decision Making And Clinical Audit (2004) [7] Herbin, M., Venot, A., Devaux, J.Y., Piette, C.: Colour Quantitation Through Image Processing in Dermatology. IEEE Transactions on Medical Imaging 9(3) (September 1990) [8] Herbin, M., Bon, F.X., Venot, A., Jeanlouis, F., Dubertret, M.L., Dubertret, L., Strauch, G.: Assessment of Healing Kinetics Through True Colour Image Processing. IEEE Transactions on Medical Imaging 12(1) ( March 1993) [9] Mekkes, J.R., Westerhof, W.: Image Processing in the Study of Wound Healing. Clinics in Dermatology 13(4), 401–407 (1995) [10] Berris, W., Sangwine, S.J.: A Colour Histogram Clustering Technique for Tissue Analysis of Healing Skin Wounds. In: IPA 1997, July 15-17 (1997) [11] Zheng, H., Bradley, L., Patterson, D., Galushka, M., Winder, J.: New Protocol for Leg Ulcer Tissue Classification from Colour Images. In: Proceedings of the 26th Annual International Conference of the IEEE EMBS San Francisco, CA, USA, September 1-5 (2004) [12] Galushka, M., Zheng, H., Patterson, D., Bradley, L.: Case-Based Tissue classification for Monitoring Leg Ulcer Healing. In: Proceedings of the 18th IEEE Symposium on Computer-Based Medical Systems (CBMS 2005), pp. 1063–7125 (2005) [13] Wannous, H., Treuillet, S., Lucas, Y.: Supervised Tissue Classification from Colour Images for a Complete Wound Assessment Tool. In: Proceedings of the 29th Annual International Conference of the IEEE EMBS, Lyon, France, August 23-26 (2007) [14] Wannous, H., Lucas, Y., Treuillet, S., Albouy, B.: A Complete 3D Assessment Tool for Accurate Tissue Classification and Measurement. In: 15th International Conference on Image Processing, ICIP 2008, pp. 2928–2931 (2008) [15] Donner, C., Weyrich, T., d’Eon, E., Ramamoorthi, R., Rusinkiewicz, S.: A Layered, Heterogeneous Reflectance Model for Acquiring and Rendering Human Skin (2009) [16] Anderson, R.R., Parrish, J.A.: The Optics of Human Skin. Journal of Investigative Dermatology 77, 13–19 (1981) [17] Cotton, S.D., Claridge, E.: Developing a Predictive Model of Human Skin Colouring. In: Proceedings of SPIE Medical Imaging, vol. 2708, pp. 814–825 (1996) [18] Tsumura, N., Haneishi, H., Miyake, Y.: Independent Component Analysis of Skin Colour Image. Journal of Society of America 16(9), 2169–2176 (1999) [19] Cotton, S., Claridge, E., Hall, P.: A Skin Imaging Method Based on a Colour Formation Model and its Application to the Diagnosis of Pigmented Skin Lesions. In: Proceedings of Medical Image Understanding and Analysis, pp. 49–52. BMVA, Oxford (1999) [20] Claridge, E., Cotton, S.D., Hall, P., Moncrieff, M.: From Colour to Tissue Histology: Physics Based Interpretation of Images of Pigmented Skin Lesions. In: Dohi, T., Kikinis, R. (eds.) MICCAI 2002. LNCS, vol. 2488, pp. 730–738. Springer, Heidelberg (2002) [21] Tsumura, N., Haneishi, H., Miyake, Y.: Independent Component Analysis of Skin Color Images. In: The Sixth Color Imaging Conference: Color Science, Systems, and Applications (1999) [22] Hyvärinen, A., Oja, E.: Independent Component Analysis: Algorithms and Applications. Neural Networks 13(4-5), 411–430 (2000) [23] Langlois, D., Chartier, S., Gosselin, D.: An Introduction to Independent Component Analysis: InfoMax and FastICA algorithms. Tutorials in Quantitative Methods for Psychology 6(1), 31–38 (2010) [24] Gonzalez, R., Woods, R.E.: Digital Image Processing, 2nd edn. Prentice Hall (2006) [25] Fung, G.: A Comprehensive Overview of Basic Clustering Algorithms (June 2001)

New Color Image Histogram-Based Detectors Taha H. Rassem and Bee Ee Khoo Universiti Sains Malaysia, School of Electronic and Electrical Engineering 14300, Seberang Perai Selatan, Nibong Tebal, Penang, Malaysia [email protected], [email protected]

Abstract. Detecting an interest point in the images to extract the features from it is an important step in many computer vision applications. For good performance, these points have to be robust against any transformation that can be done on the images such as viewpoint change, scaling change, rotation, and illumination and, etc. Many of the suggested interest point detectors are measuring the pixel-wise differences in the image intensity or image color. Lee and Chen [1] used image histogram representation instead of pixel representation to detect the interest points. They used the gradient histogram and the RGB color histogram representation. In this work, different color model's histogram representation such as Ohta-color histogram, HSV-color histogram, Opponent color histogram and Transformed-color histogram are implemented and used in the proposed interest point detector. These detectors are evaluated by measuring their repeatability and matching score between the detected points in the image matching task and the classification accuracy in the image classification task. It is found that as compared with intensity pixels detectors and Lee's histogram detectors, the proposed histogram detectors performed better under some image conditions such as illumination change, blur and some other conditions. Keywords: Interest point detector, histogram, image classification, image matching, Visual Informatics.

1 Introduction To compare similar images, it is enough to detect some edges or lines in each image and make the comparison. This is an easy task. Unfortunately, all images are not similar in natural. The images may be exposed to certain circumstances or some transformations that make the comparison task so difficult. This prompted the computer vision researchers and scientists for searching and detecting strong points in the image that are invariant. These points should be invariant under scaling change, rotation, illumination change, compression variation and blur change that may occur to the image. Point detector step is an important step in many computer vision applications such as matching applications, classification, retrieval, recognition and, etc. The different types of point detectors can be shown in Fig 1. H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 151–163, 2011. © Springer-Verlag Berlin Heidelberg 2011

152

T.H. Rassem and B.E. Khoo

Fig. 1. Image detectors types. 1(a) Original image 1(b) Random point detector 1(c) Interest point detector 1(d) Dense point detector 1(e) Dense interest point detector.

In the random point detector, the regions are selected randomly without any criteria; it is an easy and bad point detector, whereas the interest point detector is selecting high information content regions [2]. In fact, that, the density of features is playing the important role than interest point features, especially in the recognition and classification tasks, the dense sampling point detector is suggested [3]. As shown in Fig 1(d), it is covering all locations inside the image by regular patches with different scales [4]. Dense interest point detector is suggested by Tuytelaars [2]. It is selecting all patches such as the dense sampling detector then selects the interest points from these points. The dense interest point detector can be shown in Fig 1(e). The interest point detectors are exploring the variation of the image representation such as pixel intensity value or pixel color value. Recently interest point detectors such as Harris, Hessian and etc are measuring the pixel-wise differences between image pixel intensity or color [5, 6]. These interest point detectors can be called pixelbased detector. Harris, Hessian [5,7-9], Maximally stable extremal regions(MSER) [10], detector based on intensity extreme (IBR) [11], an edge-based region detector (EBR) [12], salient region detector, log of Gaussian (LOG) and difference of Gaussian (DOG) detectors are pixel interest point detectors. A complete survey about local feature detectors can be found in [8, 9, 13]. Some of these detectors are exposed to some modification to increase their invariant property. Multi-scale Harris and Hessian affine detectors are most reliable detectors. To explain the Harris detector briefly. Firstly, the first derivatives of moving Gaussian window around is computed. Then, the second moment matrix SC is computed from these derivatives. Finally, the trace and determinant of the second moment matrix are used to decide and identify the interest points [14]. SC , σ, σ′

G X, σ′

I I I

,σ ,σ

I I I

,σ ,σ

(1)

Where σ is the integration scale of the Gaussian window, while σ is the differentiation scale of Gaussian kernels to compute the local image derivative. In case of Hessian detector, second derivative is used. It has strong response, especially on regions and blobs [15]. HE

HE , σ

h h

h h

I I

,σ ,σ

I I

,σ ,σ

(2)

The researchers are trying to use different image representation other than intensity in the detection process. Lee and Chen [1] are suggested to use the histogram representation instead of pixel intensity to build a new detector. This detector can be

New Color Image Histogram-Based Detectors

153

called histogram-based detector. The discrete quantity b , with L levels (bins) can be derived from either intensity or color from the image pixel point , . At each pixel location , , the value of the k-th bin of a weighted histogram , h , ,….. is computed as ,

Z

∑

,

,

,

,

(3)

Where the w , is a Gaussian weighting function, and . is the indictor function. The set Ω , defines a neighborhood around , , and Z is a normalization term , 1. to ensure∑L To find the interest point, Bhattacharyya coefficient is used to measure the , and shifted point , [16]. All similarity between the histogram at calculations details can be found in [1]. At last, the response function R of Hessian matrix that used to determine whether a pixel is an interest point or not is defined as det where the hessian matrix ,

(4)

is defined by

∑

,

,

,

,

,

(5)

Any low level features can be used for constructing histogram such as intensity gradient or color histogram. Lee and Chen used RGB color histogram to construct RGB histogram based interest point and gradient of intensity histogram based interest point. They quantized each color channel in RGB channel into 8 bins and total 512 bins histogram. In each point, the discrete quantity can be computed from the following quantization function ,

, 32

8

, 32

8

, 32

8

1

where , , , and , are the RGB values of pixel , . In intensity gradient histogram, they quantized the orientation of gradient into 8 bins; each covers 45-degree angle and the magnitude into 8 bins. In total, 64 oriented gradient histogram is used in [1]. RGB and gradient histogram based interest point detectors performed well in some conditions as mentioned in [1]. In this paper, new color histogram representations are used as interest point detectors such as Ohta-color histogram, Transformed-color histogram, Opponent color histogram and HSV-color histogram. The same process is used as the Lee and Chen histogram process after converting the image into the different color model. Compared with intensity pixels detectors and Lee's histogram detectors, our color histogram detectors performed better under some image conditions such as illumination change and blur and some other conditions. The new detectors are evaluated by measuring their repeatability and matching scores (distinctively) in the image matching task and their accuracy performance in the image classification task. This is due to the good properties, and the advantages of some color models. The rest of the paper is organized as follows:

154


the proposed color histogram based detectors are explained in Section 2. In section 3, image matching experimental results and image classification experimental results are explained and provided. Finally, the paper concludes along with discussion in Section 4.

2 Proposed Color Histogram Based Interest Point Detectors New color histogram based interest point detectors are explained in this section. The same process is used as Lee and Chen histogram process after converting the image into the suggested color spaces. The Ohta color space [17], HSV color space, Opponent color space, and Transformed color space are used instead of RGB color space. We believed, each color space has special properties and these properties may affect the detection process. In each color space, each channel is divided into 8 bins and totally 256 histogram bins in each histogram. The quantization function can be described as , 32

,

, 32

8

, 32

8

8

1

where , , , and , are the value of pixel , in A, B and C channels in each color space. The Ohta color space channels, the Opponent color space channels and the Transformed color space channels can be described as shown in Table 1. Fig 2 shows an example of the Lee's histogram detectors and our proposed histogram detectors. Table 1. Color spaces conversation equations Ohta color channels R

I

R

I I

G 3

2G

Opponent color channels B

B R 2

R

O O

B O

G R

√2 R

G

2B G

√6 R

Transformed color channels

G √3

B

B

R

μR σR

G

μG σG

B

μB σB

3 Experiments The proposed detectors are applied to the image matching task. These images are subjected to different conditions. The aim is to measure the repeatability and matching score between the detected points locating at the same category. This is the first part of experiments. In the second part, the proposed detectors are applied to image classification task. Lee and Chen histogram detectors and Harris and Hessian pixel detectors are compared with our proposed detectors in each experiment part.


155

Fig. 2. Histogram-based detectors. 2(a) Gradient histogram-based interest point. 2(b) RGB histogram-based interest point. 2(c) Opponent histogram-based interest point. 2(d) HSV histogram-based interest point. 2(e) Ohta histogram-based interest point. 2(f) Transformed color histogram-based interest point.

3.1 Experiments on Image Matching Find the similarity between two similar scenes under different conditions such as different illumination, different scales or different viewpoint is an objective of the image point detectors. All the detectors in the literature are trying to overcome these different conditions in which sometimes lead to mismatch between the similar scenes. The proposed detectors are evaluated by measuring the repeatability and matching score for the detected points. Mikolajczyk et al. suggested a special data set for image matching task [14]. It has eight sets of images, and each set has six similar images in scene and category but under different conditions. Actually, this data set has two different scene images; structural scenes that contain homogenous regions with distinctive boundaries such as graffiti, boat and bikes and textured scenes that contain repeated texture of different forms such as tree, wall and bark. In each scene, three different variations are included such as scale change, blur change and view point change. The remaining two sets contain two other different variations such as JPEG compression and illumination change. Fig 3 shows some examples of these images set of the test data set. Any two regions in the left image in Fig 3(a) and other images in Fig 3(a) can be defined as corresponding regions if the overlap error between them is small. Many corresponding regions mean high repeatability and mean that the more stable interest point detector. Moreover, extracting the features from these corresponding regions is used to distinguishing and measuring the correct matching scores [14]. In the following figures, the results of image matching task under each image condition are shown. The same settings are used in these experiments as [2]. The regions are normalized to 30 pixels in size while the threshold of the overlap error is 40%. From Fig 4 to Fig 11, we can conclude that HSV color histogram, Transformed color histogram and Opponent color histogram based interest point detector performed better than another histogram-based interest point detector. Moreover, they performed better than the intensity point detector in some situations. This can be observed clearly in some textured scene such as well (view point change) and tree (blur change) and illumination change. The proposed detectors had acceptable performance in other situations such as the textured scene (Scale change),

156


structured scene (blur change) and structured scene (scale change). In addition, they performed bad in JPEG compression and worst performance in the structured scene (viewpoint change).

Fig. 3. Part of test data set. 3(a) Zoom + rotation (structural scene) 3(b) Viewpoint change (structural scene) 3(c) Viewpoint change (textured scene) 3(d) Blur change (structural scene).

(a)

(c)

(b)

(d)

Fig. 4. Textured scene bark images (scale change). 4(a) Repeatability 4(b) Number of corresponding regions 4(c) Number of correct matching regions 4(d) Matching score.


(a)

(b)

(c)

(d)

(a)

(b)

157

Fig. 5. Structured scene bikes images (Blur change). 5(a) Repeatability 5(b) Number of corresponding regions 5(c) Number of correct matching regions 5(d) Matching score.

(c)

(d)

Fig. 6. Structured scene boat images (zoom + rotation change). 6(a) Repeatability 6(b) Number of corresponding regions 6(c) Number of correct matching regions 6(d) Matching score.

158


(a)

(b)

(c)

(d)

Fig. 7. Structured scene Graf images (viewpoint change). 7(a) Repeatability 7(b) Number of corresponding regions 7(c) Number of correct matching regions 7(d) Matching score.

(a)

(b)

(c)

(d)

Fig. 8. Leuven images (light change). 8(a) Repeatability 8(b) Number of corresponding regions 8(c) Number of correct matching regions 8(d) Matching score.


(a)

(b)

(c)

(d)

159

Fig. 9. Textured scene trees images (blur change). 9(a) Repeatability 9(b) Number of corresponding regions 9(c) Number of correct matching regions 9(d) Matching score.

(a)

(b)

(c)

(d)

Fig. 10. Ubc images (JPEG comperssion). 10(a) Repeatability 10(b) Number of corresponding regions 10(c) Number of correct matching regions 10(d) Matching score.

160


(a)

(b)

(c)

(d)

Fig. 11. Textured scene wall images (Viewpoint change). 11(a) Repeatability 11(b) Number of corresponding regions 11(c) Number of correct matching regions 11(d) Matching score.

3.2 Experiments on Image Classification In this section, all the detectors mentioned in this paper are compared and evaluated in the image classification task. Different data sets with different properties are used. Object recognition data sets such as Caltech 04, Caltech 101 and Graz02 and Scene recognition data sets such as Oliva and Torralba 4 man-made categories (OT4MM), Oliva and Torralba 4 natural categories (OT4N) and Oliva and Torralba 8 categories (OT8)[18]. With each dataset, the interest points are detected using each interest point detector. Then SIFT are extracted from these points as descriptors. Bag-of features are used to represent the SIFT descriptors and obtain the descriptor's histogram. Table 2, shows the number of vocabulary word that are used with each data set. K-means clustering is used to build the visual code book and non-linear χ2 is used for classification. Table 2. Data sets information and number of vocabulary word Data set

# of Vocabulary Word

# of Categories

# of Training Images

# of Testing Images

Caltech04

300

6

600

300

Caltech101

600

102

3060

1355

GARZ02

500

4

600

300

OT8

500

8

1200

800

OT4MM

500

4

600

400

OT4N

500

4

600

400


161

In each data set, the experiments are repeated 20 times with randomly training and testing images using each detector. The mean and standard deviation of the 20 times results are considered as the classification accuracy. Table 3 shows the classification results of image classification for Caltech04, Caltech101, Graz02, OT8, OT4N and OT4MM. From the table results, Hessian affine detector is the best for Caltech101, Graz02 and OT4MM. This is because of the images in these data sets are more structured images than textured images. Furthermore, these data sets are complex cluttered background. Opponent histogram detector and HSV histogram detector are better for the remaining data sets (OT4N, Caltech04 and OT8). The classification performance using the interest point detectors is less than the dense detector, specially the Caltech101 classification which is worse when compared with a lot of previous works in [4, 19-21]. Nowak et al.[3] proved that important factor to achieve high accuracy in the classification and recognition systems is extracting the features densely. The dense detectors are guaranteed to cover all objects or scenes in the images. Moreover, dense detectors covered all low contrast regions rather than other interest point based detectors. Table 3. Image classification accuracy Detector type

OT4MM

OT4N

OT8

Hessian affine detector Harris affine detector RGB histogram based interest point Gradient histogram based interest point HSV histogram based interest point Transformed color histogram based interest point Opponent histogram based interest point Ohta histogram based interest point Detector type Hessian affine detector Harris affine detector RGB histogram based interest point Gradient histogram based interest point HSV histogram based interest point Transformed color histogram based interest point Opponent histogram based interest point Ohta histogram based interest point

62.17±2.892 60.95±1.561 58.70±1.354 61.96±2.257 61.68±2.535 57.86±1.961 61.88±2.702 58.94±1.758 Caltech 04 81.05±2.007 73.80±3.039 83.55±2.562 83.20±1.939 86.23±2.274 80.08±2.339 86.25±1.767 82.73±2.129

51.35±2.371 47.44±2.240 61.08±2.051 54.96±1.729 60.74±2.993 59.09±2.439 62.55±2.014 59.64±2.289 Caltech 101 21.68±0.619 17.52±0.935 15.85±0.909 17.21±1.005 16.29±0.782 13.83±0.861 17.07±0.708 16.40±0.667

44.82±1.437 42.89±1.437 44.99±1.596 46.11±1.448 47.08±1.786 44.35±1.259 47.74±1.412 46.08±1.580 Graz02 65.09±3.167 64.09±3.105 57.91±2.671 62.11±2.901 59.76±3.287 54.91±3.154 59.20±4.167 59.42±3.021

4 Conclusions In this paper, we present four new color histogram-based detectors. Instead of using the pixel intensity, the histogram representation is used in the detection process to detect the interest points. Their performance evaluated for the image matching task using textural and structural data set scenes [14] and for image classification using the number of object and scene data sets. In the image matching, the proposed detectors achieved good repeatability and matching scores in the textural scenes especially in case of illumination and blur change and acceptable to worse repeatability in the structural scenes. In the image classification, the proposed detectors achieved better classification accuracy than other interest point detectors in OT4N, OT8 and Caltech04 data sets and worse classification accuracy in other data sets. Lastly, the image classification results were worse using the interest point detectors compared

162


with the results using the dense detector, especially the Caltech101 classification. This is evidence that the densely extracted feature from images is a crucial stage in the image classification systems. In future work, we plan to use the histogram representation to implement dense histogram-based detector to take the advantage of the densely features. In addition, to implement a hybrid dense histogram-based interest point detector.

References 1. Wei-Ting, L., Hwann-Tzong, C.: Histogram-based interest point detectors. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1590–1596 (2009) 2. Tuytelaars, T.: Dense interest points. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2281–2288 (2010) 3. Nowak, E., Jurie, F., Triggs, B.: Sampling Strategies for Bag-of-Features Image Classification. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, pp. 490–503. Springer, Heidelberg (2006) 4. Bosch, A., Zisserman, A., Munoz, X.: Image Classification using Random Forests and Ferns. In: IEEE 11th Conference on Computer Vision, pp. 1–8 (2007) 5. Harris, C., Stephens, M.: A Combined Corner and Edge Detection. In: Proceedings of The Fourth Alvey Vision Conference, pp. 147–151 (1988) 6. Montesinos, P., Gouet, V., Deriche, R., Pelé, D.: Matching color uncalibrated images using differential invariants. Image and Vision Computing 18, 659–671 (2000) 7. Lindeberg, T.: Feature Detection with Automatic Scale Selection. Int. J. Comput. Vision 30, 79–116 (1998) 8. Mikolajczyk, K., Schmid, C.: A Performance Evaluation of Local Descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 27, 1615–1630 (2005) 9. Tuytelaars, T., Mikolajczyk, K.: Local invariant feature detectors: a survey. Found. Trends. Comput. Graph. Vis. 3, 177–280 (2008) 10. Matas, J., Chum, O., Urban, M., Pajdla, T.: Robust wide baseline stereo from maximally stable extremal regions. In: British Machine Vision Conference, pp. 384–393 (2002) 11. Tuytelaars, T., Van Gool, L., D’Haene, L., Koch, R.: Matching of affinely invariant regions for visual servoing. In: IEEE International Conference on Robotics and Automation, pp. 1601–1606 (1999) 12. Kadir, T., Zisserman, A., Brady, M.: An Affine Invariant Salient Region Detector. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, pp. 228–241. Springer, Heidelberg (2004) 13. Schmid, C., Mohr, R., Bauckhage, C.: Evaluation of Interest Point Detectors. Int. J. Comput. Vision 37, 151–172 (2000) 14. Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T., Gool, L.V.: A Comparison of Affine Region Detectors. Int. J. Comput. Vision 65, 43–72 (2005) 15. Szeliski, R.: Computer Vision: Algorithms and Applications, 1st edn. Springer, Heidelberg (2011) 16. Comaniciu, D., Ramesh, V., Meer, P.: Real-time tracking of non-rigid objects using mean shift. In: IEEE Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 142–149 (2000) 17. Ohta, Y., Kanade, T., Sakai, T.: Color Information for Region Segmentation. Computer Graphics and Image Processing 13, 222–241 (1980)


163

18. Oliva, A., Torralba, A.: Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope. Int. J. Comput. Vision 42, 145–175 (2001) 19. Boiman, O., Shechtman, E., Irani, M.: In defense of Nearest-Neighbor based image classification. In: IEEE Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2008) 20. van der Sande, K., Gevers, T., Snoek, C.: Evaluating Color Descriptors for Object and Scene Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 32, 1582–1596 (2010) 21. Rassem, T.H., Khoo, B.E.: Object class recognition using combination of color SIFT descriptors. In: IEEE Imaging Systems and Techniques (IST), pp. 290–295 (2011)

Digital Training Tool Framework for Jawi Character Formation Norizan Mat Diah and Nor Azan Mat Zin Department Of Information Science, Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, 43600 UKM BANGI, Selangor, Malaysia [email protected], [email protected]

Abstract. There have been a number of studies carried out on computer-based Jawi writing. However, most of the studies focused on academic learning based on Roman letters. Jawi is a writing script for the Malay language, adopted from Arabic alphabets. Jawi characters are considered very difficult for children to write as it requires fine motor skills and need to be properly written. This article will discuss about the attributes involve in constructing the framework for digital training tools for Jawi character formation. There are four attributes concerning the pertaining to the formation of Jawi characters highlighted in this study. The four leading dimensions of the study are: Pre-writing activity, Jawi character formation process, Practice activities and Jawi character recognition. Previous and current studies discussed in this paper suggest that all of the dimensions highlighted are compatible with the formation of Jawi characters, even though most of the reviews are not about Jawi script. Keywords: Jawi character formation, training tool.

1 Introduction The history of education in Malaysia shows that the Islamic Education has been taught either by using books or texts written in Jawi script and is still being practiced today. A study by Amin [1], revealed that students are weak in the area of Jawi writing and reading skills. Consequently, students who are unable to master Jawi reading and writing skills will be left out in the Islamic subjects as Jawi is the medium of instruction of these subjects [2] Nik Yaacob [3] emphasized that learning to write Jawi must be taught at an early age. A study by Tulisan [4] found that children learning Jawi display a higher degree of weakness in mastering Jawi writing than Jawi reading. This finding is proven by Penguasaan [5] in a survey on Jawi writing skills among school children. The survey showed that some children were unable to master Jawi writing even at the age of 9. Jawi is currently taught in schools, however the methods used to teach Jawi is not motivating [3]. The teaching methods used are the ‘chalk and board’ approach and flash cards [6]. A proper technique needs to be adopted to make learning Jawi interesting activity. Hence, it is important to have suitable learning tools such as digital tools equipped with appropriate interactivity environment and proper interface design to aid children in developing an interest towards learning Jawi. H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 164–175, 2011. © Springer-Verlag Berlin Heidelberg 2011

Digital Training Tool Framework for Jawi Character Formation

165

Technology has become an influential factor when it comes to education. Advances in computer technology have allowed educators to develop new technology mediated spaces for teaching and learning. This advantage gives students the opportunity to choose what they are interested in learning. The term Courseware or Computer-Aided Instruction (CAI) covers a range of computer-based packages, which aim to provide interactive environment for learning. By adding the elements of interactivity into teaching, it becomes easier to understand lessons through audiovisual aids [7].Students prefer interactive learning provided by tools such as digital tools to motivate and aid them learning Jawi, and not just using the traditional flash cards.There are many computerized products for learning Jawi in the market but not many contribute towards Jawi handwriting within an interactive learning environment. Feedback provided by the computer is one of the interactive learning environments that can be applied in the field of Jawi education. The aim of this research is to develop an interactive tool for learning to write Jawi alphabets. Real data input via a touch screen device is used to formulate feedback. In this paper, we present the first part of our research, a framework to develop an interactive training tool to aid learners in learning to form Jawi characters using a touch screen device. The paper is divided into four sections; we start with the introduction as the first section. Section two describes the background of the study. Section three presents the proposed framework of the research. Finally we provide the conclusion for the paper in section four.

2 Background of the Study Motor skills are important not only at the early age of a child but also throughout their lives, as these skills especially those involving fine motor skills are needed to implement activities or movements of complex operations that require strong and proper control. As mentioned by Lippincott [8], activities that children are doing now lay the foundation for their skills in writing, cutting and the use of other abilities needed for grade school. Fine motor skills require the movements of a group of small muscles which are located above the hand muscles (palm). This group of muscles is essential for a variety of activities such as the coordination of the eyes, the movements of the tongue and lips, and the wriggling of the toes. Handwriting is an activity that involves the use of hand and is one of the important parts of fine motor skill, young children are involved in reading and writing abilities of children increased their visual skills and fine motor skills [9]. Research in motor skill development for handwriting, particularly cursive, has shown that many difficulties may challenge the early stages of handwriting development, or even before [10]. Handwriting is a very important fine motor skill learned during the early school years and formal handwriting instruction may begin as early as the kindergarten years. [11] reported that children in kindergarten spent about 45% of their activities on fine motor skills using paper and pencil. The developing of writing ability is not only important in building a child's self-esteem, but is also considered an essential ingredient for success in school [12]. This is because handwriting performance has a direct effect on academic performance [13], whereby children struggling with handwriting may totally avoid writing and decide that they cannot write.

166

N.M. Diah and N.A. Mat Zin

Such problems as reported in [13] appear to occur in learning Jawi as reported by [14]. He emphasized that learning to write Jawi must start at an early age, with the proper way and proper techniques. The Jawi script is the basic medium for teaching and learning of Islamic Education in Malaysian schools today, especially for learning the Qur'an [15]. The aim of Jawi lessons is to equip students with the skills in reading and writing Jawi to ensure that students understand the various fields of knowledge in Islamic Education. Jawi is a script derived from the Arabic alphabets and was adopted for the use of Malay language writing. The adaptation of Arabic script with six extra characters to accommodate Malay vocal sounds made up the total Jawi characters of twenty five as presented in Figure 1.

Fig. 1. Jawi Characters

Jawi writing which has evolved since the arrival of Islam to the Malay peninsular, was widely used in official communications and transactions during the era of the Malay Malacca sultanate [16]. Unfortunately, Jawi has been replaced with Roman script from the time of the British colonialization era until the present. Currently many people do not know how to read or write in Jawi since it is not widely used [17]. in addition, Jawi is confined only to Islamic matters and is primarily used in the Islamic courts, mosques, religion offices and religious schools [18]. Students are not interested in learning Jawi due to lack of motivation and the fact that there are only a few teachers who are proficient in the Jawi script [18]. The young generation also has negative attitudes towards learning Jawi, perceive it as difficult to learn and not important since it is not evaluated in examination [19]. Furthermore, the current teaching and learning process of Jawi is monotonous, not interesting and lacking in the use of effective teaching tools [20]. The research by [18] also shows that majority of the students feel that Jawi script is very complicated compared to Romanized characters. Hence, they become uninterested and quickly become bored in the traditional classroom environment. One way to overcome this problem is by using technology to aid children to learn and master Jawi character writing skill because nowadays children spend more time using computers. Computers have proven to be beneficial to the field of education. However a computer with an interface based on keyboard and mouse has limited practical use in schools [21]. This is because computers cannot be used when children are involved with handwriting activity. Hermann [22] suggested that handling the keyboard interferes with the writing process, and if this is the case, then the use of more natural interfaces, such as speech and handwriting recognition device, may be desirable. The


167

advantage a pen (digital pen) has over a keyboard is that it supports the activities of spelling and is a natural interface to the computer [23]. The availability and affordability of graphic tablets and digital pens, and tablet PCs today, make entering a response to computer via the computer handwriting mode a workable alternative to the keyboard. Pen-based computers have several crucial advantages that can help student with electronic handwriting. The advantages include handwriting recognition engine, handwriting-based interfaces, and development of handwriting-based “killer application” [24]. Facilities provided by the pen-based technology in computer handwriting can help solve the problem of Jawi writing difficulty for children. There is a need to study approaches to create software for processing Jawi characters by storing data (hand writing input via tablets or similar touch screen device) in the form of digital image. Suitable pattern recognition algorithms will be applied to formulate and then generate feedbacks to the user. Next, we present results from literature analysis, a framework for developing a training tool for children to learn forming Jawi characters, using intelligent character recognition technology.

3 Framework for Jawi Writing Training Tool The framework is composed of all dimensions involved in developing a training tool that aids the teaching of children in forming Jawi characters on a touch screen of a computer monitor. A stylus pen is a pen-shaped instrument that is used to select menus and to write Jawi characters onto devices that contain a touch screen. There are four components in the framework as shown in Figure 2.These components are discussed in detail in the following subsections. i. ii. iii. iv.

Pre-writing activity Jawi character formation Practice activities Jawi character recognition

Fig. 2. Framework for Jawi writing training tool

168


Young children from three to five years of age used their hands to explore and learn about the environment and themselves. Developing hand and other pre-writing skills helps young children to prepare for the next step in the world of writing. The method of implementing pre-writing activities for pre-school children is a great approach to build essential foundational fine motor skills. With reference to deDiegoCottinelli and Barros [25], children practice tracing and drawing with pencils, crayons or even with their finger to gain basic pencil-control skill in the learning phase. Prewriting activity is a part of Graphomotor skills to achieve basic movements that are essential part of letter writing [26]. According to Carlson and Cunningham [27], Graphomotor skill is the coordination of the finger, hand, and arm to make desired marks when drawing or writing. Graphomotor helps children achieve the basic movements that are part of letter writing [28]. It is a phase of the writing process that emphasizes the basic hand movements and muscle development necessary for future penmanship. A pre-writing activity is defined as a whole set of tasks that should be accomplished to practice and assess motor skills for handwriting [26]. Before handwriting becomes an additional and complementary ways of expression for children, they need to become familiar with the use of writing tools. Pen-based activities are the leading activities for pre-school children before entering the world of writing. The pen-based skills will include hand strength, directional movement patterns, and effective hand position, which will facilitate making lines, letters, and shapes. Initially, very young children engage in picture drawing and scribbling that looks like writing. According to Clay [29], as children gain experience with printed materials and writing, this scribbling begins to take on the characteristics of print (linearity and separate units). Drawing and scribbling are addressed to be the obvious strategies children used to explore writing at the beginning [30]. Levin and Bus [31] provided evidence for continuity from children’s drawing and scribbling to writing letters. The pictures and scribbles children call “word” indicate many interesting trends: children firstly scribble randomly, and then gradually start to draw directional lines, and show some pictures containing meanings [32] Several studies have explored the connection between young children’s drawing and writing as a way to explore [33]. According to [33], children draw pictures and write to organize ideas and construct meaning from their experiences. [31] found that children’s writing began with drawing activities. By the time children reach the age of four, adults could distinguish children’s writings from their drawings, in part because children used writing-like features including segmentation into small units and linearity, as well as letters and letter-like shapes in their writing [34]. In a syllabic language, children’s ability of drawing and scribbling in emergent writing are acknowledged to help children gain familiarity with the lines and shapes of letters, the appropriate size of words and the direction of sentences [35]. Use of tools in writing begins at the nursery schools by the practice of drawing and continues at primary school mainly through copying task [36]. Tracing helps young child develop a sense of order for how writing is done, and develop his concentration and work on his fine motor skills. It has been shown that by using tracing/copying exercises students perform significantly better than students who are not allowed to participate in these exercises [37][38]. Research by [39] shows that there is a good correlation between copying, tracing, and writing skill.


169

Therefore it can be concluded that four types of activities are required by preschool children in pre-writing skills before they learn to form Jawi letter (characters). The activities involved are a) scribbling, b) drawing, c) tracing and d) copying. In the formation of manuscript letters, the child will need to be able to form straight lines in horizontal, vertical, and diagonal positions, and to form circles and partial circles. The directions of the stroke which are top-to-bottom and left-to-right also constitute as factors in letter formation [40]. Normally pre-school children should be developmentally ready to form the basic lines (vertical, horizontal, circular and oblique) that constitute manuscript letters by the time they enter school at the age of five [41]. The type of the stroke applied in the drawing and tracing activities for the children include the practice of drawing straight lines, then more challenging lines such as curves, zigzags and diagonal lines. Children will scribble continuous line and sketch soft and continuous pattern of basic line such as shapes of letters involving straight line (vertical and horizontal), curves line and oblique line. The pre-writing activities proposed by the [42] include dot to dot drawing of pictures, objects, shapes and numbers. Based on the literature scribbling, reviewed, it can be concluded that all pen-based activities apply three types of strokes; scribbling, a) straight line, b) oblique line and c) cursive line, to ensure that children are ready to form the Jawi characters for the next activity. 3.2 Jawi Character Formations Jawi character formation is slightly different from roman letters because the Jawi alphabet involves extensive use of curves and doted. Although Jawi writing uses a lot of curves, the character formation is virtually the same as for all of other characters, such as Roman, Japanese, Indian and Chinese. Early study on handwriting by [43] compared the usefulness of copying and tracing as methods of initial handwriting instructions. Only practice in copying can lead to improved performance. [44], studied the effects of copying strokes and letter forms, tracing or connecting dots placed at various points of the letter, and discriminating between correct and incorrect letter forms on handwriting performance. The results from the study showed that children were more engaged in connecting dots rather than in learning to write letters. [45], reported that copying letters improved when the models to be copied are models that depicted motion. A way of teaching handwriting is to explain the form and the order of letter strokes in addition to copying exercises [46]. Generally, the teaching methods for handwriting include tracing, copying, exercises, and drill [47]. Many commercially prepared systems of instruction introduce beginners to handwriting in two stages: tracing letters and independent production of letters [47]. Recently, [48] introduced a four-way method for teaching handwriting. The method consists of: i. The child wrote the letter after seeing the instructor writing it, ii. The child wrote the letter after seeing a copy of it with arrow indicating the order and direction of each stroke, iii. The child wrote the letter from memory after examining a copy of it, iv. The child wrote the letter from memory after examining a copy of it with arrows indicating the order and direction of each stroke.

170


Children must be able to both perceive the shape of the model and evaluate the deviation between their own handwriting product and the standard. Based on the findings of the research discussed scribbling, together with findings from , [49] [13], three important elements that constitute a handwriting exercise have been identified and will be considered in this research, for Jawi character formation. The elements are a) tracing, b) copying letters, and c) producing letters under dictation. 3.3 Practice Activities The process of transferring behaviors that are model-dependent (as displayed in tracing and copying), to self-initiated and self-organized sequence of behaviors that are carried out without any outside support, is a complex set of learning [29]. Strategies for learning letter formation include modeling, tracing, copying, dictating, composing, self-monitoring, and peer recording [50]. Researcher by [51] indicated that good teaching requires the student to have systematic and extra instruction in handwriting. According to [52], newborn to eight-year-old children learn best from methods that are consistent with developmentally appropriate practice. Handwriting as a complex skill can be improved through practice, repetition, feedback and reinforcement [53]. Three of this elements come from the teachinglearning model, called an acquisitional model [54], derived from theories of learning. This model are instructional programs used by occupational therapists and are based on developmental and acquisitional theories. McCluskey [55] conducted a study on practice involving repetition of different letters, using different writing implements, classroom assignments, therapist feedback, and child self-evaluation. The findings suggest that programs involving writing practice, repetition, and feedback may improve handwriting slightly more than when no treatment is given. Quoting an example gives by [56], if a student who has consistently been unable to write any letters of a spelling dictation test begins to write the first letter sound of the dictated word, praise should be provided to that student. The teacher can then increase the criteria for the reinforcement so that only correct answers will be rewarded [57]. Furthermore [58] states that practice through repetition, feedback and reinforcement is essential if the skill of letter formation is to be mastered. A computer-based tool is an effective environment in which to practice as user need not be supervised but can be still monitored. Thus the three elements of a) repetition, b) feedback and c) reinforcement will be used as practice activities in the Jawi character formation tool. 3.4 Jawi Character Recognition Research into handwriting recognition suggests that accuracy rates of between 87% and 93% are achievable with constrained writing [59]. [60] found that secondary school children enjoyed using speech recognition technology but gave a disappointing 80% ratings for character recognition. Handwriting recognition is generally implemented in a computing environment in which a pen (digital pen) can be used for direct manipulation of screen objects [61].


171

Normally, handwriting is digitally captured at the point of creation; this is generally done with a graphics tablet or a special stylus or pen. The data input by user is then displayed as script and stored as an ‘ink’ data type. The data type has information about the position and time, and it is this data that is subsequently ‘recognized’ by the recognition algorithms [62]. Handwriting recognition system is a system that automates the process of turning handwritten work into a computer readable form [5]. When the handwriting is in the form of binaries, the computer is able to recognize them and this can ease the process of giving feedback to children especially on how to improve their handwriting [63]. There are researches on children’s handwriting for instance research on the use of optical character recognition for text entry on a PC for children. In this work, the authors compare different methods for children to input data to a computer. A project by [64] is aimed at automatically assessing exercises in which the pupils have to fill in the gaps using a given set of words. In a research on handwriting recognition by [65] writing using tablets was found to be quite efficient, and children were able to write reasonable stories. The Jawi character formation activity and practice activity requires a character recognition algorithm. This is because handwriting teaching involves showing children how to reproduce a letter according to a standard form. In order to produce a character, children must be able to perceive both the shape of the character in its standard form and the deviation between their own handwritten product and the standard, and software must be able to give feedback about on the accuracy and quality of products. Therefore, character recognition technology is an important component in the proposed framework.

4 Conclusion The framework proposed in this paper is based on an extensive review of the literature. However, most previous studies focused on the Romanized letters used in English writings, and several other characters such as Chinese and Japanese kanji. Researches concerning writing Jawi or Arabic characters are very limited; hence a new study is needed to overcome this gap. Findings from the literature on other characters will help guide our study. Although the way other characters are formed differ slightly, the basics of letter/character formation is similar. There are four components in the proposed framework: i. ii. iii. iv.

Pre-writing activity Jawi character formation Practice activities Jawi character recognition

These components can help in developing an effective training tool to aid learners in learning to form Jawi characters using the touch screen technology. The proposed framework needs to be verified through implementation in interactive learning software.

172


References 1. Amin, A.M.: Tulisan Jawi ke arah penggunaan yang lebih meluas dan berkesan. Journal Dewan Bahasa 33(10), 937–941 (1989) 2. Endut, M.: Pendidikan Islam KBSM: Satu Kajian Tentang Masalah Pengajaran di Sekolahsekolah di Negeri Johor. In: Tesis, M. (ed.), Universiti Kebangsaan Malaysia, Bangi (1992) 3. Nik Yaacob, N.R.: Pengusaan Jawi dan Hubungan dengan Minat dan Pencapaian Pelajar dalam Pendidikan Islam. Journal Pendidik dan Pendidikan 22, 161–172 (2007) 4. Ismail, M. R.: Tulisan Jawi dan Isu – Isunya dalam Pendidikan, Malaysia (2008), http://www.scribd.com/doc/2582591/tulisanjawi 5. Ahmad, M.A.: Penguasaan Tulisan Jawi di Kalangan Murid-murid Tahun Tiga (2011), http://ww.scribd.com/doc/18584642/kajian-jawi 6. Abdullah, N.A., Raja Ahmad Kamaruddin, R.H., Razak, Z., Mohd Yusoff, Z.: A Toolkit Design Framework for Authoring Multimedia Game-Oriented Educational Content. In: 8th IEEE International Conference on Advanced Learning Technologies, pp. 144–145. IEEE Press, New York (2008) 7. Becker, H. J., Raviz, J. L., Wong, Y.T.: Teacher and Teacher-Directed Student Use of Computer and Software Teaching, Learning, and Computing 1998National Survey Report #3. Center for Research on Information Technology and Organizations University of California, Irvine and University of Minnesota (1999) 8. Lippincott, C.: Fine motor activities for preschoolers (2004), http://make-thegradeot.com/Fine%20Motor%20Activities%20For%20Preschoolers.p df 9. Steffani, S., Selvester, P.M.: The Relationship of Drawing, Writing, Literacy, and Math in Kindergarten Children. Reading Horizons 49(2), 125–142 (2009) 10. Grant, A.: How can development of gross and fine motor skills benefit gifted children who struggle with cursive handwriting? (2004), http://www.canterbury.ac.uk/education/professionaldevelopment/projects/gifted-and-talented/docs/alexgrant%2520.doc 11. Marr, D., Cermack, S.A., Cohn, E.S., Henderson, A.: Fine motor activities in Head Start and Kindergarten classrooms. American Journal of Occupational Therapy 57, 550–557 (2003) 12. Feder, K.P., Majnemer, A.: Handwriting development, competency, and intervention. Developmental Medicine and Child Neurology 49(4), 312–317 (2007) 13. Graham, S., Harris, K.R.: Improving the writing performance of young struggling writers: Theoretical and programmatic research from the center on accelerating student learning. The Journal of Special Education 39, 19–33 (2005) 14. Ismail, M. I. :Tulisan Jawi dan Isu – isunya dalam Pendidikan, Malaysia (2001), http://www.scribd.com/doc/2582591/tulisanjawi 15. Seouk, K.K.: Perkembangan tulisan Jawi dalam masyarakat Melayu. Dewan Bahasa dan Pustaka, Kuala Lumpur (1990) 16. Kratz, I.E.U.: Jawi Spelling and Orthography: A Brief Review, Indonesia and the Malay World, vol. 30(86), pp. 21–26. Carfax Publishing (2002) 17. Diah, N.M., Ismail, M., Ahmad, S., Syed Abdullah, S.A.S.: Jawi on Mobile devices with Jawi wordsearch game application. In: IEEE International Conference Science and Social Research (CSSR), pp. 326–329. IEEE Press, New York (2010)


173

18. Hairul Aysa Abdul Halim Shitiq, H.A.A., Moahmud, R.: Using an Edutainment Approach of a Snake and Ladder game for teaching Jawi Script. In: IEEE International Conference on Education and Management Technology, pp. 228–232. IEEE Press, New York (2010) 19. Yazid, M.: Penguasaan Jawi di Kalangan Pelajar-Pelajar Sekolah Menengah di Negeri Kelantan: Satu Kajian Khusus di Daerah Kota Bharu. Universiti Malaya, Kuala Lumpur (1991) 20. Muda, Z.: Faktor-Faktor Psikologi Dan Permasalahan Dalam Pembelajaran Dan Pembelajaran Jawi Di Kalangan Murid Darjah Lima Di Sekolah Kebangsaan Bukit Tunggal, Kuala Terengganu, Malaysia, Kementerian Pendidikan Malaysia: Jabatan Pendidikan Islam Dan Moral (1996) 21. Iwayama, N., Akiyama, K., Tanaka, H., Tamura, H., Ishigaki, K.: Handwriting-based Learning Materials on a TabletPC: A Prototype and its Practical Studies in an Elementary School. In: Ninth International Workshop on Frontiers in Handwriting Recognition, pp. 533–538 (2004) 22. Hermann, A.: Research into Writing and Computers: Viewing the Gestalt. In: Annual Meeting of the Modem Language Association, pp. 20–30 (1987) 23. Bearne, E.: Making Progress in English. Routledge, London (1998) 24. Nakagawa, M.: Toward Ideographic Human Interfaces, Technical Report of IEICE, PRMU2001-240, vol. 101(712), pp. 41-48 (2002) 25. Diego-Cottinelli, B.D., Barros, B.: TRAZO: A Tool to Acquire Handwriting Skills Using. In: Tablet-PC Devices Proceedings of Interaction Design and Children, pp. 278–281 (2010) 26. Barros, B., Conejo, R., de Diego-Cottinelli, A., Garcia-Herreros, J.: Modeling pre-writing tasks to improve graphomotricity processes. In: European Conference on TechnologyEnhanced Learning (2008) 27. Calrson, K., Cunningham, J.L.: Effect of pencil diameter on the Grphomotor skill of preschools. Journal of Early childhood Research Quarterly 5, 279–293 (1990) 28. Cuthbert, S.C., Barras, M.: Developmental Delay Syndromes: Psychometric Testing Before and After Chiropractic Treatment of 157 Children. Journal of Manipulative and Physiological Therapeutics 32(8), 660–669 (2009) 29. Clay, M.: What did I write? Heinemann, Australia (1975) 30. Freeman, E., Sanders, T.: Kindergarten Children’s Emerging Concepts of Writing Functions in the Community. Early Childhood Research Quarterly 4, 331–338 (1989) 31. Levin, I., Bus, A.G.: How is Emergent Writing Based on Drawing? Analyses of Children’s Products and Their Sorting by Children and Mothers. Developmental Psychology 39, 891– 905 (2003) 32. Chen, S., Jing Zhou, J.: Creative writing strategies of young children: Evidence from a study of Chinese emergent writing. Thinking Skills and Creativity, 138–149 (2010) 33. Baghban, M.: Scribbles, Labels, and Stories: The Role of Drawing in the Development of Writing. Young Children 62(1), 20–26 (2007) 34. Diamond, K.E., Gerde, H.K., Powell, D.R.: Development in Early Literacy Skills During the Pre-kindergarten Year in Head Start: Relations Between Growth in Children’s Writing an Understanding of Letters. Early Childhood Research Quarterly 23, 467–478 (2008) 35. Ritchey, K.D.: The Building Blocks of Writing: Learning to Write Letters and Spell Words. Reading and Writing 21, 27–47 (2008) 36. Rémi, C., Frélicot, C., Courtellemon, P.: Automatic Analysis of the Structuring of Children Drawing and Writing. Pattern Recognition 35(5), 1059–1069 (2002)

174


37. McLaughlin, T.F.: An Analysis of Token Reinforcement: A Control Group Comparison with Special Education Youth Employing Measures of Clinical Significance. Child Behavior Therapy 3, 43–51 (1981) 38. Park, C., Weber, K.P., McLaughlin, T.F.: Effects of Fading, Modeling, Prompting, and Direct Instruction on Letter Legibility for Two Preschool Students with Physical Disabilities. Child & Family Behavior Therapy 29(3), 13–21 (2007) 39. Graham, S.: Handwriting and spelling instruction for students with learning disabilities: A review. Learning Disability Quarterly 22, 78–98 (1999) 40. Wright, J.P., Allen, E.G.: The Elementary School Journal 75(7), 430–435 (1975) 41. Beery, K.E.: The Beery-Buktenica developmental test of visual-motor integration. Modern Curriculum Press, New Jersey (1997) 42. Smith, J.: Activities for Fine Motor Skills Development. Teacher Created Resources (2004), http://www.teachercreated.com/products/activities-forfine-motor-skills-development-grade-prek-1-3689 43. Hertzberg, O.E.: A Comparative Study of Different Methods Used in Teaching Be-ginners to Write. Unpublished doctoral dissertation. Columbia University, New York (1926) 44. Hirsch, E., Niedermeyer, F.C.: The Ef-fects of Tracing Prompts and Discrimina-tion Training on Kindergarten Handwriting Handwriting Performance. Journal of Educational Research 67, 81–86 (1973) 45. Wright, C.D., Wright, J.P.: Handwriting: the effectiveness of copying from moving versus still models. Journal of Educational Research 74, 95–98 (1980) 46. Berninger, V.W., Graham, S., Vaughan, K.B., Abbott, R.D., Abbott, S.P., Woodruff Rogan, L.W., et al.: Treatment of handwriting problems in beginning writers: Transfer from handwriting to composition. Journal of Educational Research 89, 652–666 (1997) 47. Kirk, U.: Learning to Copy Letters: A Cognitive Rule-Governed. The Elementary School Journal 81(1), 28–33 (1980) 48. Bara, F., Gentaz, E.: Haptics in teaching Human movement science, handwriting: The role of perceptual and visuo-motor skills. Corrected Proof, pp. xxx–xxx (2011) 49. Vinter, A., Chartrel, E.: Effects of Different Types of Learning on Handwriting Movements in Young Children. Journal Learning and Instruction 20(6), 476–486 (2010) 50. Peterson, C.Q., Nelson, D.L.: Effect of an Occupational Intervention on Printing in Children with Economic Disadvantages. The American Journal of Occupational Therapy Official Publication of the American Occupational Therapy Association 57(2), 152–160 (2003) 51. Graham, S., Harris, K., Fink, B.: Extra handwriting instruction: Preventing writing difficulties right from the start. Teaching Exceptional Children 33, 88–91 (2000) 52. NAEYC position statement: Developmentally appropriate practice in early childhood programs serving children from birth through age eight. Washington DC (1996), http://www.naeyc.org/about/positions/pdf/PSDAP98.PDF 53. Mahshid Hosseini, H.: Dose Your Child have Dificulty with Printing and hand writing (2008), http://www.skilledkids.com 54. Mosey, A.C.: Occupational therapy: Configuration of a profession. Raven Press, New York (1981) 55. McCluskey, A.R.: The Log Handwriting Program Improved Children’s Writing Legibility: A Pretest–Posttest Study Nadine Mackay. Occupational Therapy 64(1), 30–36 (2010) 56. Morgret, K., Weber, N., Woo, L.: Behaviorist Approach To Spelling (2001), http://ldt.stanford.edu/~lwoo/behavior.pdf 57. Stipek, D.: Motivation to Learn: from Theory to Practice. Allyn and Bacon, Massachusetts (1993)


175

58. Nicol, T., Snape, L.: A Computer Based Letter Formation System for Children. In: Proceedings of Conference on Interaction Design and Children, p. 165 (2003) 59. MacKenzie, I.S., Chang, L.: A performance comparison of two handwriting recognizers. Interacting with Computers 11(3), 283–297 (1999) 60. O’Hare, E.A., McTear, M.F.: Speech Technology in the secondary school classroom, an exploratory study. Computers and Education 33, 27–45 (1999) 61. Frankish, C., Morgan, P., Hull, R.: Recognition Accuracy And Usability of Pen-Based Interfaces. In: IEE Colloquium on Interfaces - The Leading Edge (Digest No.1996/126), pp. 7/6 - -7/6 (1996) 62. Read, J.C., MacFarlane, S., Horton, M.: The Usability of Handwriting Recognition for Writing in the Primary Classroom. People and computer XVII-Design for life 4, 135–150 (2005) 63. Saad, M.N., Abd. Razak, A.H., Yasin, A., Aziz, N.S.: Redesigning the user interface of handwriting recognition system for preschool children. International Journal of Computer Science Issues 8(1) (2011); ISBN: 1694–0814 64. Allan, J., Allen, T., Sherkat, N.: Confident assessment of children’s handwritten responses. In: Proceedings of the Eighth International Workshop on Frontiers in Handwriting Recognition, p. 508 (2002) 65. Read. C : CobWeb - a Handwriting Recognition Based Writing Environment for Children, http://academic.research.microsoft.com/Publication/5258217/c obweb-a-handwritingrecognition-based-writing-environmentfor-children

Empirical Performance Evaluation of Raster to Vector Conversion with Different Scanning Resolutions Bilal Abdulrahman T. Al-Douri, Hasan S.M. Al-Khaffaf, and Abdullah Zawawi Talib School of Computer Sciences Universiti Sains Malaysia 11800 USM Penang, Malaysia [email protected], {hasanm,azht}@cs.usm.my

Abstract. Empirical performance evaluation of raster to vector conversion is a means of judging the quality of line detection algorithms. Many factors may affect line detection. This paper aims to study scanning resolution of raster images and its effects on the performance of line detection. Test images with three different scanning resolutions (200, 300, and 400 DPI) are vectorised using available raster to vector conversion software. The Vector Recovery Index scores calculated with reference to the ground truth images and the detected vectors are then obtained. These values are analysed statistically in order to study the effects of different scanning resolutions. From the results, Vextractor is found to be better (on average) compared to VPstudio and Scan2CAD. For all the three resolutions, Vextractor and VPstudio perform better than Scan2CAD. Different scanning resolutions affect the software differently. The performance of Vextractor and VPstudio increases from low resolution to moderate resolution, and then decreases with high resolution. The performance of Scan2CAD decreases with the increase in the resolutions. Keywords: Empirical Performance Evaluation, Raster to Vector Conversion, Vector Recovery Index, Statistical Analysis, Visual Informatics.

1 Introduction Empirical performance evaluation of raster to vector conversion is an important topic in the area of graphics recognition. It is used as a means to judge the quality of line detection algorithms. There are many factors that affect quality of line detection and they may lead to better or poorer line detection rate. Some of these factors have been studied such as noise level, vectorisation software, and cleaning methods [1] and the other factors have yet to be studied such as scanning resolution of raster images. Chhabra and Ihsin [2] used research prototypes and commercial software. The criterion of the performance was EditCost Index. Liu et al. [3] evaluated the performance of several research prototypes. They used different types of noise (Gaussian noise, high frequency, hard pencil, and geometrical noise). VRI was used to measure the performance of the methods. Only solid circular arcs were included in their tests. H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 176–182, 2011. © Springer-Verlag Berlin Heidelberg 2011

Empirical Performance Evaluation of Raster to Vector Conversion

177

Liu [4] also evaluated research prototypes methods, but the test was performed on real images. He used noisy images by adding salt-and-paper noise. Shafait et al. [5] evaluated the performance of vectorisation methods on one research prototype and three commercial software. They have used real scanned images (with no artificial noise). The criterion used was Vectorial Scores. Al-Khaffaf et al. [6] evaluated the performance of vectorisation methods on two research prototypes and three commercials software. They took real scanned images scanned from two old text books. No artificial noise and no degradation model were used in their tests. In order to focus on the detection ability of the software, Dv (detection rate) score was used as performance evaluation criterion. Two tests were performed: a test between different methods and a test between different versions of the same method. This paper aims to study scanning resolution of raster images and its effects on the performance of line detection. The images are vectorised using available raster to vector conversion software and the Vector Recovery Index (VRI) of the detected vectors will be used as the quality criterion. The VRI values will be analysed statistically to study the effects of different scanning resolutions.

2. Image Data and Raster to Vector Conversions Methods Eight images in three resolutions (200, 300, 400 DPI) were used in this study. The images are mechanical engineering drawings that were taken from Arc Segmentation Contest 2011 dataset [7]. The contest was held in conjunction with the IAPR International Workshop on Graphics Recognition 2011 (GREC’11). The focus is on drawings that contain arc segments because these drawings are mathematically complex than straight lines. Also the detection of straight lines is fairly straight forward in the process of performance evaluation and obsolete for research purpose. Many commercial raster to vector methods are available. However, only a number of the available software are useable due to their limitations. Important features that should be available in the software to make it usable in our study include: detection of circles/arcs, saving circles/arcs as circular curves rather than polylines, and saving of the detected vectors in the DXF file format. So, three commercials software (VPstudio(v9) [8], Scan2CAD(v8) [9], and Vextractor(v3.6) [10]) were used since they are readily available and have been used by other researchers [5] [6] (except for Vextractor which has not been reported in the literature).

3 Performance Evaluation VRI was chosen for this study because it is well accepted by other researchers and an implemented tool is readily available. It is calculated as follows [11]: VRI

Dv

1

Fv

(1)

where Dv is the detection rate and Fv is the false alarm rate. The comparison of the detected vector file with the ground truth file was carried out and the VRI score was output using performance evaluation tool (ArcEval2005).

178

B.A.T. Al-Douri, H.S S.M. Al-Khaffaf, and A.Z. Talib

4 Experimental Resu ults and Discussions Each of the eight test imaages was vectorised using the three commercial softw ware (VPstudio, Scan2CAD, an nd Vextractor). From each vectorised image and grouund truth image pair we got one o VRI score. The total number of VRI scores in this experiment is 72 values. The VRI values obtain ned in the evaluation using VPstudio, Scan2CAD and Vextractor software are sho own as bar chart in Figs. 1, 2, and 3, respectively.

0.7 0.6

VRI

0.5 0.4

2000DPI

0.3

3000DPI

0.2

4000DPI

0.1 0

Image Fig. 1. Bar diagram of VRI values and VPstudio

0.6 0.5

VRI

0.4 2000DPi 3000DPI 4000DPI

0.3 0.2 0.1 0

Image Fig. 2. Bar B diagram of VRI values using Scan2CAD


179

0.7 0.6

VRI

0.5 0.4

200DPI

0.3

300DPI

0.2

400DPI

0.1 0

Image Fig. 3. Bar B diagram of VRI values using Vextractor

In this experiment we haave two independent variables (vectorisation software and scanning resolution) and one dependent variable (VRI). Since each image produuces more than one score and there are more than one independent variable, repeaated measure ANOVA is a suittable statistical analysis method to be used in this stuudy. There are three requiremen nts to run this statistical test [1,12], (a) order effect shoould be avoided, (b) the data should be normally distributed, and (c) the Sphericcity condition should not be violated. The first requirement is assured since the original image is always used during the addition of the treatments (i.e. level of a factor). In order to validate the secoond requirement of the test, we performed normality test by relying on Shapiro-Wilk. T The equations below show the null n and alternative hypotheses for Shapiro-Wilk. H0: There is no difference between b the distribution of the dataset and the normal. (2) H1: There is a difference beetween the distribution of the dataset and the normal. (3) If ρ (Sig. in Table 1) is lesss than or equal to .05 then null hypothesis will be rejectted. Otherwise, the null hypotheesis will not be rejected. From the normality test (Tablee 1) it can be seen that ρ for alll the cells are greater than .05. Hence, we fail to reject the null hypothesis and we willl consider the data to be normally distributed. Hence, the second condition for runnin ng repeated measure ANOVA is not violated. The Mauchly’s Test is neeeded to insure that the Sphericity condition is not violaated (satisfying the third condiition of repeated measure ANOVA). Only then, we can proceed with the next step of the analysis. From the Mauchly’s Test (Table 2) it can y condition is not violated (ρ > .05) for the two facttors be seen that the Sphericity (vectorisation software and scanning resolution) and for the interaction between theem. Hence, we proceed with thee analysis (see next paragraphs).

180

B.A.T. Al-Douri, H.S.M. Al-Khaffaf, and A.Z. Talib Table 1. Tests of Normality Vectorisation Software

Vector Recovery Index

Resolution Factor

VPstudio 9

200DPI 300DPI 400DPI 200DPI 300DPI 400DPI 200DPI 300DPI 400DPI

Scan2CAD 8

Vextractor 3.6

Shapiro-Wilk df Sig. 8 .707 8 .773 8 .177 8 .425 8 .527 8 .426 8 .250 8 .494 8 .691

Table 2. Mauchly's Test of Sphericityb

Within Subjects Effect

Mauchly's W

Approx. Chi-Square

df

Sig.

Software

.963

.227

2

.893

Resolution

.515

3.986

2

.136

Software * Resolution

.250

7.519

9

.602

b

Tests the null hypothesis that the error covariance matrix of the orthonormalized transformed dependent variables is proportional to an identity matrix.

The null and alternative hypotheses for the vectorisation software variable are shown below: (4) H0: MeanVPstudio = MeanScan2CAD = MeanVextractor (5) H1: Not all the means are equal The vectorisation software variable is significant, F(2,14) = 8.255, ρ < .05 indicating that the difference in the means for the three vectorisation methods are not the same. Hence we will reject the null hypothesis and accept the alternative hypothesis. Table 3. Pairwise Comparisons

(I) Software VPstudio

Scan2CAD

(J) Software

Mean Difference (I-J)

Scan2CAD Vextractor

.143 -.073

Vextractor

-.217*

Std. Error

Sig.a

.056 .057

.114 .727

.049 .009

Based on estimated marginal means a

. Adjustment for multiple comparisons: Bonferroni. *. The mean difference is significant at the .05 level.


181

By investigating Table 3, it can be seen that the mean difference between Vextractor compared to Scan2CAD is significant. This result can also be seen in Fig 4. In the figure we can see that Vextractor has better overall results compared to both Scan2CAD and VPstudio. Vextractor is also much better than Scan2CAD.

Fig. 4. Performance of the three software based on scanning resolutions of 200, 300, and 400 DPI

5 Conclusions and Future Work The efficiency of three raster to vector software namely VPstudio, Scan2CAD, and Vextractor have been studied and analysed. From the result of the study, it can be concluded that for Vextractor and VPstudio, moderate resolution (300 DPI) has given a good detection rate compared to the other resolutions. The results have also shown that Scan2CAD works well with lower resolution and its performance drops with higher resolutions. Based on the study, it has also been found that Vextractor is efficient and robust with respect to different scanning resolutions. Finally, Vextractor has shown a better overall performance compared to VPstudio and Scan2CAD.

182

B.A.T. Al-Douri, H.S.M. Al-Khaffaf, and A.Z. Talib

References 1. Al-Khaffaf, H.S.M., Talib, A.Z., Abdul Salam, R.: Empirical performance evaluation of raster-to-vector conversion methods: A study on multi level interactions between different factors. IEICE Trans. INF. & Syst. E94-D(6), 1278–1288 (2011) 2. Phillips, I.T., Chhabra, A.K.: A Benchmark for Raster to Vector Conversion Systems. In: Proceedings of SSPR/SPR 1998, pp. 242–251 (1998) 3. Wenyin, L., Zhai, J.: Extended Summary of the Arc Segmentation Contest. In: Blostein, D., Kwon, Y.-B. (eds.) GREC 2001. LNCS, vol. 2390, pp. 343–349. Springer, Heidelberg (2002) 4. Liu, W.: Report of the Arc Segmentation Contest. In: Lladós, J., Kwon, Y.-B. (eds.) GREC 2003. LNCS, vol. 3088, pp. 364–367. Springer, Heidelberg (2004) 5. Shafait, F., Keysers, D., Breuel, T.M.: GREC 2007 Arc Segmentation Contest: Evaluation of Four Participating Algorithms. In: Liu, W., Lladós, J., Ogier, J.-M. (eds.) GREC 2007. LNCS, vol. 5046, pp. 310–320. Springer, Heidelberg (2008) 6. Al-Khaffaf, H.S.M., Talib, A.Z., Osman, M.A., Wong, P.L.: GREC’09 Arc Segmentation Contest: Performance Evaluation on Old Documents. In: Ogier, J.-M., Liu, W., Lladós, J. (eds.) GREC 2009. LNCS, vol. 6020, pp. 251–259. Springer, Heidelberg (2010) 7. Arc Segmentation Contest (2011), http://www.cs.usm.my/ArcSeg2011/downloads.html 8. VPstudio, http://www.softelec.com 9. Scan2CAD, http://www.scan2cad.com 10. Vextractor, http://www.vextractor.com 11. Wenyin, L., Dori, D.: A protocol for performance evaluation of line detection algorithms. Machine Vision and Applications 9(5), 240–250 (1997) 12. Roberts, M.J., Russo, R.: A Student’s Guide to Analysis of Variance. Routledge, Taylor Francis Ltd., United Kingdom (1999)

Visualizing the Construction of Incremental Disorder Trie Itemset Data Structure (DOSTrieIT) for Frequent Pattern Tree (FP-Tree) Zailani Abdullah1, Tutut Herawan2, and Mustafa Mat Deris3 1

Department of Computer Science, Universiti Malaysia Terengganu Faculty of Computer System and Software Engineering, Universiti Malaysia Pahang 3 Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia [email protected], [email protected], [email protected] 2

Abstract. In data mining, visual representation can help in enhancing the ability of analyzing and understanding the techniques, patterns and their integration. Recently, there are varieties of visualizers have been proposed in marketplace and knowledge discovery communities. However, the detail visualization processes for constructing any incremental tree data structure from its original dataset are rarely presented. Essentially, graphic illustrations of the complex processes are easier to be understood as compared to the complex computer pseudocode. Therefore, this paper explains the visualization process of constructing our incremental Disorder Support Trie Itemset (DOSTrieIT) data structure from the flat-file dataset. DOSTrieIT can be used later as a compressed source of information for building Frequent Pattern Tree (FP-Tree). To ensure understandability, an appropriate dataset and its processes are graphically presented and details explained. Keywords: Visual Representation, Data Structure, Data Mining, Visual Informatics.

1 Introduction Data mining, also popularly known as Knowledge Discovery in Database (KDD), defines as “the nontrivial extraction of implicit, previously unknown, and potentially useful of data” [1] and “the science of extracting useful information from large dataset or databases” [2]. Often, these two terms are used exchangeable but indeed data mining is a part and core of KDD process. Hence, KDD is more broad process of finding knowledge from data. It usually emphasizes on the high-level of application and applies with the particular data mining techniques. The other synonym terms that are referring to data mining are data dredging, knowledge extraction and pattern discovery. Frequent pattern mining is one of the most important topics in data mining. For the past decade, it has invited many research interests [3-6] due to its good track records in handling several domain applications. The main aim at frequent pattern mining is H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 183–195, 2011. © Springer-Verlag Berlin Heidelberg 2011

184

Z. Abdullah, T. Herawan, and M.M. Deris

to search for interesting relationships among set of items (itemset) in the database repositories. It was first introduced by Agrawal et al. [7] and still playing an important role in data mining. A set of item in pattern (or transaction) is also defined as an itemset. The itemset is said to be frequent, if it appears equal or exceed the predefined minimum supports. Nowadays, more than hundreds of research papers have been published including new or modification of the algorithms. Generally, in term of frequent pattern mining, the result can be obtained from two types of wellknown algorithms; Apriori-based [7] or Frequent Pattern Tree (FP-Tree) [3] based algorithms. FP-Tree data structure is more dominant than Apriori since it is more compact in storing the vast number of transactions. However, from the past researches [8-11], the construction of FP-Tree is inefficient if it relies on the original dataset. One of the promising areas that can be applied in frequent pattern mining is visualization. Visualization is an approach of using the recent technology by applying certain principles in interpreting the data. In term of data mining, it can be classified into three categories [12]. First, apply the visualization techniques that are independence from data mining algorithm. Second, use the visualization to represent pattern and mining result from mining process graphically. Third, integrate both mining and visualization algorithm so that the immediate steps of a data mining algorithm can be visualized. Presently, several variations of visualizations in this area have been proposed. Yang [13] suggested a system to visualize both association rules and frequent patterns. In this system, domain items are sorted according to their frequencies and are evenly distributed along each vertical axis. Munzner [14] designed Powerset Viewer to visualize frequent patterns in the context of the powerset universe. Here, all patterns are grouped together based on cardinality. Leung [15] proposed Frequent Itemset Visualizer to present the frequent patterns by a polyline. A year later, Leung [16] again proposed FpViz to visualize the frequent patterns. At this time, the pattern is presented according to an orthogonally laid out node-link diagram instead of using the polyline diagram. Undeniable, the visual data mining is playing an important role to give the new insights of information and related processes rather than the textual form. However, the details visualization processes that are directly involved in constructing the incremental tree data structure of frequent pattern mining are rarely became a focal point. Therefore, in this paper we visualize the detail processes of constructing our proposed DOSTrieIT data structure in an attempt to replace the original flat-file dataset. Indeed, DOSTrieIT is an incremental tree data structure and it can be used for constructing FP-Tree in the future. Here, we graphically depict a step by step and explain in details the processes involved during the transformation. A suitable dataset that has been widely used in other research papers in frequent pattern mining is also employed. In summary, there are two main contributions from this work. First, we propose a novel and incremental tree data structure; DOSTrieIT that can replace the dependency of using the original transactional database to construct FP-Tree. Second, we present the detail processes of constructing DOSTrieIT from its database using visualization (graphically illustration) rather than the conventional approach (eg. textual form, pseudocode, etc.).

Visualizing the Construction of Incremental Disorder Trie Itemset Data Structure

185

The rest of the paper is organized as follows. Section 2 describes the preliminaries of association rules. Section 3 discusses the proposed method. This is followed by result and discussion in section 4. Finally, conclusion and future direction are reported in section 5.

2 Association Rules

{

}

Throughout this section the set I = i1 , i 2 ,  , i A , for A > 0 refers to the set of literals

{

}

called set of items and the set D = t1 , t 2 ,, t U , for U > 0 refers to the data set of

{

}

transactions, where each transaction t ∈ D is a list of distinct items t = i1 , i2 , , i M , 1 ≤ M ≤ A and each transaction can be identified by a distinct identifier TID. A set X ⊆ I is called an itemset. An itemset with k-items is called a k-itemset. The support of an itemset X ⊆ I , denoted supp( X ) is defined as a number of transactions contain X. Let X , Y ⊆ I be itemset. An association rule between sets X and Y is an implication of the form X  Y , where X  Y = φ . The sets X and Y are called antecedent and consequent, respectively. The support for an association rule X  Y , denoted supp( X  Y ) , is defined as a number of transactions in D contain X  Y . The confidence for an association rule X  Y , denoted conf ( X  Y ) is defined as a ratio of the numbers of transactions in D contain X  Y to the number of transactions in D contain X. Thus conf ( X  Y ) = supp( X  Y ) / supp ( X ) . An itemset X is called frequent item if supp ( X ) > β , where β is the minimum support. The set of frequent item will be denoted as Frequent Items and Frequent Item = {X ⊂ I | supp ( X ) > β } .

3

Proposed Model

3.1 DOSTrieIT Rudimentary

In this section, DOSTrieIT data structure is details explained. In order to easily comprehend the whole process, some required definitions together with a sample transactional data of the proposed model are presented. Definition 1. Disorder Support Trie Itemset (DOSTrieIT) is defined as a complete tree data structure in canonical order of itemsets (or in any form and it is free from any support order). The order of itemset is not based on the support descending order. DOSTrieIT contains n-levels of tree nodes (items) and their support. Moreover, DOSTrieIT is constructed in online manner and for the purpose of incremental pattern mining. Example 2. Let T = {{1,2,5}, {2,4}, {2,3}, {1,2,4}, {1,3}, {2,3,6}, {1,3}, {1,2,3,5}, {1,2,3}} . A step by step to construct DOSTrieIT is explained in the next section. Graphically, an item

186


is represented as a node and its support is appeared nearby to the respective node. A complete structure of DOSTrieIT is shown as follows in Fig. 1. Definition 3. Single Item without Extension (SIWE) is a prefix path in the tree that contains only one item or node. SIWE is constructed upon receiving a new transaction and as a mechanism for fast searching of single item support. It will be employed during tree transformation process but it will not be physically transferred into the others tree. Example 4. From Example 2, the transactions have 6 unique items and it is not sorted in any order. In Fig. 1, SIWE for DOSTrieIT i.e

SIWE = {2,1,3,4,5,6}

Fig. 1. DOSTrieIT and SIWE Path arranged in descending order

Definition 5. Disorder-Support Itemset Path (DSIP) is a selected prefix path from DOSTrieIT. The selected path must be linked directly to the root and it branches are ignored temporarily. Example 6. From Example 2, the longest prefix path that originated from the root is {1,2,3,5} . Thus, Disorder-Support Itemset Path for the prefix path, i.e.,

DSIP(1,2,3,5) = {1,2,3,5} Definition 7. Parent Item (ParentItem) is an item(s) in DSIP that has an extension or others prefix paths. In DSIP, it may or may not contain the ParentItem. During transformation process to FP-Tree, DSIP will be removed from DOSTrieIT and its ParentItem will be replicated. Example 8. Let used again the example of DSIP(1,2,3,5) . The ParentItem is item 1

and 2. This item is shared by another three prefix paths, (1,3) , (2,4) and (2,5) . While

removing DSIP(1,2,3,5) from DOSTrieIT during transformation process, its ParentItem 1 and 2 is still maintained.

Definition 9. Child Item (ChildItem) is prefix paths that rooted from the ParentItem in DSIP. The ChildItem that is also appeared as part of DSIP is called direct ChildItem. During transforming DOSTrieIT to FP-Tree, the direct ChildItem will be removed from DOSTrieIT but the others (indirect) ChildItem will be remained.


187

Example 10. For DSIP(1,2,3,5) , the ChildItem are 3 (1,3) , 4 (2,4) and 5 (2,5) . During transformation to FP-Tree, all these ChildItem and their ParentItem will be maintained in DOSTrieIT. Definition 11. Items and Support (ItemSupp) is a list of items and their support. The data in ItemSupp is updated instantly upon receiving a new line of transaction. The items in ItemSupp are flexible and not sorted in support descending order. An itemset X is called frequent item if supp ( X ) > β , where β is the minimum support threshold. Definition 12. Frequent Itemset (FRIT) is a set of frequent items that fulfils the minimum support threshold (β ) and sorted in support descending order. FRIT is generated by extracting the information from SIWE. Example 13. For simplicity, let fixed the minimum support threshold (β ) to 20%.

Thus, itemset in FRIT is {2,1,3,4,5} . Item 6 is excluded since its support value is less than the minimum support threshold. Definition 14. Ordered-Support Itemset Path (OSIP) is a prefix path and it itemset is sorted based on the itemset in FRIT. OSIP is produced by applying the intersection operation between FRIT and DSIP. The itemset arrangement in OSIP is following the arrangement as shown in FRIT. Example 15. Using the similar example, the intersection between FRIT(2,1,3,4,5)

and DSIP(1,2,3,5) is OSIP(2,1,3,5) . The itemset arrangement in OSIP is very important during constructing FP-Tree structure. Definition 16. Minimum Itemset Support (MISupp) is a uniform support value determined from the minimum items supports as appeared in OSIP. Once the MISupp is identified, all items in OSIP will be employed with this value. The support of replicated ParentItem (if any) of DSIP will be also deducted by MISupp. The support of indirect ChildItem will be unchanged. Example 17. Let used the example of DSIP(1,2,3,5) and OSIP(2,1,3,5) . MISupp for

OSIP(2,1,3,5) is 1. The replicated ParentItem of DSIP(1,2,3,5) is item 2 with the original support of 3. Therefore, the latest support of the replicated ParentItem 2 in DOSTrieIT is 2 (3 – MISupp). 3.2 DOSTrieIT Algorithm

This section describes a step by step of constructing DOSTrieIT and its algorithm. DOSTrieIT compresses the transactional database into tree data structure. It is online construction of tree data structure and constructed by only a single scanning of transactional database. Illustration of tee construction is based on transactional data as presented in Example 2. The fundamental steps of constructing DOSTrieIT are as follows: Step 1, the algorithm starts by scanning the content of the current DOSTrieIT. If existed, the existing data will be transferred into SortedList class. SortedList is a collection of key/value pairs that are sorted by the keys. The data in SortedList is a list

188


of all prefixes paths and their support from the previous transactions. Otherwise, the structure of DOSTrieIT is assigned with null. Step 2, a transaction is a main input in the algorithm. A line of transaction which consists of items is read and transformed into String format. The items in the String format (horizontal) are then converted into vertical ArrayList. ArrayList in a single dimension array. The specification of array size is not required for this type of array and it is auto-extent by itself. Step 3, hash data structure is employed to transform the items (keys) from the ArrayList into the index of an array element. All items in a line of transaction are converted into another hash data structure. At this step, a technique based on the intersection operation between two hashes data structure is employed. This technique helps in reducing the computational times rather than typical brute-force technique. Step 4, if there is no intersection operation taken place between the line of transaction and the existing prefix paths hashes; a new prefix path linked to the root is produced. The support of all items (nodes) in the prefix path is the same and equal to MISupp. In addition, a list of items and their support is created. Once created, the list will be regularly and consistently updated. Step 5, if intersection operation is occurred between the line of transaction and the prefix paths hashes, and if the total number of items of intersection is the same, prefix path will not be produced. The support of involved items (nodes) in the prefix path is increased by summation with the current MISupp. However, if the total number of items of intersection is different, the support of similar intersected items is increased by summation with current MISupp and for dissimilar one, a new prefix path is created and its support is equal to current MISupp.

The final output from the processes is DOSTrieIT. The details implementation is shown in the following algorithm. DOSTrieIT Algorithm 1 : Input : Transaction 2 : Output : DOSTrieIT 3 : if DOSTrieIT ≠ null do 4: Read DOSTrieIT 5: PrefixPath∈ DOSTrieIT 6: ItemSupp∈ DOSTrieIT 7 : else 8: Create DOSTrieIT ← null 9: PrefixPath ← null 10 : end if 11: Read Transaction 12: for DTpath ∈ DOSTrieIT(path) j

13: 14:

for DTitem ∈ DTpath (item)

(

i

j

)

if Linepattern ∈ Transaction (pattern) ≠ DTpath do j


15:

((

)

a

i

Insert PrefixPath from root PrefixPathitem ,supp ← Lineitem ,supp

18:

ItemSuppitem ,supp ← Lineitem ,supp

22:

)

for Lineitem ∈ Transaction(item) ∉ DTitem do

16: 17: 19: 20: 21:

189

a

a

a

a

a

a

a

a

end for loop end if for Lineitem ∈ Transaction (item) ∈ DTitem do

((

(

)

a

i

if Lineitem ∈ Transaction(item) = DTitem a

i

) ) do

23:

PrefixPathsupp ← PrefixPathsupp + Linesupp

24:

ItemSuppsupp ← ItemSuppsupp + Linesupp

25: 26:

end if if Lineitem ∈ Transaction(item) ≠ DTitem do

i

i

i

(

a

i

a

a

i

)

27:

Insert PrefixPath from PrefixPathitemi-1

28:

PrefixPathitem ,supp ← Lineitem ,supp

29:

ItemSuppitem ,supp ← Lineitem ,supp

a

a

a

a

a

a

a

a

end if 30: 31: end for loop 32: end for loop 33: end for

4 Result and Discussion The discussion in this section is based on the transactions as shown in Table 2. In general, the items shown in the transaction of T1 to T9 will be read, divided and converted into vertical format before transferring to DOSTrieIT. The visualization as shown from Fig 2 to Fig. 10 together with its elaboration will help in understanding precisely the step by step of constructing DOSTrieIT. Table 2. Transactions for constructing DOSTrieIT ID T1 T2 T3 T4 T5 T6 T7 T8 T9

Transactions 125 24 23 124 13 236 13 1235 123

190


Fig. 2. DOSTrieIT Construction for T1

From Fig. 2 and start from the empty tree, the first transaction (T1: 1 2 3) will be arranged first in numerical sorting and then inserted at the root. Three Single Item without Extension (SIWE) which is (1, 2, 5) will be also directly inserted from the root. At this stage, all nodes in the tree have an equal support count, which is 1.


For the second transaction (T2: 2 4) as in Fig. 3, only item 2, is matched with the existing item in SIWE. Thus, the support count of item 2 in SIWE is updated into 2. A new SIWE for item 4 with the support count of 1 will be created. For the first item in transaction T2, it is not matched to any first level items in the tree (SIWE is ignored). Therefore, a new prefix path will be created which is originated from the root and the support count for each item is 1.


From Fig 4, only item 2 in the third transaction (T3: 2 3) matched with the existing item in SIWE. Therefore, the support count of item 2 in SIWE is now updated into 3. A new SIWE for item 3 with the support count of 1 will be created. For the first item


191

in transaction T3, it is matched with item 2 in the first level items in the tree (SIWE is ignored). Therefore, a new prefix path which is originated from the item 2 will be created and the support count for item 2 and item 3 will become 2 and 1, respectively.


For the fourth transaction (T4: 1 2 4) as in Fig. 5, item 1 and 2 are matched with the existing item in SIWE. Thus, the support count for item 1 and 2 in SIWE are updated into 2 and 4, respectively. For the first item in transaction T4, it is matched with the existing first level item. For the next item in transaction T4, it is also matched to the consequence item in the same prefix path. Therefore, the support count for item 1 and 2 in that prefix path are updated into 2 and 2, respectively. Since for the next item (4) in transaction T4 is not matched to the consequence item in the prefix path, thus a new prefix path which is originated from item 2 will be created. The support count for item 4 in this prefix path is 1.


From Fig. 6, as in the fifth transaction (T5: 1 3), item 1 and 3 are matched with the existing item in SIWE. Thus, the support count for item 1 and 3 in SIWE are updated into 3 and 2, respectively. For the first item in transaction T5, it is matched with the existing first level item. For the next item in transaction T5, it is unmatched to the consequence item in the same prefix path. Thus, the support count for item 1 in that prefix path is updated into 3. Since for the next item (3) in transaction T5 is not matched to the consequence item in the prefix path, thus a new prefix path which is originated from item 1 will be created. The support count for item 3 in this prefix path is 1.

192



At Fig. 7, item 2 and 3 in the sixth transaction (T6: 2 3 6), are matched with the existing item in SIWE. Thus, the support count for item 2 and 3 in SIWE are updated into 5 and 2, respectively. For the first item in transaction T6, it is matched with the existing first level item. For the next item in the transaction, it is again matched to the consequence item in the same prefix path. Therefore, the support count for item 2 and 3 in that prefix path are now updated into 3 and 2, respectively. Since for the next item (6) in transaction T6 is not matched to the consequence item in the prefix path, thus a new prefix path which is originated from item 3 will be created. The support count for item 6 in this prefix path is 1.


For the seventh transaction (T7: 1 3) as in Fig. 8, item 1 and 3 are matched with the existing item in SIWE. Thus, the support count for item 1 and 3 in SIWE are updated into 5 and 3, respectively. For the first item in transaction T7, it is matched with the existing first level item. For the next item in transaction T7, it is also matched to the consequence item in the same prefix path. Thus, the support count for item 1 and 3 in that prefix path is updated into 4 and 2, respectively.



193

As shown in Fig. 9, item 1, 2, 3 and 5 in the eighth transaction (T8: 1 2 3 5) are matched with the existing item in SIWE. Thus, the support count for item 1, 2, 3 and 5 in SIWE are updated into 6, 6, 4 and 2, respectively. For the first item in transaction T8, it is matched with the existing first level item. For the next item in transaction T8, it is also matched to the consequence item in the same prefix path. Therefore, the support count for item 1 and 2 in that prefix path are updated into 5 and 3, respectively. Since for the next item (3) in transaction T8 is not matched to the consequence item in the prefix path, thus a new prefix path which is originated from item 2 will be created. The support count for both item 3 and 5 in this prefix path is 1. For the eighth transaction (T9: 1 2 3) as presented in Fig. 10, item 1, 2 and 3 are matched with the existing item in SIWE. Thus, the support count for item 1, 2 and 3 in SIWE are updated into 7, 7 and 5, respectively. For the first item in transaction T9, it is matched with the existing first level item. For the next item in transaction T9, it is also matched to the consequence item in the same prefix path. Moreover, the last item in the transaction is also matched to the consequence item in the same prefix path. Therefore, the support count for item 1, 2 and 3 in that prefix path are now updated into 6, 4 and 2, respectively. Fig. 11 depicts DOSTrieIT data structure with SIWE that are sorted in support descending order.


Fig. 11. Illustration of complete constructing DOSTrieIT based on frequent items

5 Conclusion Visual representation in data mining is playing an important role since it can comprehend the complex techniques, patterns and their integration. For the past

194


decades, one of the popular research areas in data mining is frequent pattern mining. A real challenge is how to construct FP-Tree without depending on its original flatfile dataset. Even though, some of them had suggested a new tree data structure, but the details explanation is not graphically presented. Essentially, graphic presentations of the complex processes are easier to be understood as compared to the typical computer algorithm or mathematically notations. Therefore, here we propose a new incremental tree data structure to represent all itemsets called Disordered Support Trie Itemset (DOSTrieIT). To ensure the better understanding of constructing DOSTrieIT, all the detail processes and appropriate sample data are visually illustrated. In a near future, we are going to evaluate and compare the computational performance of transferring the data from DOSTrieIT against the original dataset or other tree data structures into FP-Tree. Acknowledgement. The authors would like to thanks Universiti Malaysia Pahang for supporting this work.

References 1. Frawley, W., Piatetsky-Shapiro, G., Matheus, C.: Knowledge Discovery in Databases: An Overview. AI Magazine, 213–228 (Fall 1992) 2. Hand, D., Mannila, H., Smyth, P.: Principles of Data Mining. MIT Press, Cambridge (2001) 3. Han, J., Pei, H., Yin, Y.: Mining Frequent Patterns without Candidate Generation. In: The Proceedings of the 2000 ACM SIGMOD, pp. 1–12 (2000) 4. Han, J., Pei, J.: Mining Frequent Pattern without Candidate Itemset Generation: A Frequent Pattern Tree Approach. Data Mining and Knowledge Discovery (8), 53–87 (2004) 5. Zaki, M.J., Parthasarathy, S., Ogihara, M., Li, W.: New algorithms for fast discovery of association rules. In: The Proceedings of the Third International Conference on Knowledge Discovery and Data Mining, pp. 283–286. AAAI Press (1997) 6. Zheng, Z., Kohavi, R., Mason, L.: Real World Performance of Association Rule Algorithms. In: The Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 401–406. ACM Press (2001) 7. Agrawal, R., Imielinski, T., Swami, A.: Database Mining: A Performance Perspective. IEEE Transactions on Knowledge and Data Engineering 5(6), 914–925 (1993) 8. Cheung, W., Zaïane, O.R.: Incremental Mining of Frequent Patterns without Candidate Generation of Support Constraint. In: Proceeding the 7th International Database Engineering and Applications Symposium, IDEAS (2003) 9. Hong, T.-P., Lin, J.-W., We, Y.-L.: Incrementally Fast Updated Frequent Pattern Trees. International Journal of Expert Systems with Applications 34(4), 2424–2435 (2008) 10. Tanbeer, S.K., Ahmed, C.F., Jeong, B.S., Lee, Y.K.: Efficient Single-Pass Frequent Pattern Mining Using a Prefix-Tree. Information Science 279, 559–583 (2009) 11. Totad, S.G., Geeta, R.B., Reddy, P.P.: Batch Processing for Incremental FP-Tree Construction. International Journal of Computer Applications 5(5), 28–32 (2010) 12. Ankerst, M.: Visual Data Mining with pixel-oriented visualization techniques. In: Proceeding of ACM Special Interest Group on Knowledge Discovery and Data Mining (SIGKDD) Workshop on Visual Data Mining San Francisco (2001)


195

13. Yang, X., Asur, S., Parthasarathy, S., Mehta, S.: A visual-analytic toolkit for dynamic interaction graphs. In: The Proceedings of Knowledge Discovery of Database 2008, pp. 1016–1024 (2008) 14. Munzner, T., Kong, Q., Ng, R.T., Lee, J., Klawe, J., Radulovic, D., Leung, C.K.-S.: Visual mining of power sets with large alphabets, Technical report UBC CS TR-2005-25, Department of Computer Science, The University of British Columbia, Vancouver, BC, Canada (2005) 15. Leung, C.K.-S., Irani, P.P., Carmichael, C.L.: FIsViz: a Frequent Itemset Visualizer. In: Washio, T., Suzuki, E., Ting, K.M., Inokuchi, A. (eds.) PAKDD 2008. LNCS (LNAI), vol. 5012, pp. 644–652. Springer, Heidelberg (2008) 16. Leung, C.K.-S., Carmichael, C.L.: FpViz: A visualizer for frequent pattern mining. In: The Proceedings of VAKD 2009, pp. 30–39 (2009)

The Gradient of the Maximal Curvature Estimation for Crest Lines Extraction Pan Zheng1, Bahari Belaton1, Iman Yi Liao1, and Zainul Ahmad Rajion2 1

School of Computer Sciences, Universiti Sains Malaysia, Minden, Penang 11800 Malaysia 2 School of Dental Sciences Universiti Sains Malaysia, Kubang Kerian, Kelantan 16150 Malaysia [email protected], {bahari,iman}@cs.usm.my, [email protected]

Abstract. Crest lines are one of the many types of feature lines that illustrate the prominent characteristics of an object’s surface. In this study, we investigate one vital component to extract crest lines, the gradient of the maximal curvature. Most of geometry properties required to calculate crest lines can be obtained from the volume data during the process of the surface construction using implicit surface polygonizer. Nevertheless the gradient of the maximal curvature cannot be obtained due to the nature of the surface construction algorithm. Hence we proposed three weight function based methods in accordance with the knowledge of the surface mesh. We implemented our methods and conducted both qualitative and quantitative analysis on our methods to find the most appropriate method. We also put forward a simple filtering mechanism as a post-processing procedure to enhance the accuracy of the crest lines, which is addressed at the end of this study. Keywords: Crest lines, Feature Lines, Gradient, Maximal Curvature, Weight Function, Visual Informatics.

1

Introduction

Crest lines are one of the many types of feature lines that illustrate the prominent characteristics of an object’s surface. Various definitions of crest lines can be found from different sources in different domains of research. In this study, crest line is defined as one type of feature lines that illustrate the ridges and ravines of a surface. A ridge is the elevation with a narrow, elongated crest. Geometrically it can be understood as a constricted protrudent portion or line formed by the juncture of two slopping planes. A ravine is basically a valley which is a hollow resembling with predominant extent in one direction, by considering surface as a landscape. In a similar research context [1, 2], crest lines are defined as the locus of points on a surface whose largest curvature is locally maximal in the associated principal direction. Mathematically it is formed by connecting the extremal points on the surface. H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 196–205, 2011. © Springer-Verlag Berlin Heidelberg 2011

The Gradient of the Maximal Curvature Estimation for Crest Lines Extraction

197

Practical application of crest lines can be found in geo-science such as terrain synthesis [11]. In biometric field, crest lines are used for identity verification with palm skin [12], finger print recognition [13] and face detection [14] and recovery [15]. Vessel extraction [16], bone illustration [17] and cerebral atrophy analysis are some application of crest lines addressed in medical fields. Some example applications of crest line in the domain of illustrative rendering and None Photorealistic Rendering (NPR) are mentioned in [8, 9].Existing methods for crest lines extraction are typically categorised based on the type of data they used, i.e. volumetric data and mesh data (triangle patches). In the later case, a mesh representing the object is formed first, says using iso-surface technique or derived from laser scanning devices. Crest lines and other feature lines are then extracted from this mesh. In volumetric data case, crest lines are identified and extracted directly from the raw data. Some would argue that this approach provides more accurate results. Not surprising though, most of crest lines or more generally feature lines extraction methods are mesh-based. For instance, Stylianou et. al. [3] extract crest lines from a 3D triangulated mesh using a bivariate polynomial to approximate the surface locally and calculate the principal curvature on every vertex on the mesh and developed a tracing algorithm. Hildebrndt et. al. [4] perform a crest lines extraction on a smooth piecewise linear mesh by utilizing discrete differential operators. Yoshizawa et. al. [5] provide a local polynomial fitting scheme for generating crest lines on dense triangle mesh and later some enhancements are added [6]. One common characteristic of mesh-based methods is that they solely depend on the quality of the meshes, which are the smoothness, dense, hole-free and etc. In practice however, noise/hole free, well shaped and smooth meshes are challenging qualities to fulfil or even to be guaranteed, especially in domains where the complexities of the objects are apparent, like in medical and engineering fields. These methods however, produce reasonable results and they are able to achieve these results at a relatively good speed of execution. In contrast, there has been little research works in the domain of crest lines extraction from volumetric data. One notable research in this domain is the work carried by Thirion et. al. [1]. They proposed a marching line algorithm, to trace and extract crest lines directly from the raw volumetric data. The research illustrates the application of their algorithm to extract crest lines from human skulls. While the results they produced appeared to be in good quality. The formulas with high order derivatives are quite sensitive to noise or regular pattern in data, especially when the extraction procedures are conducted directly on the volumetric data. In the process of surface extraction, we use the implicit surface polygonizer [7, 19] to extract the surface. There are several advantages to use implicit polygonizer than conventional exhaustive volume traversal algorithm i.e. Marching Cubes [20]. Firstly, the complexity of implicit polygonizer is approximated to O(n2),whereas exhaustive algorithms usually have at least O(n3). Secondly, the higher order derivatives of the implicit function can be directly acquired based on the information of the volume data with finite difference methods. Thirdly, the size of the traversal cubes of the implicit polygonizer can be easily controlled; hence we can extract surfaces in different resolutions. On account of the nature of the implementation of implicit surface polygonizer, we are not able to procure all the geometry properties required to assemble crest lines of a

198

P. Zheng et al.

surface based on the information from volume data itself. The very element needed is the gradient of the maximal curvature. In this research, we study this vital component that concerns crest lines extraction, and propose three weight sum function based methods by exploring the information based on mesh and experiment these methods accordingly. Both qualitative and quantitative analyses are carried out. A simple filtering mechanism to eliminate the less important crest lines is provided at the end of the paper.

2

Theoretical Background

Crest lines of a surface is formed by connecting the zero crossing of extremal value e of each vertices of a triangle patch of an isosurface. Figure 1 gives a general picture of crest lines.

Fig. 1. Crest lines and related geometry properties

The extremal value can be obtained by extremality function, ·

.

(1)

) in maximal principal which is the gradient of the maximal curvature ( k direction( t ). k is a vector that can be written as ⁄∂x , ∂k ⁄∂y , ∂k ⁄∂z . It means the rate of changes of maximal ∂k principal curvature in x, y and z directions, herefrom we call it the gradient of the is shown below, maximal curvature, whereby the maximal principal direction t t

α Δ

S

√Δ f K

f ,α

√Δ f

f ,α

√Δ f

f

.

(2) (3)

where K and S are the Gaussian and mean curvatures. f is the implicit surface function. The subscript of f denotes the derivative in respective direction in x, y and z. The formulas for calculating S, K, α , α , α and also the maximal curvature (denoted as k) are presented in Thirion’s paper [18], which contain high order


199

derivatives. With respect of the properties of the implicit function, the high order derivatives can be resolved with the instrument of finite difference methods. The calculation of these geometry elements and the error analysis is addressed in [21].

3

Methodology

By considering different facts and their influences in a calculation, weight function can be used to estimate derivatives in a discrete setting. After extracting the surface, the surface mesh provides us several geometry components which can be utilized to approximate the gradient of the maximal curvature. We proposed and investigated three weight function based methods for the estimation of the gradient of the maximal curvature. The components we contemplate are angles, distances and “intensity differences” [22]. The angle is defined as the corner formed by the vectors pointing from origin to the target vertex and the vector pointing from origin to the respective neighbour. The distance is defined as the interspace reach between the target vertex and the respective neighbour. The neighbourhood are simply the first ring adjacent vertices. The definitions are depicted in figure 2 below. “intensity difference” is addressed in a later context of this paper. The reason to choose these components is based on the important concept of geometry invariance. These components are geometry invariants under rigid transformation, which means that these components preserve their intrinsic geometry qualities and provide the stability we desired. The three components are mutually paired in our weight functions.

Fig. 2. First-ring neighbourhood of the target vertex, and the illustration of the angle and the distance

The general weight sum function to estimate the gradient of can be defined as

respect to x direction

200

P. Zheng et al. ∑

.

∑

|

. |

.

(4)

is the maximal curvature of the target vertex. where . is the weight function. is the maximal curvatures of the first-ring neighbourhood. is the total number of the neighbours. The first method of the weight function to approximate the gradient of the maximal curvature considers the angles and the distances. Ergo it is expounded as .

·

1

. .

. .

.

(5)

is the weight control parameter. We believe that the “distance” ( . . ), is inverse proportional to the influence of the neighbour to the target vertex, so is the “angle” ( . . ). Hence, the ctan of the “angle” is direction proportional to the influence ki to k0. So we have . .

.

.

(6)

, and are the coordinates of the target vertex and , and are the coordinates of the respective neighbour. To calculate the weight function of the distance, we can just employ the general formula for Euclidian distance calculation in 3D space. Owing to the fact that the distance is abundantly miniature, we normalized the distance described below .

∑

(7)

⁄ ⁄ . The “distance” function is The similar calculation applies to and the same. The “angle” function needs to be altered corresponding to desired directions. The second method we take the “angle” and the “intensity difference” into the consideration. The “angle” function is defined as same as it in the first method. The new component is the “intensity difference” [22].The idea is based on the fact that the gradient of the maximal curvature value at a vertex is influenced mainly by the vertices which are spatially close (first ring neighbourhood) and have similar “intensity”. Intensity difference is defined to be the projection of the normal on the surface normal . The intensity difference of difference vector two vertices can be calculate using ·

.

(8)

where the is the normal vector of target vertex and the is the normal vector of the respective neighbour. The third method is reckoned with the “distance” function and the “intensity difference” function. The above methods do not only apply to our implementation of implicit polygonizer surfaces. The methods are generalized to estimate the gradient of the maximal curvature of the surface obtained using other mesh surfaces as well, so long as the maximal curvature of each vertex of the mesh surface is known. The methods can also be potentially used for other derivative related geometry properties of a mesh surface.


4

201

Experiment and Discussion

We implement all three methods on a synthesised implicit surface function with OpenGL and C. there are some crest lines extraction results with our methods shown below.

Fig. 3. Crest lines extraction on an example implicit surface

The first row is the crest lines result extracted based on the gradient of the maximal curvature estimated using the “angle” and the “distance” weight sum function. The second row shows the “angle” and “intensity difference” result. The third row is the “distance” and “intensity difference” results. All three methods are tested with three different control values which is the lambda. It is assigned to be 0.3, 0.5 and 0.7, which are shown in column-wise. As we can see from these qualitative results, the “angle” plays a vital role to ensure the accuracy of the crest lines identified in predominant surface feature; whereas in the third method, some surface features are extracted, but not desired. We also found that the control parameter in the weight function does not affect the visual results much, so we did a quantitative analysis on the errors of different method with different control parameter. We focus our analysis on the first two prime methods which provide more righteous patterns of crest lines. There are two methods and three control parameters, so totally six cases. For the comparison purpose, the surface function we experiment is a mathematical function surface; therefore the ground truth can be established with accurate calculations. We collect the ground truth

202

P. Zheng et al.

data with MATLAB and compare the actual values with our proposed methods for quantitative analysis. Figure 4 presents the comparisons.

Fig. 4. The Average MSE of the gradient of the maximal curvature against the maximal curvature (left: angle-distance method; right: angle-“intensity difference” method)


203

We divide the overall values of maximal curvature into ten even intervals and calculate the average mean squared errors (MSE) in each interval with the formula below. Average MSE

∑

MSE

.

(9)

where n is the number of vectors that fall into the interval and v is the corresponding vector. The gradient of the maximal curvatures are vectors. The mean square error is calculated per vertex basis with the equation below. .

(10)

is the gradient vector estimated by our methods and g , g , g is where ( , , the ground truth.Figure 4 histograms exhibit that the errors mainly manifest in the area where n has a large number, which is rational. It also shows that the “angle” and “intensity difference” combined weight sum function shows a better precision. In the qualitative display results, we can see that crest lines appears in the flat region as well. This is mainly due to the numerical errors accumulated in the calculation of high order derivatives. Consequently we need to provide a filtering mechanism to eliminate the crest lines presented in less feature intensive regions. The filtering method is simple. In each surface triangle patch, we evaluate the value of the maximal curvature. If the maximal curvature of a vertex is close to zero, then we manual set it to zero. The threshold of the zeroing is defined case by case basis for different surfaces. If the maximal value of the maximal curvature is m, then we define the zeroing threshold to be m/10 or m/5 depending on the surface feature variations. Subsequently in the process of crest lines extraction, if any vertex of a triangle patch has a maximal curvature of zero, then the triangle patch will not be considered for a crest line drawing operation. Figure 5 shows the filtering result.

Fig. 5. Filtered crest lines display on the smoothed cube surface (left) and crest lines on a more complicated surface (right)

204

5

P. Zheng et al.

Conclusion and Future Works

In this study we investigate a method to extract crest lines for an isosurface base on information from both volume data and mesh surface, proposed weight sum function methods, compared the methods and found out the most suitable method applied. In the post-processing phase, we also provided a simple filtering mechanism to enhance the accuracy of the crest lines extraction. The end result is convincing. In our future works, we will try to implement this method on real world data, such as 3D medical data to extract silent features to facilitate practical applications. Acknowledgement. This study is conducted under the auspices of Research University Grant (1001 / PKOMP / 814109) Universiti Sains Malaysia.

References 1. Thiron, J.P., Gourdon, A.: The 3D Marching Lines Algorithm and Its Application to Crest Lines Extraction. Rapport De Recherche, INRIA. No.1672 (1992) 2. Monga, O., Benayoun, S.: Using Partial Derivatives of 3D Images to Extract Typical Surface Features. Computer Vision and Image Understanding 61(2), 171–189 (1995) 3. Stylianou, G., Farin, G.: Crest Lines Extraction from 3D Triangulated Meshes. In: Hierarchical and Geometrical Methods in Scientific Visualization, pp. 269–281. Springer, Heidelberg (2003) 4. Hildebrandt, K., Polthier, K., Wardetzky, M.: Smooth Feature Lines on Surface Meshes. In: Third Eurographics Symposium on Geometry Processing (SGP 2005), pp. 85–90 (2005) 5. Yoshizawa, S., Belyaev, A., Seidel, H.-P.: Fast and Robust Detection of Crest Lines on Meshes. In: ACM Symposium on Solid and Physical Modelling (SPM 2005), pp. 227–232 (2005) 6. Yoshizawa, S., Belyaev, A., Seidel, H.-P.: Fast, Robust and Faithful Method for Detecting Crest Lines on Meshes. Computer Aided Geometric Design 25, 545–560 (2008) 7. Bloomenthal, J.: Polygonization of Implicit Surfaces. Computer Aided Geometric Design 5, 341–355 (1988) 8. Burns, M., Klawe, J., Rusinkiewicz, S., et al.: Line Drawings from Volume Data. ACM Transactions on Graphics (Proc. SIGGRAPH) 24(3), 512–518 (2005) 9. Yuan, X., Chen, B.: Procedural Image Processing for Visualization. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Remagnino, P., Nefian, A., Meenakshisundaram, G., Pascucci, V., Zara, J., Molineros, J., Theisel, H., Malzbender, T. (eds.) ISVC 2006. LNCS, vol. 4291, pp. 50–59. Springer, Heidelberg (2006) 10. Howard Zhou, H., Sun, J., Turk, G., Rehg, G.M.: Terrain Synthesis from Digital Elevation Models. IEEE Transactions on Visualization and Computer Graphics 13(4), 834–848 (2007) 11. Li, Z.J.J., Ferrer, M.A., Travieso, C.M., Alonso, J.B.: Biometric Based on Ridges of Palm Skin Over the Head of the Second Metacarpal Bone. Electronics Letters 42(7), 391–393 (2006) 12. Zhao, Y.L., Jiang, C.F., Xu, W., et al.: New Algorithm of Automation Fingerprint Recognition. In: International Conference on Engineering and Computer Science, pp. 1–4 (2009)


205

13. Du, T.L.H., Duc, D.A., Vu, D.N.: Ridge and Valley based Face Detection. In: International Conference on Research, Innovation and Vision for the Future, pp. 237–243 (2006) 14. Lengagne, R., Fua, P., Monga, O.: 3D Face Modelling from Stereo and Differential Constraints. In: International Conference on Pattern Recognition, pp. 148–153 (1998) 15. Prinet, V., Monga, O., Rocchisani, J.M.: Multi-dimensional Vessels Extraction Using Crest Lines. In: IEEE Conf. Eng. in Medicine and Bio., vol. 1, pp. 393–394 (1997) 16. Elmoutaouakkil, A., Peyrin, F., Elkafi, J., Laval-Jeantet, A.-M.: Segmentation of Cancellous Bone from High-resolution Computed Tomography Images: Influence on Trabecular Bone Measurements. IEEE Trans. Med. Imaging 21(4), 354–362 (2002) 17. Subsol, G., Roberts, N., Boran, M., et al.: Automatic Analysis of Cerebral Atrophy. Magnetic Resonance Imaging 15(8), 917–927 (1997) 18. Thirion, J.P.: New Feature Points Based on Geometric Invariants for 3D Image Registration. International Journal of Computer Vision 18(2), 121–137 (1996) 19. Pan, Z., Belaton, B., Liao, I.Y.: Isosurface Extraction of Volumetric Data Using Implicit Surface Polygonization. In: Third Asia International Conference on Modelling and Simulation, pp. 555–559 (2009) 20. Lorensen, E., Cline, H.E.: Marching Cubes: A High Resolution 3D Surface Construction Algorithm. Computer Graphics 21(4), 163–169 (1987) 21. Pan, Z., Belaton, B., Liao, I.Y., Rajion, Z.A.: Finite Difference Error Analysis of Geometry Properties of Implicit Surfaces. In: IEEE Symposium on Computers and Informatics, pp. 413–418 (2011) 22. Lee, K.W., Wang, W.P.: Feature-Preserving Mesh Denoising via Bilateral Normal Filtering. In: Ninth International Conference on Computer Aided Design and Computer Graphics, pp. 275–280 (2005)

AdaBoost-Based Approach for Detecting Lithiasis and Polyps in USG Images of the Gallbladder Marcin Ciecholewski Institute of Computer Science, Jagiellonian University, ul. Lojasiewicza 6, 30-348 Krakow, Poland [email protected]

Abstract. This article presents the application of the AdaBoost method to recognise gallbladder lesions such as lithiasis and polyps in USG images. The classifier handles rectangular input image areas of a specific length. If the diameter of areas segmented is much greater than the diameter expected on the input, wavelet approximation of input images is used. The classification results obtained by using the AdaBoost method are promising for lithiasis classification. In the best case, the algorithm achieved the accuracy of 91% for lithiasis and of 80% when classifyingpolyps, as well as the accuracy of 78.9% for polyps and lithiasis jointly. Keywords: Adaptive Boosting classification, Medical Image Analysis, Gallbladder lesions, Ultrasonography (USG), Visual Informatics.

1 Introduction 10-15% of the adult population of Europe and US is estimated to have gallstones in their gallbladders [9]. Another widespread disease is the presence of polyps in the gallbladder, including cancerous ones which are found in 5% of the global population [8]. The early detection of gallbladder diseases plays a huge role in improving the efficacy of their treatment. If gallbladder lesions are detected at an early development stage, then the disease progress can be stopped, making further treatment easier. Consequently, if software supporting the analysis of gallbladder USG images for the early detection of lesions such as gallstones and polyps (including those of cancerous origin) is developed, it will find its use. It will be of high significance as for the gallbladder there are no ready, practical solutions to help the physician in their work. The purpose of this project was to develop an application based on adaptive boosting method (AdaBoost) [5, 7] for classifying USG gallbladder images, including those showing lesions such as lithiasis or polyps inside the gallbladder shape. Fig. 1(a) shows an image without lesions, while Fig. 1(b) shows an image with polyp. In Fig. 1(c) lithiasis is visible. Unlike neural networks [12], the boosting method is based on the idea of combining many classifiers into a single reliable one. The method calls for set of weak classifiers able to give the final answer which may, however, be simplified and contain some degree of error. However, the scheme for training these classifiers has H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 206–215, 2011. © Springer-Verlag Berlin Heidelberg 2011

AdaBoost-Based Approach for Detecting Lithiasis and Polyps in USG Images

207

Fig. 1. Examples USG images of the gallbladder. (a) An image of a gallbladder free of lesions. (b) An image showing a polyp inside the gallbladder. (c) An image with visible cholecystolithiasis (gallstone).

been developed so that the error in the final classification is small. Of the many boosting methods, the Adaptive Boosting (AdaBoost) classifier was selected, which had already been applied to medical image analyses. For example, in publication [10] the AdaBoost classifier was used to locate blood vessel borders in intravascular ultrasound images. In publication [10], two classifiers – AdaBoost and SVM – were combined to distinguish Alzheimer disease lesions from healthy brains in cerebral MRI images. This yielded an 85% accuracy. The section below describes the designed system using the AdaBoost method to classify lesions like lithiasis and polyps. The following section shows the processing of input USG gallbladder images. The technique of classification with the use of the AdaBoost method is presented in the third section below. The fourth section below presents the experiments conducted and selected research results. The last section provides a summary.

2 System Description The design of a system enabling lesions of the gallbladder such as lithiasis and polyps to be detected is based on a machine learning method using the AdaBoost classifier. The general structure is presented in Algorithm 1. Only errors introduced during the preparation of the files will be corrected. The methods presented in Algorithm 1 comprise several steps. First, every USG image is pre-processed by segmenting the shape of the gallbladder and eliminating the area of uneven contrast background. The approximate edge of the gallbladder was found by applying one of the active contour models [1, 2]. Then, the fragment of the image located outside the gallbladder contour is eliminated from the image. In the second step, a set of rectangular areas is selected from every available image from the database of gallbladder USG images. Selected areas are partitioned into patterns designated for learning and validating. Then, a set of features is distinguished for every pattern. Features from the learning set are designated for training the algorithm which will then be able to take a binary decision about the image features supplied to it. After the learning process, the classifier is validated using features of

208

M. Ciecholewski

the image from the validating set in order to evaluate the accuracy of the classifier. In particular, a confusion matrix with the dimensions 2x2: Ai,j is calculated. The element ai,j matrix represents the number of samples belonging to class i, assigned by the algorithm to class j, (i, j ∈ {1 – lesion, 0 - nonlesion}). As finding a lesion is treated as a positive diagnosis, while the lack of lesions as a negative one, the confusion matrix may contain four possible values determining the classifiers behaviour.

a1.1 , true negative a1.0 + a1.1 a1.0 and false negative ratio . a1.0 + a1.1

These are as follows: true positive ratio specified as the ratio

a0.0 a0.1 , false positive ratio a0.0 + a0.1 a0.0 + a0.1

The true positive ratio is also referred to as the sensitivity of the classifier, while the true negative as its specificity. Finally, the overall accuracy, or the success rate of the classifier can be quantified as

a1.1 + a0.0 . a0.0 + a0.1 + a1.0 + a1.1

3 Analysed Data An image database obtained from the Image Diagnostics Department of the Gdansk Regional Specialist Hospital, Poland, was used in the research on USG image analysis. The database of images used in this research contained 800 images, including 600 images without lesions and 200 images containing them, specifically 90


209

images showing polyps and 110 images depicting gallbladder stones. USG images of the gallbladder were processed by the histogram normalization transformation to improve their contrast, and the gallbladder shape was segmented using active contour models [1, 2]. Then the background area of uneven contrast was eliminated from images. The entire 800 database of gallbladder images was processed in this fashion. For illustration, Figure 2(a) shows an unprocessed USG image depicting lithiasis from the database provided, while Figure 2(b) is an image after the histogram normalisation transformation, with the approximate edge marked. The contour was initiated manually inside the gallbladder shape and on the outside of the single lesion or several lesions present, as illustrated (dashed line) in Figure 2 (b). Figure 2(c) is an image of a gallbladder from which the uneven background of the image was

Fig. 2. Pre–processing of the input USG image of the gallbladder. (a) Input image obtained from the USG image database. (b) An image with improved contrast after the histogram normalisation transformation and the edge marked using the active contour method. The dashed line shows manually initiated contour inside the gallbladder shape. (c) A USG gallbladder image with the background eliminated. (d) The marked rectangular image fragment with the gallstone in the centre of the region.

210

M. Ciecholewski

eliminated by setting the black colour to pixels located outside the contour. For images showing lesions, a rectangular bounding region containing the disease was captured. Then, the centre of the lesion was approximated, after which the rectangular region was cropped so as to contain the lesion at its centre. An example is shown in Fig. 2(d). These transformations generated a set of rectangular regions of various dimensions, every one of which contained a lesion in the centre. The above procedure made it possible to obtain samples of image regions containing lesions. To obtain samples without lesions, an region equal in size to that obtained for a lesion was delineated in images from healthy individuals. As there are more images without lesions, for every sample with a lesion there were 3 samples identified with the same area free of lesions, coming from different patients. The set of 200 samples with lesions, containing 90 of polyps and 110 of stones and 600 samples from images without lesions was divided into two sets: ─ The training set 300 of samples, containing 200 samples without lesions and 100 samples with lesions (60 of stones and 40 of polyps). ─ The validation set 500 of samples, containing 400 samples without lesions and 100 samples with lesions (50 of stones and 50 of polyps). The number of cases with lesions in the validating set represents 25% of all samples in that set. The classifier handles rectangular input image areas of a specific length. If the diameter of areas segmented is much greater than the diameter expected on the input, wavelet approximation of input images is used. There are two scenarios for selecting the level of this approximation. Fixed Scaling: every sample is downscaled several times using Daubechies-4 wavelet [3] method. Then the centrally located rectangular area of the required size is segmented. The diameter of the lesion is not used. Variable Scaling: Every sample is downscaled using the Daubechies-4 wavelet method. However, the number of subsequent approximations depends on the diameter of the lesion. The number is selected so that the lesion diameter is always smaller than the width of the classifier window, but greater than its half. Then, the centrally located area of the rescaled sample is segmented. For images showing a potentially healthy gallbladder, the diameter of lesions is immaterial until lesions are detected. This is why a constant threshold of wavelet approximation is used in the first scenario. Under the second scenario, the approximation level is randomly selected from the range of scales found in samples containing lesions. What is more, the distribution of scales approximates the distribution of scales for samples containing lesions. In addition, the AdaBoost method is used to classify image windows of fixed width. The AdaBoost method uses additional filtering of input images to increase the accuracy.

4 Boosting Method in Detecting Irregularities The AdaBoost classifier is adaptive in the sense that every subsequent classifier created is tweaked for instances wrongly classified by previous classifiers. The process of training the classifier is summarized in the form of Algorithm 2. During the


211

training phase, every vector of a pattern from the training set is weighted. Initially, the weights are identical for all vectors. Then, in every iteration, the weak classifier is trained to minimise the weight error for all samples. Every iteration causes the weights to be changed by deducting a value dependent on the value of the weak classifier’s error in the entire training set from the weight. However, weights are reduced only for instances which the classifier correctly classified in the current iteration. The weight of a weak classifier in the whole ensemble is also related to the error of its learning. The assignment of no nuniform, time–varying weights to vectors during the training is necessary to minimise the error rate of the final aggregated classifier. During the training, the ability of the classifier to classify items in the training set constantly increases. This is because weak classifiers used in AdaBoost are complementary. Thus samples of vectors incorrectly classified by some weak classifiers are classified correctly by other weak classifiers.

In particular, for every round t of all rounds T, the weak classifier ht is trained using the training set Tr with the weights Dt. The training set is made up of instances from domain X, marked with labels from set C. The training of the weak classifier is left for the unspecified algorithm WeakLearner, which should minimise the training error ɛt of the resultant weak classifier depending of weights Dt. Based on the error ɛt of the weak classifier ht, the parameters αt and βt are calculated. The first parameter defines the weight of the classifier ht in the final, joint classifier. The second parameter provides the multiplication constant, used to reduce the weights {Dt+1(i)} of the correctly classified instances {i}. Weights of instances that have been wrongly

212

M. Ciecholewski

classified are not changed. Thus, after normalising the new weights {Dt+1(i)}, the appropriate weights of wrongly classified instances from the training set are increased. Consequently, in round ht+1, WeakLearner focuses on those instances and thus the chance increases that classifier ht+1 will learn to classify them correctly. The final strong classifier hfin uses a weighted voting scheme on the results of weak classifiers ht. The weights of single classifiers are defined by the constant αt. Two special instances which are treated separately during the operation of the algorithm are distinguished. The first instance is when ɛt is 0. In this instance, weights Dt+1(i) are equal to Dt and ht+1 is equal to ht. Consequently, the algorithm does not execute the rest of the training process. The second instance occurs when ɛt ≥ 0.5. In this instance, the theoretical restrictions for ht are not met and the algorithm cannot continue new training rounds. One of the most important issues in the use of the AdaBoost scheme is the selection of the weak classifier which segregates instances into two distinguishable classes. Following [14], a classifier that selects a single feature from the entire feature vector is used. The training of the weak classifier consists in selecting the best feature and choosing the threshold of value for this feature, which threshold ensures the best segregation of instances belonging to one class from those belonging to the other. The selection contributes to minimising the error of weights for the training set. The feature set consists of features calculated as the differences of the sum of pixels intensities inside two, three or four adjacent rectangles. These rectangles are of different sizes and are located differently inside the image window but they must adjoin.

5 Training Phase and Selected Experiment Results This section contains the research results and an assessment of the AdaBoost algorithm based on the experiments completed. The tests were carried out on data sets presented in Section 3. The algorithm used central, rectangular part of the images with size equal to 24 x 24 pixels. 5.1 Input Image Filtering Before the wavelet scaling is applied, the input image is filtered using various methods. The filters listed below are a group of typical, elementary filters used in image processing. The following filters have been used [6]: No filtering – No filtering Laplacian – Laplacian filter Unsharp – Unsharp contrast enhancement filter, i.e., negation of the Laplacian LoG – Laplacian of Gaussian filter Dilate – Greyscale dilation using a disk structuring element


213

Table 1. Results of the AdaBoost classifier in different configurations for a validation sets. For fixed wavelet scaling, the level of approximation is specified. Filtering type Scaling Type Sensitivity (%) Specificity (%) Accuracy (%) No filtering Fixed – 3 75.2% 79% 78.9% No filtering Fixed – 4 80.4% 73.3% 74.7% No filtering Variable 71.4% 70.8% 72.3% Laplacian Fixed – 3 49.1% 63.7% 62.2% Laplacian Fixed – 4 42.2% 79.1% 75.5% Laplacian Variable 51.1% 66.1% 64.2% Unsharp Fixed – 3 78.1% 75.2% 75.7% Unsharp Fixed – 4 75.2% 75.8% 81% Unsharp Variable 67.3% 73.2% 72.6% LoG Fixed – 3 52.2% 71.3% 69.5% LoG Fixed – 4 54% 79.2% 77.3% LoG Variable 53.1% 67.2% 66.3% Dilate Fixed – 3 78.2% 76.3% 77% Dilate Fixed – 4 76% 78.9% 78.3% Dilate Variable 73.5% 68.2% 68.9% Test 1. Samples containing 100 lesions and 400 free of lesions No filtering Fixed – 3 82% 89.6% 89.1% No filtering Fixed – 4 77% 91.8% 90.5% No filtering Variable 94.5% 90.5% 90.8% Laplacian Fixed – 3 38.6% 70% 68.3% Laplacian Fixed – 4 48.5% 78.5% 76.7% Laplacian Variable 73.5% 64.6% 65.2% Unsharp Fixed – 3 76.5% 86.8% 86.5% Unsharp Fixed – 4 83.5% 91% 90.6% Unsharp Variable 88.6% 90.7% 90.6% LoG Fixed – 3 43.5% 66.4% 65.5% LoG Fixed – 4 53.5% 83.8% 82.3% LoG Variable 81% 63.4% 64.5% Dilate Fixed – 3 76% 89.9% 89.2% Dilate Fixed – 4 86% 86.7% 86.3% Dilate Variable 86% 91.3% 91% Test 2. Samples containing 50 lithiasis and 400 free of lesions No filtering Fixed – 3 65% 81.4% 80% No filtering Fixed – 4 75.5% 68.4% 69% No filtering Variable 62% 58% 59.2% Laplacian Fixed – 3 58.3% 69.2% 68.6% Laplacian Fixed – 4 76% 66.4% 67.5% Laplacian Variable 46.3% 75.9% 74.4% Unsharp Fixed – 3 46% 74.5% 74% Unsharp Fixed – 4 71% 68.5% 68.4% Unsharp Variable 41% 59.3% 58.4%

214

M. Ciecholewski Table 1. (Continued) LoG LoG LoG Dilate Dilate Dilate

Fixed – 3 52.1% 71.7% Fixed – 4 54.8% 79.4% Variable 53.5% 67.2% Fixed – 3 78.5% 76.8% Fixed – 4 76% 78.9% Variable 73.5% 68.3% Test 3. Samples containing 50 polyps and 400 free of lesions

69.8% 77.6% 66.3% 77% 78.6% 68.4%

After the wavelet scaling is completed, the classifier undergoes the training stage. The number of rounds for training the classifier was assumed as 300. For each different filtering type and wavelet scaling mode, a different classifier is trained. For every set consisting of the applied scaling method, the total accuracy, sensitivity and specificity are determined. Test results of the first test for the validation set containing 400 samples without lesions and 50 lithiasis and 50 polyps are presented in Table 1 Test1. The Test 1 demonstrated that the best results were achieved for wavelet fixed scaling. The highest sensitivity (81%) was obtained for the 4–level scaling. The total accuracy of the classifier is the greatest (78.9%) if the 3-level scaling is used. To find which type of lesion contributed to these results, the classifier was trained and then tests were conducted only for samples containing gallstones and samples containing no lesions (Test 2), while the third test scenario covered samples containing polyps and instances without lesions (Test 3). Results are presented in Table 1. In polyps detection, the dilation of the image resulted in some improvement of the results. In particular, it increased the sensitivity, as a cost of slight decrease in specificity in all three scaling configurations. To summarise test results it can be said that the AdaBoost classifier achieved significantly better results for lithiasis than for polyps. This can be attributed to the nature of the polyps presented in USG images which are highly non–uniform and not as well localized as gallstones. In the best case, the algorithm achieved the accuracy of 91% for lithiasis and of 80% when classifying polyps.

6 Summary and Further Research Directions This article presents a method of recognising lithiasis and polyps of the gallbladder in USG images, developed for a computer system supporting the early diagnostics of gallbladder lesions. USG images of the gallbladder were first processed by the histogram normalisation transformation to improve their contrast, and the gallbladder shape was segmented using active contour models. Then the background region of uneven contrast was eliminated from images. Subsequently, the Adaboost classifier was used to analyse a set of rectangular regions of various sizes. If the diameter of areas segmented is much greater than the diameter expected on the input, wavelet approximation of input images is used. Results achieved using the AdaBoost method are promising for lithiasis classification. Classifier accuracy and sensitivity reached 90% for lithiasis using variable scalling and dropped to some 77% for polyps and gallstones jointly when the fixed wavelet scalling was used with no filtering, unsharp or dilate. This drop in accuracy and sensitivity in the presence of both types of lesions was due to the classifier’s inability to recognise polyps. In recognising only polyps the algorithm achieved the best accuracy of 80% but sensitivity is just only of 65%


215

when the same fixed 3–level scalling was used. Further research will focus on the local processing of data from the image, i.e. developing specialised algorithms for analysing gallbladder USG images which will enable image features to be extracted. It is also necessary to develop and apply filters which will make it possible to set off lesions in USG gallbladder images. Further research will also aim at developing a fast and effective algorithm for selecting suspiciously–looking areas within gallbladder USG images in which possible lesions may be present. Acknowledgements. This research was financed with state budget funds for science for 2009-2012 as research project of the Ministry of Science and Higher Education: N N519406837.

References 1. Ciecholewski, M.: Gallbladder Segmentation in 2-D Ultrasound Images Using Deformable Contour Methods. In: Torra, V., Narukawa, Y., Daumas, M. (eds.) MDAI 2010. LNCS (LNAI), vol. 6408, pp. 163–174. Springer, Heidelberg (2010) 2. Ciecholewski, M.: Gallbladder Boundary Segmentation from Ultrasound Images Using Active Contour Model. In: Fyfe, C., Tino, P., Charles, D., Garcia-Osorio, C., Yin, H. (eds.) IDEAL 2010. LNCS, vol. 6283, pp. 63–69. Springer, Heidelberg (2010) 3. Daubechies, I.: Orthonormal bases of compactly supported wavelets. Comm. Pure Appl. Math. 41, 909–996 (1988) 4. Freund, Y., Schapire, R.E.: A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting. In: Vitányi, P.M.B. (ed.) EuroCOLT 1995. LNCS, vol. 904, pp. 23–37. Springer, Heidelberg (1995) 5. Freund, Y., Schapire, R.: A short introduction to boosting. J. Jpn. Soc. Art. Intell. 14(5), 771–780 (1999) 6. Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd edn. Prentice Hall (2008) 7. Meir, R., Ratsch, G.: An Introduction to Boosting and Leveraging. In: Mendelson, S., Smola, A.J. (eds.) Advanced Lectures on Machine Learning. LNCS (LNAI), vol. 2600, pp. 118–183. Springer, Heidelberg (2003) 8. Myers, R.P., Shaffer, E.A., Beck, P.L.: Gallbladder polyps: Epidemiology, natural history and management. Can J Gastroenterol 16(3), 187–194 (2002) 9. Portincasa, P., Moschetta, A., Palasciano, G.: Cholesterol gallstone disease, Lancet, vol. 368, pp. 230–239 (2006) 10. Pujol, O., Rosales, M., Radeva, P., Fernandez–Nofrerıas, E.: Intravascular Ultrasound Images Vessel Characterization Using AdaBoost. In: Magnin, I.E., Montagnat, J., Clarysse, P., Nenonen, J., Katila, T. (eds.) FIMH 2003. LNCS, vol. 2674, pp. 242–251. Springer, Heidelberg (2003) 11. Savio, A., García-Sebastián, M., Graña, M., Villanúa, J.: Results of an Adaboost Approach on Alzheimer’s Disease Detection on MRI. In: Mira, J., Ferrández, J.M., Álvarez, J.R., de la Paz, F., Toledo, F.J. (eds.) IWINAC 2009. LNCS, vol. 5602, pp. 114–123. Springer, Heidelberg (2009) 12. Sehad, S., Desarnaud, S., Strauss, A.: Artificial neural classification of clustered microcalcifications on digitized mammograms. In: Proc. IEEE Int. Conf. Syst. Man Cybernet., pp. 4217–4222 (1997) 13. Smola, A.J., Scholkopf, B.: On a kernel-based method for pattern recognition, regression and approximation. Algorithmica 22, 211–231 (1998) 14. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vision 57(2), 137– 154 (2004)

Assessing Educators’ Acceptance of Virtual Reality (VR) in the Classroom Using the Unified Theory of Acceptance and Use of Technology (UTAUT) Niwala Haswita Hussin, Jafreezal Jaafar, and Alan G. Downe Department of Computer and Information Sciences Universiti Teknologi PETRONAS Perak, Malaysia [email protected], [email protected], [email protected]

Abstract. Teaching and learning processes can be enhanced through the adoption of Virtual Reality (VR) Technology in the classroom. However, educators in Malaysia have been slow to adopt VR approaches to classroom content delivery. Little is known about the specific factor that impact to the intention to employ VR in educational environment. The purpose of this present research paper is to investigate and provide a preliminary analysis of a framework that predicts level of technology acceptance in a post secondary institution in Perak, Malaysia. A questionnaire based on factor identified in the Unified Theory of Acceptance and the Use Of Technology (UTAUT) was administered to 41tutoring level educators to measure the effect of performance expectancy (PE), Effort Expectancy (EE), and Social Influence (SI) on Behavioral Intention (BI) towards VR. Regression analysis identified that EE and SI significantly influenced BI. The need for future study of technology acceptance in education was discussed. Keywords: Educators’ Informatics.

acceptance,

Virtual

Reality,

UTAUT,

Visual

1 Introduction Traditional classroom technique has move towards the evolutions of technology in term of integration, adoption, and communication for teaching and learning purposes. It has been reported that by having utilization of CMC, accessibility and quality in education has improve to the next level [1]. All syllabus or programs of study in worldwide are promoting learning through CMC (See, for example, [2], [3], [4], [5],[6]). Virtual Reality (VR) can be considered as computer mediated communication (CMC) since it has the involvement of technology not only the computer but also other electronic communication devices to create a virtual environment (VE). VE is a scenario where user can participate in real time application within a situation created by computer technology. The illusions created are very likely to be similar with the H. Badioze Zaman et al. (Eds.): IVIC 2011, Part I, LNCS 7066, pp. 216–225, 2011. © Springer-Verlag Berlin Heidelberg 2011

Assessing Educators’ Acceptance of Virtual Reality (VR) in the Classroom

217

real world. VR immersive world can be control real-time without users intervention physically and limit the risk to users. VR eliminating difficulties occurred in the real world [7]. It has been used in many areas such as pilot training that promote safe training zones even under extreme conditions, virtual oil drilling activities, medical and healthcare applications, games, and so on. Bonwell inquired for additional research to be done comprehensively for it to be a platform as a learning direction in the class [8]. An extensive research have to be conducted on particular technology with 3D virtual worlds, research of VR enrichment that grant learners to associate with the knowledge that are impossible in normal class setting ([9]; [10]; [11]; [12]). One factor that has been understudied relates to the acceptance of VR in the classroom by educators. There are a numbers of theories had been proposed to determine user acceptance towards technology. For instance, Technology Acceptance Model (TAM). However, in a critical review, Legris et.al contends to use a theory that will cover more variable in order to get wider clarification towards technology acceptance and adoption [13]. To solve this problem, the present authors have employed the Unified Theory of Acceptance and The Use of Technology (UTAUT) model [14] which outline eight other models including the Social Cognition Theory (SCT), Innovation Diffusion Theory (IDT), The Motivational Model (MM), Model of PC utilization (MPC), Theory of Acceptance Model and Theory of Planned Behavior (TAM - TPB), Theory of Acceptance Model (TAM), Theory of Planned Behavior (TPB), and Theory of Reason Action (TRA).

2 Literature Review Study of distribution and adoption of Information Communication Technology (ICT) can be divided into two levels which are individual level and organizational level. This study is considered as individual level because it focused more on educators’ acceptance of technology individually [15]. Factors in explaining or predicting the technologies use becoming a main focus by a variety of researcher. Many models have been structured to forecast technology acceptance. Figure 1 shows the original Unified Theory of the Acceptance and Use of Technology (UTAUT) [14]. Venkatesh et. al suggested four direct determinants for user acceptance and usage behavior: Performance Expectancy (PE), Effort Expectancy (EE), Social Influence (SI), and Facilitating Conditions (FC). PE refers to the degree to which individuals believe that using a VR system will help them accomplish an increments in work performance, EE defines the scale of ease related with the use of VR system, SI is the scale to which individuals perceive that important others believe they should use the new VR system, and FC indicates the scale to which individuals believe that institution and technical infrastructures support the use of a VR system. These relations are moderated by gender, age, experience and voluntariness.

218

N.H. Hussin, J. Jaafar, and A.G. Downe

Performance Expectancy Behavioral Intention

Effort Expectancy

Use Behaviour

Social Influence Facilitating Condition Gender

Age

Experien

Voluntarine ss Of Use

Fig. 1. Original UTAUT (Venkatesh et al., 2003)

2.2 Educators’ Acceptance of Technology There have been many studies of educators’ acceptances to implement ICT in school. Some research confirms the technology competence can amend teaching and learning processes, but in real implementations computer are not fully utilized as expected and left as dummy devices in a classroom. If an educator does use the devices, it was not utilized effectively. There were 81.8% teachers does not utilized a computer technology during teaching and learning process. The conclusion for this scenario may caused by teachers’ lack of confidence and professional development [17]. Sometimes, teachers did not realize benefits of technology towards teaching and learning process. Other explanations found for this situation are teachers’ did not have enough of technical support and lack of self confidence [18]. Some teachers preferred to use games to assist learning rather than computer technology [19].

3 Research Design 3.1 Unified Theory of Acceptance and the use of technology (UTAUT) As illustrated in Figure 2, this study examined determinant PE, EE, and SI will have a significant effect on educators’ acceptance towards VR. Moderator gender was used as controlling variables for data analysis purposes. Respondent will not to be exposed to a prototype of VR classroom at this stage. Thus, for this preliminary study Facilitating Condition (FC) will be excluded from research model, but it will be comprised in the model during Post Test session.


219

Performance Expectancy sZ ,s/KZ> /EdEd/KE

Effort Expectancy Social Influence Gender

Fig. 2. Research Model

3.2 Research Hypotheses Preliminary survey was done to get an overview from respondents using UTAUT questionnaires as a guideline. Data were collected using questionnaires consist of demographics questions and several questions for each variable in the research model. Based on the research model, the following research hypotheses were formulated: H1a : Performance Expectancy has a significant effect on Behavioral Intention H1b : Gender significantly moderates the relationship between Performance Expectancy and Behavioral Intention. H2a : Effort Expectancy have a significant effect on Behavioral Intention H2b : Gender significantly moderates the relationship between Effort Expectancy and Behavioral Intention. H3a : Social Influence has a significant effect on Behavioral Intention H3b : Gender significantly moderates the relationship between Social Influence and Behavioral Intention.

4 Research Methodology 4.1 Sample Participants in this study were college educators at Politeknik Ungku Omar,(PUO) Perak. As shown in Table 1, there were a total of 41 participants with 13 males and 28 females ranging in age from 20 to 50 and above. Most of them had relatively lengthy teaching experience and were attached to 7 different academic departments. A total of 41 usable questionnaires were returned.

220

N.H. Hussin, J. Jaafar, and A.G. Downe Table 1. Demographic Summary For Participants

Categories

Frequency

Percentage

A) Gender Male Female Total Sample size

13 28 41

32 % 68 % 100 %

B) Age 20 to 24 25 to 29 30 to 34 35 to 39 40 to 44 50 and above Total Sample size

2 9 17 7 4 2 41

5% 22 % 41 % 17 % 10 % 5% 100 %

C)Working Experience Teaching) Less than 1 year 1 – 5 years 6 -10 years More than 10 years Total Sample size

0 3 10 28 41

7% 24 % 68 % 100 %

4.2 Measures A survey mechanism was considered to quantify five constructs in the research model. It was comprised of two sections: Part A- Demographic information and Part B- contained 17 statements based on the UTAUT factors and adapted from the earlier instrument developed by Venkatesh et al. [14]. Please refer to Appendix A for Table 6 for items asked in the Questionnaires. Each statement was evaluated with a five-point Likert Scale with 1= Strongly Disagree, to 5= Strongly Agree. There were 17 questions covering the 5 construct indicated in Figure 2. 4.3 Reliability of the Instrument Table 2. Reliability Table

SN 1 2 3 4

Construct Performance Expectancy (PE) Effort Expectation (EE) Social Influence (SI) Behavioral Intention (BI)

Cronbach Alpha 0.8382 0.9324 0.9103 0.8784


221

All UTAUT constructs performance expectation (PE), effort expectation (EE), social influence (SI) and behavioural intention (BI) had high values of Cronbach alpha coefficient 0.8382, 0.9324, 0.9103 and 0.8784 respectively, and the values were considered high. Since all alpha values exceed 0.7, thus, the data can be considered as acceptable for construct reliability. In other words, the result showed, can be trusted for next process of data analysis [21],[22],[23] .

5 Results and Discussion A standard stepwise regression analysis was utilized on the collected preliminary data for getting all statistical result in this study. The analysis was in two phases: in the first phase the regression analysis was carried out without using a moderator. The analysis gave a low R value of 0.376 and R2 value of 0.141, implying that the measured variables accounted for about 14% of the variation in reasons for acceptance. Only the relationship between social influence and behavioural intention was significant with a coefficient of 0.0.376 at p

Visual Informatics: Sustaining Research and Innovations, Part I - IVIC 2011

Visual Informatics: Sustaining Research and Innovations, Part II - IVIC 2011

Informatics Engineering and Information Science, Part I - ICIEIS 2011

NETWORKING 2011, Part I

The Semantic Web: Research and Applications - ESWC 2011, Part I

Communication and Networking, Part I - FGCN 2011

Informatics Engineering and Information Science, Part III - ICIEIS 2011

Informatics Engineering and Information Science, Part IV - ICIEIS 2011

Informatics Engineering and Information Science, Part II - ICIEIS 2011

Brain Informatics - BI 2011

Advances in Visual Computing, Part I

Neural Information Processing, Part I - ICONIP 2011

ENTERprise Information Systems, Part I - CENTERIS 2011

The Semantic Web - ISWC 2011, Part I

Quantum Bio-Informatics I

Artificial Intelligence Applications and Innovations, Part II - Eann 2011, Aiai 2011

Applied Informatics and Communication, Part V

Intelligence and Security Informatics - PAISI 2011

The Semantic Web: Research and Applications, Part II - ESWC 2011

Artificial Neural Networks and Machine Learning, Part I - ICANN 2011

Advances in Information Technology and Education, Part I - CSE 2011

Intelligent Robotics and Applications, Part I - ICIRA 2011

Image Analysis and Recognition, Part I - ICIAR 2011

Innovative Computing and Information, Part I - ICCIC 2011

Ubiquitous Computing and Multimedia Applications - UCMA 2011, Part I

Advances in Image and Video Technology, Part I - PSIVT 2011

Information Computing and Applications, Part I - ICICA 2011

Applied Economics, Business and Development, Part I - ISAEBD 2011

Design, User Experience, and Usability, Part I - DUXU 2011

Adaptive and Natural Computing Algorithms, Part I - ICANNGA 2011

Automata, Languages and Programming, Part I - ICALP 2011

Visual Informatics: Sustaining Research and Innovations, Part I - IVIC 2011

Visual Informatics: Sustaining Research and Innovations, Part II - IVIC 2011

Informatics Engineering and Information Science, Part I - ICIEIS 2011

NETWORKING 2011, Part I

The Semantic Web: Research and Applications - ESWC 2011, Part I

Communication and Networking, Part I - FGCN 2011

Informatics Engineering and Information Science, Part III - ICIEIS 2011

Informatics Engineering and Information Science, Part IV - ICIEIS 2011

Informatics Engineering and Information Science, Part II - ICIEIS 2011

Brain Informatics - BI 2011

Advances in Visual Computing, Part I

Neural Information Processing, Part I - ICONIP 2011

ENTERprise Information Systems, Part I - CENTERIS 2011

The Semantic Web - ISWC 2011, Part I

Quantum Bio-Informatics I

Artificial Intelligence Applications and Innovations, Part II - Eann 2011, Aiai 2011

Applied Informatics and Communication, Part V

Intelligence and Security Informatics - PAISI 2011

The Semantic Web: Research and Applications, Part II - ESWC 2011

Artificial Neural Networks and Machine Learning, Part I - ICANN 2011

Advances in Information Technology and Education, Part I - CSE 2011

Intelligent Robotics and Applications, Part I - ICIRA 2011

Image Analysis and Recognition, Part I - ICIAR 2011

Innovative Computing and Information, Part I - ICCIC 2011

Ubiquitous Computing and Multimedia Applications - UCMA 2011, Part I

Advances in Image and Video Technology, Part I - PSIVT 2011

Information Computing and Applications, Part I - ICICA 2011

Applied Economics, Business and Development, Part I - ISAEBD 2011

Design, User Experience, and Usability, Part I - DUXU 2011

Adaptive and Natural Computing Algorithms, Part I - ICANNGA 2011

Automata, Languages and Programming, Part I - ICALP 2011

Recommend Documents