Ubiquitous Computing and Multimedia Applications: UCMA 2011, Part II

Communications in Computer and Information Science

151

Tai-hoon Kim Hojjat Adeli Rosslin John Robles Maricel Balitanas (Eds.)

Ubiquitous Computing and Multimedia Applications Second International Conference, UCMA 2011 Daejeon, Korea, April 13-15, 2011 Proceedings, Part II

13

Volume Editors Tai-hoon Kim Hannam University, Daejeon, Korea E-mail: [email protected] Hojjat Adeli The Ohio State University Columbus, OH, USA E-mail: [email protected] Rosslin John Robles Hannam University, Daejeon, Korea E-mail: [email protected] Maricel Balitanas Hannam University, Daejeon, Korea E-mail: [email protected]

ISSN 1865-0929 e-ISSN 1865-0937 ISBN 978-3-642-20997-0 e-ISBN 978-3-642-20998-7 DOI 10.1007/978-3-642-20998-7 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011926692 CR Subject Classification (1998): H.5.1, I.5, I.2, I.4, F.1, H.3, H.4

© Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Foreword

Ubiquitous computing and multimedia applications are areas that attract many academic and industry professionals. The goal of the International Conference on Ubiquitous Computing and Multimedia Applications is to bring together researchers from academia and industry as well as practitioners to share ideas, problems and solutions relating to the multifaceted aspects of ubiquitous computing and multimedia applications. We would like to express our gratitude to all of the authors of submitted papers and to all attendees, for their contributions to and participation in UCMA 2011. We believe in the need for continuing this undertaking in the future. We acknowledge the great effort of all the Chairs and the members of advisory boards and Program Committees of the above-listed event, who selected 15% of over 570 submissions, following a rigorous peer-review process. Special thanks go to SERSC (Science and Engineering Research Support Society) for supporting this conference. We are grateful in particular to the following speakers who kindly accepted our invitation and, in this way, helped to meet the objectives of the conference: Sabah Mohammed of Lakehead University and Peter Baranyi of Budapest University of Technology and Economics (BME). March 2011

Chairs of UCMA 2011

Preface

We would like to welcome you to the proceedings of the 2011 International Conference on Ubiquitous Computing and Multimedia Applications (UCMA 2011) which was held during April 13-15, 2011, at Hannam University, Daejeon, Korea. UCMA 2011 is focused on various aspects of advances in multimedia applications and ubiquitous computing with computational sciences, mathematics and information technology. It provided a chance for academic and industry professionals to discuss recent progress in the related areas. We expect that the conference and its publications will be a trigger for further related research and technology improvements in this important subject. We would like to acknowledge the great effort of all the Chairs and members of the Program Committee. Out of around 570 submissions to UCMA 2011, we accepted 86 papers to be included in the proceedings and presented during the conference. This gives an acceptance ratio firmly below 20%. Regular conference papers can be found in this volume while Special Session papers can be found in CCIS 151. We would like to express our gratitude to all of the authors of submitted papers and to all the attendees for their contributions and participation. We believe in the need for continuing this undertaking in the future. Once more, we would like to thank all the organizations and individuals who supported this event as a whole and, in particular, helped in the success of UCMA 2011. March 2011

Tai-hoon Kim Hojjat Adeli Rosslin John Robles Maricel Balitanas

Organization

Organizing Committee Honorary Co-chairs:

General Co-chairs:

Program Co-chairs:

Workshop Co-chairs:

Hyung-tae Kim (Hannam University, Korea) Hojjat Adeli (The Ohio State University, USA) Wai-chi Fang (National Chiao Tung University, Taiwan) Carlos Ramos (GECAD/ISEP, Portugal) Haeng-kon Kim (Catholic University of Daegu, Korea) Tai-hoon Kim (Hannam University, Korea) Sabah Mohammed (Lakehead University, Canada) Muhammad Khurram Khan (King Saud University, Saudi Arabia) Seok-soo Kim (Hannam University, Korea) Timothy K. Shih (Asia University, Taiwan)

International Advisory Board: Cao Jiannong (The Hong Kong Polytechnic University, Hong Kong) Frode Eika Sandnes (Oslo University College, Norway) Schahram Dustdar (Vienna University of Technology, Austria) Andrea Omicini (Universit` a di Bologna, Italy) Lionel Ni (The Hong Kong University of Science and Technology, Hong Kong) Rajkumar Buyya (University of Melbourne, Australia) Hai Jin (Huazhong University of Science and Technology, China) N. Jaisankar (VIT University, India) Gil-cheol Park (Hannam University, Korea) Ha Jin Hwang (Kazakhstan Institute of Management, Economics, and Strategic Research, Republic of Kazakhstan)

X

Organization

Publicity Co-chairs:

Publication Co-chairs:

Paolo Bellavista (Universit`a di Bologna, Italy) Ing-Ray Chen (Virginia Polytechnic Institute and State University, USA) Yang Xiao (University of Alabama, USA) J.H. Abawajy (Deakin University, Australia) Ching-Hsien Hsu (Chung Hua University, Taiwan) Deepak Laxmi Narasimha (University of Malaya, Malaysia) Prabhat K. Mahanti (University of New Brunswick, Canada) Soumya Banerjee (Birla Institute of Technology, India) Byungjoo Park (Hannam University, Korea) Debnath Bhattacharyya (MPCT, India)

Program Committee Alexander Loui Biplab K. Sarker Brian King Chantana Chantrapornchai Claudia Linnhoff-Popien D. Manivannan Dan Liu Eung Nam Ko Georgios Kambourakis Gerard Damm Han-Chieh Chao Hongli Luo Igor Kotenko J.H. Abawajy Jalal Al-Muhtadi

Javier Garcia-Villalba Khaled El-Maleh Khalil Drira Larbi Esmahi Liang Fan Mahmut Kandemir Malrey Lee Marco Roccetti Mei-Ling Shyu Ming Li Pao-Ann Hsiung Paolo Bellavista Rami Yared Rainer Malaka Robert C. Hsu Robert G. Reynolds

Rodrigo Mello Schahram Dustdar Seung-Hyun Seo Seunglim Yong Stefano Ferretti Stuart J. Barnes Su Myeon Kim Swapna S. Gokhale Taenam Cho Tony Shan Toshihiro Yamauchi Wanquan Liu Wenjing Jia Yao-Chung Chang

Table of Contents – Part II

A Smart Error Protection Scheme Based on Estimation of Perceived Speech Quality for Portable Digital Speech Streaming Systems . . . . . . . . . Jin Ah Kang and Hong Kook Kim

1

MDCT-Domain Packet Loss Concealment for Scalable Wideband Speech Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nam In Park and Hong Kook Kim

11

High-Quality and Low-Complexity Real-Time Voice Changing with Seamless Switching for Digital Imaging Devices . . . . . . . . . . . . . . . . . . . . . . Sung Dong Jo, Young Han Lee, Ji Hun Park, Hong Kook Kim, Ji Woon Kim, and Myeong Bo Kim Complexity Reduction of Virtual Reverberation Filtering Based on Index-Based Convolution for Resource-Constrained Devices . . . . . . . . . . . Kwang Myung Jeon, Nam In Park, Hong Kook Kim, Ji Woon Kim, and Myeong Bo Kim Audio Effect for Highlighting Speaker’s Voice Corrupted by Background Noise on Portable Digital Imaging Devices . . . . . . . . . . . . . . . . . . . . . . . . . . Jin Ah Kang, Chan Jun Chun, Hong Kook Kim, Ji Woon Kim, and Myeong Bo Kim

19

28

39

Detection of Howling Frequency Using Temporal Variations in Power Spectrum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jae-Won Lee and Seung Ho Choi

46

Differential Brain Activity in Reading Hangul and Hanja in Korean Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hyo Woon Yoon and Ji-Hyang Lim

52

Computational Neural Model of the Bilingual Stroop Effect: An fMRI Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hyo Woon Yoon

60

The Eye Movement and Data Processing Due to Obtained BOLD Signal in V1 : A Study of Simultaneous Measurement of EOG and fMRI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hyo Woon Yoon, Dong-Hwa Kim, Young Jae Lee, Hyun-Chang Lee, and Ji-Hyang Lim

66

XII

Table of Contents – Part II

Face Tracking for Augmented Reality Game Interface and Brand Placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yong Jae Lee and Young Jae Lee

72

On-line and Mobile Delivery Data Management for Enhancing Customer Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hyun-Chang Lee, Seong Yoon Shin, and Yang Won Rhee

79

Design of LCL Filter Using Hybrid Intelligent Optimization for Photovoltaic System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jae Hoon Cho, Dong-Hwa Kim, M´ aria Virˇc´ıkov´ a, and Peter Sinˇc´ ak

90

Distributed Energy Management for Stand-Alone Photovoltaic System with Storages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jae Hoon Cho and Dong-Hwa Kim

98

Framework for Performance Metrics and Service Class for Providing End-to-End Services across Multiple Provider Domains . . . . . . . . . . . . . . . Chin-Chol Kim, Jaesung Park, and Yujin Lim

107

Design of a Transmission Simulator Based on Hierarchical Model . . . . . . . Sang Hyuck Han and Young Kuk Kim Recommendation System of IPTV TV Program Using Ontology and K-means Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jongwoo Kim, Eungju Kwon, Yongsuk Cho, and Sanggil Kang A Novel Interactive Virtual Training System . . . . . . . . . . . . . . . . . . . . . . . . Yoon Sang Kim and Hak-Man Kim

114

123 129

Recommendation Algorithm of the App Store by Using Semantic Relations between Apps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yujin Lim, Hak-Man Kim, Sanggil Kang, and Tai-hoon Kim

139

A Comparative Study of Bankruptcy Rules for Load-shedding Scheme in Agent-Based Microgrid Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hak-Man Kim and Tetsuo Kinoshita

145

The Visualization Tool of the Open-Source Based for Flight Waypoint Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Myeong-Chul Park and Seok-Wun Ha

153

Space-Efficient On-the-fly Race Detection Using Loop Splitting . . . . . . . . Yong-Cheol Kim, Sang-Soo Jun, and Yong-Kee Jun A Comparative Study on Responding Methods for TCP’s Fast Recovery in Wireless Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mi-Young Park, Sang-Hwa Chung, Kyeong-Ae Shin, and Guangjie Han

162

170

Table of Contents – Part II

XIII

The Feasibility Study of Attacker Localization in Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Young-Joo Kim and Sejun Song

180

Efficiency of e-NR Labeling for On-the-fly Race Detection of Programs with Nested Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sun-Sook Kim, Ok-Kyoon Ha, and Yong-Kee Jun

191

Lightweight Labeling Scheme for On-the-fly Race Detection of Signal Handlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Guy Martin Tchamgoue, Ok-Kyoon Ha, Kyong-Hoon Kim, and Yong-Kee Jun

201

Automatic Building of Real-Time Multicore Systems Based on Simulink Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Minji Cha and Kyong Hoon Kim

209

A Study on the RFID/USN Integrated Middleware for Effective Event Processing in Ubiquitous Livestock Barn . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jeonghwan Hwang and Hyun Yoe

221

Design of Android-Based Integrated Management System for Livestock Barns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . JiWoong Lee and Hyun Yoe

229

A Study of Energy Efficient MAC Based on Contextual Information for Ubiquitous Agriculture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hochul Lee and Hyun Yoe

234

A Service Scenario Based on a Context-Aware Workflow Language in u-Agriculture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yongyun Cho and Hyun Yoe

240

A Sensor Data Processing System for Mobile Application Based Wetland Environment Context-aware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yoon-Cheol Hwang, Ryum-Duck Oh, and Gwi-Hwan Ji

245

An Intelligent Context-Aware Learning System Based on Mobile Augmented Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jin-Il Kim, Inn-woo Park, and Hee-Hyol Lee

255

Inference Engine Design for USN Based Wetland Context Inference . . . . So-Young Im, Ryum-Duck Oh, and Yoon-Cheol Hwang Metadata Management System for Wetland Environment Context-Aware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jun-Yong Park, Joon-Mo Yang, and Ryum-Duck Oh

265

275

XIV

Table of Contents – Part II

Toward Open World and Multimodal Situation Models for Sensor-Aware Web Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yoshitaka Sakurai, Paolo Ceravolo, Ernesto Damiani, and Setsuo Tsuruta

285

Design of Intelligence Mobile Cloud Service Platform Based Context-Aware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hyokyung Chang and Euiin Choi

294

Similarity Checking of Handwritten Signature Using Binary Dotplot Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Debnath Bhattacharyya and Tai-hoon Kim

299

Brain Tumor Detection Using MRI Image Analysis . . . . . . . . . . . . . . . . . . . Debnath Bhattacharyya and Tai-hoon Kim

307

Image Data Hiding Technique Using Discrete Fourier Transformation . . . Debnath Bhattacharyya and Tai-hoon Kim

315

Design of Guaranteeing System of Service Quality through the Verifying Users of Hash Chain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yoon-Su Jeong, Yong-Tae Kim, and Gil-Cheol Park

324

Design of RSSI Signal Based Transmit-Receiving Device for Preventing from Wasting Electric Power of Transmit in Sensor Network. . . . . . . . . . . Yong-Tae Kim, Yoon-Su Jeong, and Gil-Cheol Park

331

User Authentication in Cloud Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . Hyokyung Chang and Euiin Choi

338

D-PACs for Access Control and Profile Storage on Ubiquitous Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Changbok Jang and Euiin Choi

343

An Approach to Roust Control Management in Mobile IPv6 for Ubiquitous Integrate Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Giovanni Cagalaban and Byungjoo Park

354

A Software Framework to Associate Multiple FPMN MSISDNs with a HPMN IMSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dongcheul Lee and Byung Ho Rhe

360

Uplink Interference Adjustment for Mobile Satellite Service in Multi-beam Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ill-Keun Rhee, Sang-Am Kim, Keewan Jung, Erchin Serpedin, Jong-Min Park, and Young-Hun Lee Protecting Computer Network with Encryption Technique: A Study . . . . Kamaljit I. Lakhtaria

371

381

Table of Contents – Part II

A Study on the Strategies of Foreign Market Expansion for Korean IT Venture Company . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Woong Eun and Yong-Seok Seo Binding Update Schemes for Inter-domain Mobility Management in Hierarchical Mobile IPv6: A Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Afshan Ahmed, Jawad Hassan, Mata-Ur-Rehman, and Farrukh Aslam Khan

XV

391

411

A Pilot Study to Analyze the Effects of User Experience and Device Characteristics on the Customer Satisfaction of Smartphone Users . . . . . Bong-Won Park and Kun Chang Lee

421

Exploring the Optimal Path to Online Game Loyalty: Bayesian Networks versus Theory-Based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . Nam Yong Jo, Kun Chang Lee, and Bong-Won Park

428

The Effect of Users’ Characteristics and Experiential Factors on the Compulsive Usage of the Smartphone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bong-Won Park and Kun Chang Lee

438

Leadership Styles, Web-Based Commitment and Their Subsequent Impacts on e-Learning Performance in Virtual Community . . . . . . . . . . . . Dae Sung Lee, Nam Young Jo, and Kun Chang Lee

447

Test Case Generation for Formal Concept Analysis . . . . . . . . . . . . . . . . . . . Ha Jin Hwang and Joo Ik Tak

457

Mobile Specification Using Semantic Networks . . . . . . . . . . . . . . . . . . . . . . . Haeng-Kon Kim

459

A Novel Image Classification Algorithm Using Swarm-Based Technique for Image Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Noorhaniza Wahid An Optimal Mesh Algorithm for Remote Protein Homology Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Firdaus M. Abdullah, Razib M. Othman, Shahreen Kasim, and Rathiah Hashim Definition of Consistency Rules between UML Use Case and Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Noraini Ibrahim, Rosziati Ibrahim, Mohd Zainuri Saringat, Dzahar Mansor, and Tutut Herawan Rough Set Approach for Attributes Selection of Traditional Malay Musical Instruments Sounds Classification . . . . . . . . . . . . . . . . . . . . . . . . . . Norhalina Senan, Rosziati Ibrahim, Nazri Mohd Nawi, Iwan Tri Riyadi Yanto, and Tutut Herawan

460

471

498

509

XVI

Table of Contents – Part II

Pairwise Protein Substring Alignment with Latent Semantic Analysis and Support Vector Machines to Detect Remote Protein Homology . . . . . Surayati Ismail, Razib M. Othman, and Shahreen Kasim Jordan Pi-Sigma Neural Network for Temperature Prediction . . . . . . . . . . Noor Aida Husaini, Rozaida Ghazali, Nazri Mohd Nawi, and Lokman Hakim Ismail Accelerating Learning Performance of Back Propagation Algorithm by Using Adaptive Gain Together with Adaptive Momentum and Adaptive Learning Rate on Classification Problems . . . . . . . . . . . . . . . . . . . Norhamreeza Abdul Hamid, Nazri Mohd Nawi, Rozaida Ghazali, and Mohd Najib Mohd Salleh Developing an HCI Model: An Exploratory Study of Featuring Collaborative System Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Saidatina Fatimah Ismail, Rathiah Hashim, and Siti Zaleha Zainal Abidin Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

526 547

559

571

581

Table of Contents – Part I

A SOA-Based Service Composition for Interactive Ubiquitous Entertainment Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Giovanni Cagalaban and Seoksoo Kim

1

A Study on Safe Reproduction of Reference Points for Recognition on Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sungmo Jung and Seoksoo Kim

7

Maximized Energy Saving Sleep Mode in IEEE 802.16e/m . . . . . . . . . . . . Van Cuong Nguyen, Van Thuan Pham, and Bong-Kyo Moon

11

Propose New Structure for the Buildings Model . . . . . . . . . . . . . . . . . . . . . . Tuan Anh Nguyen gia

23

Solving Incomplete Datasets in Soft Set Using Parity Bits of Supported Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ahmad Nazari Mohd. Rose, Hasni Hassan, Mohd Isa Awang, Tutut Herawan, and Mustafa Mat Deris An Algorithm for Mining Decision Rules Based on Decision Network and Rough Set Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hossam Abd Elmaksoud Mohamed A Probabilistic Rough Set Approach to Rule Discovery . . . . . . . . . . . . . . . Hossam Abd Elmaksoud Mohamed

33

44 55

A Robust Video Streaming Based on Primary-Shadow Fault-Tolerance Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bok-Hee Ryu, Dongwoon Jeon, and Doo-Hyun Kim

66

Oversampled Perfect Reconstruction FIR Filter Bank Implementation by Removal of Noise and Reducing Redundancy . . . . . . . . . . . . . . . . . . . . . Sangeeta Chougule and Rekha P. Patil

76

Designing a Video Control System for a Traffic Monitoring and Controlling System of Intelligent Traffic Systems . . . . . . . . . . . . . . . . . . . . . Il-Kwon Lim, Young-Hyuk Kim, Jae-Kwang Lee, and Woo-Jun Park

91

Approximate Reasoning and Conceptual Structures . . . . . . . . . . . . . . . . . . Sylvia Encheva

100

XVIII

Table of Contents – Part I

Edge Detection in Grayscale Images Using Grid Smoothing . . . . . . . . . . . Guillaume Noel, Karim Djouani, and Yskandar Hamam

110

Energy-Based Re-transmission Algorithm of the Leader Node’s Neighbor Node for Reliable Transmission in the PEGASIS . . . . . . . . . . . . Se-Jung Lim, A.K. Bashir, So-Yeon Rhee, and Myong-Soon Park

120

Grid-Based and Outlier Detection-Based Data Clustering and Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kyu Cheol Cho and Jong Sik Lee

129

Performance Comparison of PSO-Based CLEAN and EP-Based CLEAN for Scattering Center Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . In-Sik Choi

139

A Framework of Federated 3rd Party and Personalized IPTV Services Using Network Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Md. Motaharul Islam, Mohammad Mehedi Hassan, and Eui-Nam Huh Faults and Adaptation Policy Modeling Method for Self-adaptive Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ingeol Chun, Jinmyoung Kim, Haeyoung Lee, Wontae Kim, Seungmin Park, and Eunseok Lee Reducing Error Propagation on Anchor Node-Based Distributed Localization in Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . Taeyoung Kim, Minhan Shon, Mihui Kim, Dongsoo S. Kim, and Hyunseung Choo Attention Modeling of Game Excitement for Sports Videos . . . . . . . . . . . . Huang-Chia Shih

147

156

165

175

Discovering Art in Robotic Motion: From Imitation to Innovation via Interactive Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M´ aria Virˇc´ıkov´ a and Peter Sinˇc´ ak

183

I/O Performance and Power Consumption Analysis of HDD and DRAM-SSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hyun-Ju Song and Young-Hun Lee

191

A Measurement Study of the Linux Kernel for Android Mobile Computer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Raheel Ahmed Memon and Yeonseung Ryu

197

RF Touch for Wireless Control of Intelligent Houses . . . . . . . . . . . . . . . . . . David Kubat, Martin Drahansky, and Jiri Konecny

206

Table of Contents – Part I

Facing Reality: Using ICT to Go Green in Education . . . . . . . . . . . . . . . . . Robert C. Meurant

XIX

211

A Network Coding Based Geocasting Mechanism in Vehicle Ad Hoc Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tz-Heng Hsu, Ying-Chen Lo, and Meng-Shu Chiang

223

An In-network Forwarding Index for Processing Historical Location Query in Object-Tracking Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . Chao-Chun Chen and Chung-Bin Lo

233

Embedding Methods for Bubble-Sort, Pancake, and Matrix-Star Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jong-Seok Kim, Mihye Kim, Hyun Sim, and Hyeong-Ok Lee

243

Development of a 3D Virtual Laboratory with Motion Sensor for Physics Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ji-Seong Jeong, Chan Park, Mihye Kim, Won-Keun Oh, and Kwan-Hee Yoo

253

An Adaptive Embedded Multi-core Real-Time System Scheduling . . . . . . Liang-Teh Lee, Hung-Yuan Chang, and Wai-Min Luk

263

Query Processing Systems for Wireless Sensor Networks . . . . . . . . . . . . . . Humaira Ehsan and Farrukh Aslam Khan

273

Educational Principles in Constructivism for Ubiquitous Based Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sung-Hyun Cha, Kum-Taek Seo, and Gi-Wang Shin

283

A Study on Utilization of Export Assistance Programs for SMEs and Their Exportation Performance in Korea . . . . . . . . . . . . . . . . . . . . . . . . . . . . Woong Eun, Sangchun Lee, Yong-Seok Seo, and Eun-Young Kim

290

The Entrance Authentication and Tracking Systems Using Object Extraction and the RFID Tag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dae-Gi Min, Jae-Woo Kim, and Moon-Seog Jun

313

A Privacy Technique for Providing Anonymity to Sensor Nodes in a Sensor Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jeong-Hyo Park, Yong-Hoon Jung, Hoon Ko, Jeong-Jai Kim, and Moon-Seog Jun Study on Group Key Agreement Using Two-Dimensional Array in Sensor Network Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Seung-Jae Jang, Young-Gu Lee, Hoon Ko, and Moon-Seog Jun

327

336

XX

Table of Contents – Part I

Design and Materialization of Location Based Motion Detection System in USN Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joo-Kwan Lee, Jeong-Jai Kim, and Moon-Seog Jun Realization of Integrated Service for Zoos Using RFID/USN Based Location Tracking Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jae-Yong Kim, Young-Gu Lee, Kwang-Hyong Lee, and Moon-Seok Jun

350

361

Real-Time Monitoring System Using Location Based Service . . . . . . . . . . Jae-Hwe You, Young-Gu Lee, and Moon-Seog Jun

369

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

381

A Smart Error Protection Scheme Based on Estimation of Perceived Speech Quality for Portable Digital Speech Streaming Systems Jin Ah Kang and Hong Kook Kim School of Information and Communications Gwangju Institute of Science and Technology (GIST), Gwangju 500-712, Korea {jinari,hongkook}@gist.ac.kr

Abstract. In this paper, a smart error protection (SEP) scheme is proposed to improve speech quality of a portable digital speech streaming (PDSS) system via a lossy transmission channel. To this end, the proposed SEP scheme estimates the perceived speech quality (PSQ) for received speech data, and then transmits redundant speech data (RSD) in order to assist speech decoder to reconstruct lost speech signals for high packet loss rates. According to the estimated PSQ, the proposed SEP scheme controls the RSD transmission, and then optimizes a bitrate of speech coding to encode the current speech data (CSD) against the amount of RSD without increasing transmission bandwidth. The effectiveness of the proposed SEP scheme is finally demonstrated using adaptive multirate-narrowband (AMR-NB) and ITU-T Recommendation P.563 as a scalable speech codec and a PSQ estimator, respectively. It is shown from experiments that a PDSS system employing the proposed SEP scheme significantly improves speech quality under packet loss conditions. Keywords: Portable digital speech streaming systems, packet loss, error protection, perceived speech quality, redundant speech transmission.

1 Introduction Due to the rapid development of Internet protocol (IP) networks over the past few decades, audio and video streaming services are increasingly available via the Internet. Moreover, as these services are extended to wireless networks, the quality of service (QoS) of audio and video streaming is becoming even more critical. Specifically, portable digital speech streaming (PDSS) systems require a minimum level of speech communication quality, where the speech quality is largely related to the network conditions such as packet losses or end-to-end packet delays [1]. When the speech streaming is performed via user datagram protocol/IP (UDP/IP) networks, however, packets may be lost or arrive too late for playback due to inevitable delays. In this case, a typical PDSS system can only tolerate a few packet losses for real-time services, where these packet losses frequently occur in wireless networks due to bandwidth fluctuations [2]. T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 1–10, 2011. © Springer-Verlag Berlin Heidelberg 2011

2

J.A. Kang and H.K. Kim

Several packet loss recovery methods, implemented via the Internet and wireless networks, have been proposed for the speech streaming systems. For instance, the techniques proposed in [3] and [4] were sender-based packet loss recovery methods using forward error correction (FEC). In regards to wireless networks, the techniques proposed in [5] and [6] were based on unequal error protection (UEP) methods. In addition, the modified discrete cosine transform (MDCT) coefficients of audio signals were used as the redundant data in order to assist an MP3 audio decoder to reconstruct lost audio signals [7]. However, these methods did not take into account time-varying network conditions, i.e., packet loss rate (PLR). That is, in order to recover the lost packets based on the conventional FEC methods, the redundant data should be designed to be transmitted constantly even if the network conditions are declared as no packet losses. Therefore, a smart error protection (SEP) scheme is needed that recovers packet losses efficiently according to the time-varying characteristic of PLR. Towards this goal, this paper proposes an SEP scheme that transmits redundant speech data (RSD) adaptively according to the estimation of perceived speech quality (PSQ). To this end, the PSQ estimation is performed in real-time for received speech data by using a single-ended speech quality assessment. Simultaneously, the PLR is estimated by a moving average method. In addition, a real-time transport protocol (RTP) payload format is newly suggested as a means of supporting the proposed SEP scheme. In other words, a speech packet combines the bitstreams of the current speech data (CSD) and the RSD when the PLR is assumed to be high by the estimated PSQ and PLR. Thus, even if a speech packet is lost, the speech decoder can reconstruct the lost speech signal by using the RSD bitstreams from the previous packet. On the other hand, when the PLR is assumed to be low by the estimated PSQ and PLR, a speech packet is organized using the CSD bitstreams alone that are encoded by a higher bitrate. The effectiveness of the proposed SEP scheme is finally demonstrated by using the adaptive multirate-narrowband (AMR-NB) speech codec [8] and ITU-T Recommendation P.563 [9] as a scalable speech codec and a single-ended speech quality assessment, respectively. The remainder of this paper is organized as follows. Following this introduction, Section 2 presents the structure of a PDSS system based on the proposed SEP scheme and the RTP payload format for the proposed SEP scheme. Next, Section 3 describes the proposed SEP scheme in detail, and the performance of the proposed SEP scheme is discussed in Section 4. Finally, Section 5 concludes this paper.

2 A Portable Digital Speech Streaming System 2.1 Overview A PDSS system extends traditional speech communication services over a public switched telephone network (PSTN) to wireless networks in order to provide various mobile communication services. To this end, the PDSS system samples a continuous speech signal to discontinuous speech frames, and it encodes the speech frames to bitstreams at a lower bitrate by using a compression algorithm. Then, it transmits the bitstreams using a real-time streaming protocol after packetizing. Meanwhile, at the

A SEP Scheme Based on Estimation of PSQ for Portable Digital Speech Streaming Systems

3

Subsystem A Sender side Input Speech

Scalable Speech Encoder

RTP Payload Formatting

RTP Packetizing

Output Packet

RSD transmission request Receiver side Output Speech

PLR PSQ Estimation Estimation

Speech Frames Buffering

Scalable Speech Decoder

RTP Payload Analysis

RTP Unpacketizing

Input Packet

IP Network

Subsystem B Sender side Output Packet

RTP Packetizing

RTP Payload Formatting

Scalable Speech Encoder

Input Speech

PSQ PLR Estimation Estimation

Output Speech

RSD transmission request Receiver side Input Packet

RTP Unpacketizing

RTP Payload Analysis

Scalable Speech Decoder

Speech Frames Buffering

Fig. 1. Packet flow for a PDSS system employing the proposed SEP scheme, where Subsystems A and B represent the two communication parties

opposite PDSS system, the arriving packets are unpacketized to bitstreams, and the bitstreams are decoded to the speech frames. Finally, these speech frames are sent to an output device. Fig. 1 shows a packet flow for the PDSS system implemented in this paper, where Subsystems A and B represent both parties of the speech stream communication employing the proposed SEP scheme. First, the sender side of Subsystem A performs scalable speech encoding for the input speech frame. Next, the sender side generates a packet according to an RTP payload format, where the packet includes the CSD bitstreams with the decision result whether or not the RSD transmission is needed. Note here that the RSD bitstreams should be incorporated in this payload when the RSD transmission is requested by Subsystem B. After that, the formatted RTP packet is transmitted. Meanwhile, as the RTP packet arrives at the receiver side of Subsystem B, the receiver side analyzes the received packet according to the RTP payload format, and then extracts the CSD bitstreams and the decision result. In the case that the RTP payload format includes the RSD bitstreams, the RSD bitstreams are used to recover a

4

J.A. Kang and H.K. Kim

lost packet in the future. Next, the extracted CSD bitstreams are decoded using a scalable speech decoder and the decoded speech frames are stored in a speech buffer to be used for the PSQ estimation. Finally, the decision result regarding the RSD transmission is inserted into a RTP packet before a speech frame is sent to Subsystem A. 2.2 RTP Payload Format As mentioned in Section 2.1, a PDSS system employing the proposed SEP scheme can have an indicator for a scalable bitrate of speech coding. Moreover, in order to deliver the feedback information from Subsystem A to Subsystem B, and vice versa, there should be any fields reserved in the format to accommodate the transmission of RSD bitstreams and feedback information. Thus, we first select the RTP payload format defined in IETF RFC 3267 for the AMR-NB speech codec [10], as shown in Fig. 2. 1

0 CMR F

FT

2 Q F

FT

Q

4 byte

3 …

Speech frame 1 … Speech frame N

…

P P P P

Fig. 2. Example of the RTP payload format for AMR-NB speech codec defined in RFC 3267

In the payload format, an ‘F|FT|Q’ sequence of control fields is used to describe each speech frame. Note here that a codec mode request (CMR) field is applied to the entire speech frame. In other words, a one-bit F field indicates whether this frame is to be followed by another speech frame (F=1) or if it is the final speech frame (F=0). In addition, an FT field, comprised of 4 bits, then indicates if this frame is actually coded by a speech encoder or if it is just comfort noise. That is, a number in this field is assigned from 0 to 7, corresponding to encoding bitrates of 4.75, 5.15, 5.90, 6.70, 7.40, 7.95, 10.2, and 12.2 kbit/s, respectively. However, if comfort noise is encoded, the assigned number ranges from 8 to 11. Note that the number 15 indicates the condition that there is no data to be transmitted, and that the numbers 12 to 14 are reserved for future use. Next, a Q field, indicating the speech quality with one-bit, is set at 0 when the speech frame data is severely damaged; otherwise, it is set at 1. Finally, the CMR field, comprised of 4 bits, is used to deliver a mode change signal to the speech encoder. For example, it is set to one out of eight encoding modes, corresponding to different bitrates of AMR-NB speech codec. At the end of the payload, P fields are used to ensure octet alignment. In order to realize the proposed SEP scheme in this payload format, two new frame indices for the RSD bitstreams and the feedback information are incorporated into the FT field, which are denoted using the numbers 12 and 13, respectively. The use of the RTP payload format described above has several advantages. First, the control ability for a speech encoder, such as the CMR field, is retained by using the

A SEP Scheme Based on Estimation of PSQ for Portable Digital Speech Streaming Systems

5

RTP payload format for the speech codec employed in the implemented PDSS system. Next, the overhead of the control fields for each RSD bitstreams is required to be as small as 6 bits in ‘F|FT|Q’. Finally, no additional transport protocol for the RSD transmission request is needed since this feedback is conducted using RTP packets that are used to deliver the speech bitstreams. Therefore, the transmission overhead for the RSD transmission request is significantly reduced, compared to existing transport protocols designed for feedback such as the RTP control protocol (RTCP) [11].

3 Proposed Smart Error Protection Scheme 3.1 Packet Loss Recovery and PSQ Estimation at the Receiver Side Fig. 3 presents the procedure of packet loss recovery with the PSQ estimation at the receiver side of a PDSS system employing the proposed SEP scheme. First, a packet loss occurrence is verified through RTP packet analysis. Then, the received CSD bitstreams are decoded if it is decided that there is no packet loss. On the other hand, if it is decided that there is a packet loss, the lost speech signals are recovered by using the RSD bitstreams or by using the packet loss concealment (PLC) algorithm in the speech decoder, depending on the availability of the RSD bitstreams. Finally, the speech decoder reconstructs the speech frame data from the CSD bitstreams, and estimate PSQ and PLR with speech data once the amount of speech frames is enough to estimate a PSQ score. Start

RTP Analysis

Packet Loss Verification

True

Lost Speech Recovery

False Feedback Verification

True

Speech Coding Bitrate Control

False RSD Verification

True

RSD Buffer

False CSD Decoding

NRX >= N False

Speech Buffer

True

PSQ Estimation PLR Estimation

End

Fig. 3. Procedure of the packet loss recovery with the PSQ estimation at the receiver side

6

J.A. Kang and H.K. Kim N fra mes

sˆ( m)

sˆ( m + 1)

Speech Buffer

sˆ(m + N −1)

.. . sˆ(m + N − P )

sˆ( m + N − P + 1)

…

sˆ(m + N −1)

…

sˆ( m + 2 N − P − 1)

P frames

Fig. 4. Overlap of speech frames for the PSQ estimation at the receiver side

For the PSQ estimation, the speech data in a speech buffer are used by overlapping, as shown in Fig. 4. In the figure, sˆ( m ) is the m-th speech frame input to the speech buffer, N is the total number of frames to be used for the PSQ estimation, and P is the number of frames to be overlapped for the next PSQ estimation. In other words, the PSQ estimation is conducted when every (N-P) frames are newly received from the opposite PDSS system. In addition, the estimated PLR, Lˆ ( k ), is obtained by moving average for the previous PLR, L( k − 1), with the average PLR, L ( 0 : k − 1), as (1)

Lˆ ( k ) = ( 1 − α ) L ( 0 : k − 1 ) + α L ( k − 1 )

Finally, it is decided whether or not requesting the RSD transmission by comparing the estimated PSQ and PLR with each threshold. That is, the request for the RSD transmission, RSD (k ), is set to true or false according to the equation of ⎧ true , RSD ( k ) = ⎨ ⎩ flase ,

if Qˆ ( k ) ≤ Thres

1

and

Lˆ ( k ) ≥ Thres

2

(2)

otherwise

where Qˆ ( k ) is the estimated PSQ score, and Thres 1 and Thres 2 are threshold for Qˆ ( k ) and Lˆ ( k ) , respectively. 3.2 Scalable Speech Coding and RSD Transmission at the Sender Side Fig. 5 shows the procedure how to transmit scalable speech coding bitstreams and the RSD bitstreams at the sender side for the proposed SEP scheme. First, for given feedback information transmitted from the opposite PDSS streaming system, the sender side verifies the request for the RSD transmission and changes the bitrate of scalable speech coding according to the request. In other words, when the RSD transmission is not requested, the bitrate is set at the highest bitrate and then the CSD bitstreams are encoded alone with no additional RSD bitstreams. On the other hand, when the RSD transmission is requested, the bitrate is set at smaller bitrate than the current bitrate in order to assign the remaining bitrate for the RSD transmission. Thus, both of the CSD and RSD bitstreams are encoded. Finally, after the RTP payload format described in Section 2.2 is configured according to such adaptive RSD transmission, the RTP packets are transmitted to the opposite PDSS system.

A SEP Scheme Based on Estimation of PSQ for Portable Digital Speech Streaming Systems

7

Start

RSD Request Verification

True

Encoding of CSD and RSD

False Encoding of CSD

RTP Formatting

End

Fig. 5. Procedure of the scalable speech coding and the adaptive RSD transmission at the sender side

As described above, we can several advantages of the proposed SEP scheme as follows. First, the adaptive operation of the packet loss recovery according to the network conditions is effective since burst packet losses generally occur when the network is congested due to a sudden increase in the amount of data coming to the network [4]. Second, compared to the conventional redundant data transmission (RDT) methods that require additional network overhead, the proposed SEP scheme generates redundant data without increasing the transmission bandwidth by controlling the bitrate of a scalable speech codec. Third, in order to estimate the network conditions, the proposed SEP scheme conducts the estimation of PSQ. This is motivated by the fact that the PSQ measured as a mean opinion score (MOS) can be considered to be a clearer indicator of the speech quality than other parameters in the PDSS system.

4 Performance Evaluation In order to demonstrate the effectiveness of the proposed SEP scheme, the PDSS system was first implemented by using the AMR-NB speech codec and the ITU-T Recommendation P.563 as a scalable speech codec and a PSQ estimator, respectively. Here, the speech signals were sampled at 8 kHz, and then encoded using the AMRNB speech codec operated at 10.2 kbit/s. Thus, when the RSD transmission was needed, the bitrate of the CSD and RSD was set at 4.75 kbit/s, which was almost half the bitrate of 10.2 kbit/s. By considering the requirements of the ITU-T Recommendation P.563, N in Fig.4 was set to 200 frames for the PSQ estimation, which corresponded to 4 seconds. Moreover, P was set to 150 frames, thus the PSQ estimation was conducted whenever every new 50 frames were received. For the PLR estimation, we carried out performance evaluation of the proposed SEP scheme with the different value of α in Eq. (1), and then we set α to 0.4. Similarly, the thresholds for the estimated PSQ and PLR in Eq. (2), Thres 1 and Thres 2 , were set to 4.0 MOS and 5%, respectively.

8

J.A. Kang and H.K. Kim

In the test, 48 speech files from NTT-AT speech database [12] were used, where each speech file was about 4 seconds long and sampled at a rate of 16 kHz. These speech signals were first filtered using a modified intermediate reference system (IRS) filter followed by an automatic level adjustment [13]. Then, the speech signals were down-sampled from 16 to 8 kHz. In order to show the effectiveness of the proposed SEP scheme under different PLRs including burst loss characteristics, we generated five different PLR patterns of 3, 5, 7, 9 and 11% by using the Gilbert-Elliot channel model defined in the ITU-T Recommendation G.191 [13]. Here, the burstiness of the packet losses was set at 0.5, and the mean and maximum consecutive packet losses were measured at 1.5 and 4.0 frames, respectively. In order to demonstrate the effectiveness of the proposed SEP scheme, the speech quality of the PDSS system with the proposed SEP scheme was compared to that of the PDSS system with the regular RDT. The regular RDT was designed to transmit the RSD regularly for each speech frame by encoding the CSD and RSD bitstreams at a bitrate of 4.75 kbit/s in order to evaluate the performance without increasing transmission bandwidth. In addition, the speech quality of the PDSS system by the PLC algorithm without the proposed SEP scheme or the regular RDT was also evaluated, where the PLC algorithm was operated at the highest bitrate of 10.2 kbit/s. Note here that the PLC algorithm embedded in the AMR-NB speech decoder was always applied without regarding to the RSD transmission. As the evaluation method for the recovered speech quality, the perceptual evaluation of speech quality (PESQ) defined in the ITU-T Recommendation P.862 [14] was used. Table 1. Speech quality measured in MOS using PESQ for the different packet loss recovery methods, under PLRs ranging from 3 to 11% MOS Score Method Without the regular RDT With the regular RDT With the proposed scheme

0 3.70 3.14 3.70

3 3.16 3.07 3.16

PLR (%) 5 7 2.97 2.83 2.97 2.91 2.93 2.87

9 2.61 2.80 2.71

11 2.52 2.78 2.72

Average 2.96 2.94 3.01

Table 1 compares speech quality measured in MOS using PESQ for the different packet loss recovery methods, under PLRs ranging from 3 to 11%. As shown in the table, the proposed SEP scheme first improved the speech quality for low PLRs as the PLC algorithm without using regular RDT did. In addition, the proposed SEP scheme provided better performance than without the regular RDT as the regular RDT did for high PLRs. Consequently, the proposed SEP scheme yielded the average speech quality of 3.01 MOS, which was 0.07 MOS higher than the regular RDT.

5 Conclusion In this paper, we proposed a new smart error protection (SEP) scheme that guaranteed the speech quality without increasing transmission bandwidth for a portable digital speech streaming system (PDSS). To this end, the proposed SEP scheme was

A SEP Scheme Based on Estimation of PSQ for Portable Digital Speech Streaming Systems

9

designed to transmit redundant speech data (RSD) according to the estimation results for the perceived speech quality (PSQ) and packet loss rate (PLR), where a singleended speech quality assessment and a moving average method were used to estimate PSQ and PLR, respectively. The proposed SEP scheme was applied to the receiver and sender sides of a PDSS system. In other words, the receiver side of the PDSS system first decided the RSD transmission based on the estimation of PSQ and PLR, and then sent feedback information on the decision result to the opposite PDSS system via real-time transport protocol (RTP) packets for speech bitstreams. On the other hand, the sender side of the PDSS system controlled the RSD transmission according to the received feedback, and subsequently optimized the speech coding bitrate in order to maintain the equivalent transmission bandwidth despite of the RSD bitstreams. Finally, we evaluated the speech quality recovered by the proposed SEP scheme under PLRs and compared it with that of the conventional redundant data transmission (RDT) method. From the results, the proposed SEP scheme improved the speech quality from 2.94 to 3.01 MOS compared than the conventional method for the PLRs ranged from 3% to 11%. Consequently, the proposed SEP scheme could be applied to the PDSS streaming systems in order to improve the speech quality degraded due to packet losses efficiently. Acknowledgments. This work was supported in part by the “Fusion-Tech Developments for THz Information & Communications” Program of the Gwangju Institute of Science and Technology (GIST) in 2011, by the Mid-career Researcher Program through the National Research Foundation of Korea (NRF) grant funded by the Korea government (MEST) (No. 2010-0000135), and by the MKE (The Ministry of Knowledge Economy), Korea, under the ITRC (Information Technology Research Center) support program supervised by the NIPA (National IT Industry Promotion Agency) (NIPA-2010-C1090-1021-0007).

References 1. Wu, C.-F., Lee, C.-L., Chang, W.-W.: Perceptual-based playout mechanisms for multistream voice over IP networks. In: Proceedings of Interspeech, Antwerp, Belgium, pp. 1673–1676 (September 2007) 2. Zhang, Q., Wang, G., Xiong, Z., Zhou, J., Zhu, W.: Error robust scalable audio streaming over wireless IP networks. IEEE Transactions on Multimedia 6(6), 897–909 (2004) 3. Bolot, J.-C., Fosse-Parisis, S., Towsley, D.: Adaptive FEC-based error control for Internet telephony. In: Proceedings of IEEE International Conference on Computer Communications (INFOCOM), New York, NY, pp. 1453–1460 (March 1999) 4. Jiang, W., Schulzrinne, H.: Comparison and optimization of packet loss repair methods on VoIP perceived quality under bursty loss. In: Proceedings of 12th International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), Miami, FL, pp. 73–81 (May 2002) 5. Yung, C., Fu, H., Tsui, C., Cheng, R.S., George, D.: Unequal error protection for wireless transmission of MPEG audio. In: Proceedings of IEEE International Symposium on Circuits and Systems (ISCAS), Orlando, FL, pp. 342–345 (May 1999) 6. Hagenauer, J., Stockhammer, T.: Channel coding and transmission aspects for wireless multimedia. Proceedings of the IEEE 87, 1764–1777 (1999)

10

J.A. Kang and H.K. Kim

7. Ito, A., Konno, K., Makino, S.: Packet loss concealment for MDCT-based audio codec using correlation-based side information. International Journal of Innovative Computing, Information and Control 6, 3(B), 1347–1361 (2010) 8. ETSI 3GPP TS 26.101: Adaptive Multi-Rate (AMR) Speech Codec Frame Structure (January 2010) 9. ITU-T Recommendation P.563: Single-Ended Method for Objective Audio Quality Assessment in Narrow-Band Telephony Applications (May 2004) 10. IETF RFC 3267: Real-Time Transport Protocol (RTP) Payload Format and File Storage Format for the Adaptive Multi-Rate (AMR) and Adaptive Multi-Rate Wideband (AMRWB) Audio Codecs (June 2002) 11. IETF RFC 1889: RTP: A Transport Protocol for Real-Time Applications (January 1996) 12. NTT-AT: Multi-Lingual Speech Database for Telephonometry (1994) 13. ITU-T Recommendation G.191: Software Tools for Speech and Audio Coding Standardization (November 1996) 14. ITU-T Recommendation P.862: Perceptual Evaluation of Speech Quality (PESQ), an Objective Method for End-to-End Speech Quality Assessment of Narrowband Telephone Networks and Speech Codecs (February 2001)

MDCT-Domain Packet Loss Concealment for Scalable Wideband Speech Coding Nam In Park and Hong Kook Kim School of Information and Communications Gwangju Institute of Science and Technology (GIST), Gwangju 500-712, Korea {naminpark,hongkook}@gist.ac.kr

Abstract. In this paper, we propose a modified discrete cosine transform (MDCT) based packet loss concealment (PLC) algorithm in order to improve the quality of decoded speech when a packet loss occurs in scalable wideband speech coders using MDCT as spectral parameters. The proposed PLC algorithm is realized by smoothing MDCT coefficients between the low and high bands for scalable wideband speech coders. In G.729.1, a typical scalable wideband speech coder standardized by ITU-T, two different PLC algorithms are applied to low band and high band in time and frequency domain, respectively. Thus, the MDCT coefficients around the boundary between the low and high band can be mismatched. The proposed PLC algorithm is replaced with the PLC algorithm applied to the high band, and it compensates for the mismatch in the MDCT domain at the boundary. Finally, we compare the performance of the proposed PLC algorithm with that of the PLC algorithm employed in G.729.1 by means of perceptual evaluation of speech quality (PESQ), an A-B preference test, and a waveform comparison under different random and burst packet loss conditions. It is shown from the experiments that the proposed PLC algorithm provides significantly better speech quality than the PLC of G.729.1. Keywords: Packet loss concealment (PLC), wideband speech coding, modified discrete cosine transform (MDCT), G.729.1.

1 Introduction With the increasingly popular use of the Internet, IP telephony devices such as voice over IP (VOIP) and voice over WiFi (VoWiFi) phones have attracted wide attention for speech communications. In IP phone services, speech packets are typically transmitted using a real-time transport protocol/user datagram protocol (RTP/UDP), though RTP/UDP does not verify whether the transmitted packets are correctly received [1]. Due to the nature of this type of transmission, the packet loss rate would become higher as the network becomes congested. In addition, depending on the network resources, the possibility of burst packet losses also increases, potentially resulting in severe quality degradation of the reconstructed speech [2]. Most speech coders in use today are based on telephone-bandwidth narrowband speech, nominally limited to about 200-3400 Hz and sampled at a rate of 8 kHz. On T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 11–18, 2011. © Springer-Verlag Berlin Heidelberg 2011

12

N.I. Park and H.K. Kim

the contrary, the wideband speech coders have been developed for the purpose of smoothly migrating from narrowband to wideband quality (50-7,000 Hz) at a rate of 16 kHz in order to improve speech quality in services. That is, voice services using wideband speech not only increase the intelligibility and naturalness of speech, but also add feeling of transparent communications. Especially, G.729.1, a scalable wideband speech coder, improves the quality of speech by encoding the frequency bands left out by the narrowband speech coder, G.729. Therefore, encoding wideband speech using G.729.1 is performed in two different approaches which are applied to low band and high band in time and frequency domain, respectively. In particular, when a frame loss occurs, the low band and the high band PLC algorithm work separately. In other words, the low band PLC algorithm reconstructs excitation and spectral parameters of the lost frame from the last good frame. Also, the high band PLC algorithm reconstructs spectral parameters, e.g., typically modified discrete cosine transform (MDCT) coefficients, of the lost frame from the last good frame. Of course, the excitation of the low band could be used for reconstructing the excitation of the high band if bandwidth extension from the low band to the high band is employed in wideband speech decoding [3][4]. In this paper, we propose a modified discrete cosine transform (MDCT) based packet loss concealment (PLC) algorithm in order to improve the quality of decoded speech when a packet loss occurs in a scalable wideband speech coder using MDCT as spectral parameters. Here, we select ITU-T Recommendation G.729.1, which is a typical MDCT-based scalable wideband speech coder employing G.729 as a narrowband speech coder. Since two different PLC algorithms in G.729.1 are applied to low band and high band in time and frequency domain, respectively, the MDCT coefficients around the boundary between the low and high band can be mismatched. The proposed PLC algorithm is replaced with the PLC algorithm applied to the high band, and it compensates for the mismatch in the MDCT domain at the boundary. The remainder of this paper is organized as follows. Following this introduction, Section 2 describes a conventional PLC algorithm currently employed in the G.729.1 decoder [5]. After that, Section 3 proposes an MDCT-based PLC algorithm that is implemented by replacing the conventional PLC algorithm in G.729.1. Section 4 then demonstrates the performance of the proposed PLC algorithm, and this paper is concluded in Section 5.

2 Conventional PLC Algorithm The PLC algorithm employed in the G.729.1 standard reconstructs speech signals of the current frame based on previously received speech parameters such as excitation in the low band and the MDCT coefficients in high band, which is shown in Fig. 1 [5]. In other words, the PLC algorithm replaces the missing excitation and MDCT coefficients with an equivalent characteristic from a previously received frame, while the excitation energy gradually decays. In addition, for frame error correction (FEC) it uses a voicing classifier based on the parameter C, which is a signal classification such as voice, unvoice, transition, and silence. During the frame error concealment process for the low band, the gain or the energy parameter, E, is adjusted according to C. Next, the synthesis filter for the lost frame uses the linear predictive coding (LPC)

MDCT-Domain Packet Loss Concealment for Scalable Wideband Speech Coding

13

coefficients of the last good frame. Similarly, the pitch period of the lost frame uses the integer part of the pitch period from the previous frame. To avoid becoming desynchronized, the phase parameter, Presync, is used for the recovery after the lost voice onset period.

Fig. 1. Overview of the G.729.1 PLC algorithm

During the frame error concealment process for the high band, the high-band signals of the previous good frame are applied to the time domain bandwidth extension (TDBWE) by using the excitation generated by the low-band PLC. Next, the MDCT coefficients of previous good frame are used to generate the high-band signals. Finally, the decoded speech for the lost frame is generated through a quadrature mirror filter (QMF) synthesis, which consists of the 64-tap filter coefficients.

3 Proposed PLC Algorithm As shown in Fig. 1, according to a 4-kHz boundary frequency, the PLC algorithm for the G.729.1 speech coder consists of low-band and high-band PLCs. Note that the PLC algorithm for regenerating the low-band signals is executed in the time domain, whereas the PLC algorithm for the high-band signals is executed in the frequency domain. Since different PLC algorithms are applied for the low- and high-bands, a frequency mismatch [7] could occur.

14

N.I. Park and H.K. Kim

Contrary to this conventional PLC algorithm, the proposed algorithm is designed as shown in Fig. 2. In the figure, the PLC of the low-band signal is equivalent to the conventional PLC algorithm. On the other hand, in the high-band PLC, the synthesized low-band signal is transformed into the frequency domain by using MDCT to smooth the boundary frequency between the low- and high-bands. In this case, the weighting filter [8] for smoothing the frequency is given by ′ ( k ) = 0.6 ⋅ S high ( k ) + 0.4 ⋅ S avg , S high

k = 0,1,L,39

(1)

′ (k ) are the low-band signal, high-band signal, and where Slow (k ), S high (k ), and S high

the smoothed high-band signal in the MDCT frequency-domain, respectively, and k is the MDCT frequency bin with a range of 0 to 159. S avg is the frequency average at the boundary frequency (4 kHz) and denoted as Savg =

1 2

(Slow (159) + Shigh (0)) . After

passing through the weighting filter, the smoothed signal is transformed into the time domain by applying an inverse modified discrete cosine transform (IMDCT), as shown in the figure. Finally, the decoded speech for the lost frame is recovered through QMF synthesis, which consists of the 64-tap filter coefficients.

Fig. 2. Overview of the proposed PLC algorithm

MDCT-Domain Packet Loss Concealment for Scalable Wideband Speech Coding

15

4 Performance Evaluation

4.1 4 3.9 3.8 3.7 3.6 3.5 3.4 3.3

G.729.1-PLC Proposed PLC PESQ (MOS)

PESQ (MOS)

To evaluate the performance of the proposed PLC algorithm, we replaced the PLC algorithm currently employed in G.729.1 [5] with the proposed PLC algorithm, and then obtained perceptual evaluation of the speech quality (PESQ) scores according to ITU-T Recommendation P.862 [8]. For the PESQ test, 76 audio files were taken from the SQAM audio database [9] and processed by G.729.1 using the proposed PLC algorithm under different packet loss conditions. The performance was also compared to that using the PLC algorithm employed in G.729.1, referred to here as G.729.1PLC. In this paper, we simulated two different packet loss conditions, including random and burst packet losses. During these simulations, packet loss rates of 3, 5, and 8% were generated by the Gilbert-Elliot model defined in ITU-T Recommendation G.191 [10]. Under the burst packet loss condition, the burstiness of the packet losses was set to 0.66. Thus, the mean and maximum consecutive packet losses were measured at 1.5 and 3.7 frames, respectively.

No loss 3 5 Single Packet Loss Rate (%)

4.1 4 3.9 3.8 3.7 3.6 3.5 3.4 3.3

8

Proposed PLC PESQ (MOS)

PESQ (MOS)

3 5 Single Packet Loss Rate (%)

8

(b) G.729.1-PLC

No loss 0 0.33 0.66 Burstiness (Packet Loss Rate =3%)

(c)

Proposed PLC

No loss

(a) 4.1 4 3.9 3.8 3.7 3.6 3.5 3.4 3.3

G.729.1-PLC

4.1 4 3.9 3.8 3.7 3.6 3.5 3.4 3.3

G.729.1-PLC Proposed PLC

No loss 0 0.33 0.66 Burstiness (Packet Loss Rate =3%)

(d)

Fig. 3. Comparison of PESQ scores of the proposed PLC and G.729.1-PLC for (a) speech data and (b) music data under single packet loss conditions, and for (c) speech data and (d) music data under burst packet loss conditions

Fig. 3 compares the PESQ scores when the proposed PLC and G.729.1-PLC were employed in G.729.1 under single packet loss conditions and burst packet loss conditions at a packet loss rate of 3%, respectively. It was shown from the figure that the proposed PLC algorithm had PESQ scores comparable to the G.729.1-PLC algorithm for all conditions in the case of the speech data. However, the

16

N.I. Park and H.K. Kim

effectiveness of the proposed PLC algorithm was investigated when packet losses occurred in audio data such as music. In order to evaluate the subjective performance, we performed an A-B preference listening test, in which 6 speech files (3 males and 3 females) and 2 music files were processed by both G.729.1-PLC and the proposed PLC under random and burst packet loss conditions. Table 1 shows the A-B preference test results. As shown in the table, the proposed PLC was significantly preferred than G.729.1-PLC. Table 1. A-B preference test results

(a)

Amplitude

Burstiness/Packet loss rate 3% γ = 0.0 5% (random) 8% 3% γ = 0.66 5% 8% Average

Preference Score (%) G.729.1-PLC No difference 15.62 46.88 12.08 56.67 21.88 45.31 18.75 51.56 14.06 54.69 15.63 57.81 16.34 52.15

Proposed PLC 37.50 31.25 32.81 29.69 31.25 26.56 31.51

1 0

-1

8.1

8.2

8.3

8.4

8.5

8.6

8.7

8.8

8.9

9.0

8.1

8.2

8.3

8.4

8.5

8.6

8.7

8.8

8.9

9.0

8.1

8.2

8.3

8.4

8.5

8.6

8.7

8.8

8.9

9.0

8.1

8.2

8.3

8.4

8.5

8.6

8.7

8.8

8.9

9.0

8.1

8.2

8.3

8.4

8.5

8.6

8.7

8.8

8.9 9.0 Time (sec)

1

(b)

0 -1 1

(c) 0 1

(d)

0 -1

1

(e)

0 -1

Fig. 4. Waveform comparison: (a) original waveform, (b) decoded speech signal with no packet loss, and reconstructed speech signals using (c) packet error patterns, (d) G.729.1-PLC, and (e) proposed PLC

MDCT-Domain Packet Loss Concealment for Scalable Wideband Speech Coding

17

Finally, Fig. 4 compares the waveform comparison of speech reconstructed by different PLC algorithms. Figs. 4(a) and 4(b) show the original speech waveform and the decoded speech waveform with no loss of the original signal, respectively. After applying the packet error pattern (expressed as a solid box in Fig. 4(c)), the proposed PLC (Fig. 4(e)) reconstructed the speech signals better than in G.729.1-PLC (Fig. 4(d)).

5 Conclusion In this paper, we proposed a packet loss concealment algorithm for the G.729.1 speech coder to improve the performance of speech quality when frame erasures or packet losses occurred. To this end, we proposed an MDCT-based approach, where MDCT coefficients between the low and high bands were smoothed in order to improve the quality of decoded speech. Next, we evaluated the performance of the proposed PLC algorithm on G.729.1 under random and burst packet loss rates of 3, 5, and 8%, and then compared it with that of the PLC algorithm already employed in G.729.1 (G.729.1-PLC). It was shown from the comparison of PESQ scores, A-B preference, and waveforms that the proposed PLC algorithm provided similar or better speech quality than G.729.1-PLC for all the simulated conditions. Acknowledgments. This work was supported in part by the “Fusion-Tech Developments for THz Information & Communications” Program of the Gwangju Institute of Science and Technology (GIST) in 2011, by the Mid-Career Researcher Program through an NRF grant funded by MEST, Korea (No. 2010-0000135), and by the Ministry of Knowledge Economy (MKE), Korea, under the Information Technology Research Center (ITRC) support program supervised by the National IT Industry Promotion Agency (NIPA) (NIPA-2010-C1090-1021-0007).

References 1. Goode, B.: Voice over internet protocol (VoIP). Proceedings of the IEEE 90(9), 1495– 1517 (2002) 2. Jian, W., Schulzrinne, H.: Comparison and optimization of packet loss repair methods on VoIP perceived quality under bursty loss. In: Proceedings of NOSSDAV, pp. 73–81 (2002) 3. Gournay, P., Rousseau, F., Lefebvre, R.: Improved packet loss recovery using late frames for prediction-based speech coders. In: Proceedings of ICASSP, pp. 108–111 (2003) 4. Tommy, V., Milan, J., Redwan, S., Roch, L.: Efficient frame erasure concealment in predictive speech codecs using glottal pulse resynchronisation. In: Proceedings of ICASSP, pp. 1113–1116 (2007) 5. Rogot, S., Kovesi, B., Trilling, R., Virette, D., Duc, N., Massaloux, D., Proust, S., Geiser, B., Gartner, M., Schandl, S., Taddei, H., Yang, G., Shlomot, E., Ehara, H., Yoshida, K., Vaillancourt, T., Salami, R., Lee, M.S., Kim, D.Y.: ITU-T G.729.1: an 8-32 kbit/s scalable coder interoperable with G.729 for wideband Telephony and voice over IP. In: Proceedings of ICASSP, pp. 529–532 (2007)

18

N.I. Park and H.K. Kim

6. Taleb, A., Sandgren, P., Johansson, I., Enstrom, D., Bruhn, S.: Partial spectral loss concealment in transform coders. In: Proceedings of ICASSP, pp. 185–188 (2005) 7. ETSI ES 202 050, v1.1.3.: Speech Processing, Transmission and Quality Aspects (STQ); Distributed Speech Recognition; Advanced Front-End Feature Extraction Algorithm; Compression Algorithm (2003) 8. ITU-T Recommendation P.862. Perceptual Evaluation of Speech Quality (PESQ), and Objective Method for End-to-End Speech Quality Assessment of Narrowband Telephone Networks and Speech Coders (2001) 9. EBU Tech Document 3253: Sound Quality Assessment Material, SQAM (1998) 10. ITU-T Recommendation G.191: Software Tools for Speech and Audio Coding Standardization (2000)

High-Quality and Low-Complexity Real-Time Voice Changing with Seamless Switching for Digital Imaging Devices Sung Dong Jo1, Young Han Lee1, Ji Hun Park1, Hong Kook Kim1, Ji Woon Kim2, and Myeong Bo Kim2 1

School of Information and Communications Gwangju Institute of Science and Technology(GIST), Gwangju 500-712, Korea {sdjo,cpumaker,jh_park,hongkook}@gist.ac.kr 2 Camcorder Business Team, Digital Imaging Business Samsung Electronics, Suwon-si, Gyenggi-do 443-742, Korea {jiwoon.kim,kmbo.kim}@samsung.com

Abstract. In this paper, we propose a voice changing method to provide a seamless switchable function with a low computational complexity for digital imaging devices. The proposed method combines a waveform similarity overlap-and-add (WSOLA) algorithm with a sampling rate changing technique that operates in the time domain. In addition, the proposed method includes a noise technique in the region where the voice changing switching mode changes from on to off, and vice versa. We finally compare the performance of the proposed method with that of a conventional one in terms of the processing time and speech quality. It is shown from the experiments that the proposed voice changing method gives a relative complexity reduction of 84.5% in a resource-constrained device having an ARM processor and is more preferred than the conventional method by 76%. Keywords: Voice changing, time-scale modification, waveform similarity overlap-and-add (WSOLA), sampling rate change, digital imaging device.

1 Introduction Voice changing is a process of transforming the characteristics of speech uttered by a speaker such that a listener would not recognize the speaker [1]. This technique is useful for disguising the speaker`s voice. Applications of a voice changing method are numerous, including text-to-speech synthesis based on acoustical unit concatenation, transformation of voice characteristics to disguise the speaker`s voice, foreign language learning, audio monitoring, film/soundtrack post-synchronization, and so on [2]. The basic mechanisms behind the voice changing process often consist of source modification, filter modification, and the combination of source and filter modification [3]. Source modification tries to modify prosody of speech acoustic, thus it is usually referred to as prosodic modification. Source modification is classified into three different types such as time-scale modification (TSM), pitch-scale modification (PSM), T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 19–27, 2011. © Springer-Verlag Berlin Heidelberg 2011

20

S.D. Jo et al.

and energy modification [3]. On the other hand, filter modification tries to modify the magnitude spectrum of the frequency response of the vocal tract system [4]. The fundamental frequency of the vocal fold vibration is denoted as F0, and the perceptual feature of speech corresponding to F0 is often called “pitch” [5]. Because pitch periods generally associated with male speakers are quite different from those associated with female speakers, a modification of speech is related with controlling pitch periods, i.e., PSM. However, a PSM algorithm affects the articulation rate of the original signal, which results in changing the apparent gender of the speaker [5]. Therefore, this can be compensated by matching the original articulation rate if we use TSM instead of PSM for the voice changing. A TSM is a technique used to modify the duration of speech or audio signals, while minimizing the distortion of other important characteristics such as the pitch and timbre [2]. A TSM algorithm has been widely used in the fields of speech and audio signal processing. For example, it has been used during preprocessing in speech recognition systems to improve the recognition rate [6]. Also, TSM can be applied to speech synthesis systems in order to produce sounds more naturally [7]. In this paper, we propose a voice changing method for a real-time implementation on a digital imaging device. The proposed voice changing method combines a waveform similarity overlap-and-add (WSOLA) TSM algorithm [8] with a sampling rate changing technique that operates in the time domain. The main concern of the proposed method is how to minimize the computational complexity since the resource of a digital imaging device is highly limited. In addition, when the voice changing can be on or off in the device, a seamless on/off mode control with a low computational complexity should be available. To this end, we give an effort on the complexity reduction of the WSOLA algorithm as well as an efficient control of the voice changing mode. Next, we incorporate a noise reduction technique in the proposed method when the voice changing mode is changed from on to off or vice versa. The organization of this paper is as follows. Following this introduction, we shortly review conventional voice changing methods in Section 2. After that, we propose a voice changing method and discuss how to reduce the computational complexity of the voice changing method in Section 4. In Section 5, we compare the performance of the proposed method with that of a conventional one in terms of the computational complexity and speech quality. Finally, we conclude this paper in Section 6.

2 Basic Mechanisms of Voice Changing Methods Voice changing methods modify the sound that is produced when a person speaks or sings. In this section, we shortly review the basic mechanisms consisting of source modification, filter modification, and the combination of source and filter modifications. 2.1 Source Modification Source modification deals with the modification of prosodic aspects of speech such as rhythm, intonation, and stress, and is classified into TSM, PSM, or energy modification, as mentioned in Section 1.

High-Quality and Low-Complexity Real-Time Voice Changing

21

First of all, TSM enables us to change the apparent rate of articulation without affecting the perceptual quality of the original speech [2]. This means that the formant structure is changed at a slower or faster rate than the rate of the input speech. Thus, TSM has been used to improve the compression rate in speech and audio coding [7]. Over the last several decades, various forms of TSM algorithms have been developed. Among them, synchronized overlap-and-add (SOLA) [9], pitch synchronous overlapand-add (PSOLA) [10], and waveform similarity overlap-and-add (WSOLA) [8] show relatively a good performance in regards to output quality. Second, PSM alters the fundamental frequency in order to compress or expand the harmonic components in the spectrum while preserving the short-time spectral envelope as well as the time evolution [3]. However, it changes the local pitch, which affects the original articulation rate, while TSM maintains the original articulation rate. Third, energy modification modifies the perceived loudness of the input speech [3]. It is considered to be the simplest modification in source modification. For example, signals are just multiplied by a scale factor to amplify or attenuate. 2.2 Filter Modification It is widely accepted that magnitude spectrum carries information of a speaker’s individuality. Therefore, by modifying the magnitude spectrum of the vocal tract, speaker identity can be controlled [3]. There are two types of filter modification: one without a specific target and the other with a specific target [3]. In the case where there is no specific target, the magnitude spectrum is modified in a general way without having a specific target speaker. For example, we want to modify the overall quality of a speech signal produced by a female voice so that it sounds as if it were produced by an older female speaker. However, in the case where there is a specific target, the filter of a source speaker is modified in a way that the modified filter approximates the characteristics of the filter according to another targeted speaker. Usually, we refer to this type of modification as voice conversion. To obtain the transformed spectrum, there is a learning process using the source and target spectrum to make the transformed spectrum equal to the average spectrum of the target spectrum during a training process [3][4]. 2.3 Combination of Source and Filter Modification The prosody characteristics of a speaker become a critical cue used for the identification of the speaker. At the same time, the vocal tract characteristics are also important for identification. Therefore, if we want to modify the voice of a speaker so that it sounds like the voice of another speaker, prosody and vocal tract modifications should be combined. If a specific target speaker is provided, then this becomes another type of voice conversion. On the other hand, if no specific target is provided, it is usually referred to as voice modification [3]. Voice morphing is one example of combined source and filter modification. For example, the same sentences are uttered by two source speakers and then a third speaker can be generated having characteristics from both source speakers by applying dynamic time warping (DTW) [11] between the two sentences.

22

S.D. Jo et al.

3 Proposed Voice Changing Method In this section, we propose a voice changing method using WSOLA to reduce the computational complexity. In addition, the proposed voice changing method includes a noise reduction technique, which is useful for the transient region where voice changing is from on to off or vice versa. Fig. 1 shows the procedure of the proposed method, where input signals are stereo, sampled at a rate of 32 kHz with 16-bit resolution.

Fig. 1. Procedure of the proposed voice changing method

As shown in the figure, the proposed voice changing method mainly consists of a sampling rate changing block and a WSOLA-based TSM block. The sampling rate changing block changes the pitch of the input signals by increasing or decreasing the number of input samples according to a given voice changing level. However, this block affects the articulation rate of the original signal, as PSM does. In order to compensate for mismatching the articulation rate, the WSOLA-based TSM algorithm is applied to the signals from the sampling rate changing block, which will be explained in Section 3.1. In this paper, the proposed voice changing method is designed to provide a realtime seamless voice changing on/off mode. When the mode is on, the WSOLA algorithm is fully applied. On the other hand, instead of the WSOLA algorithm, we simply apply an overlap-and-add technique when the mode is off. In the case when the mode switches from on to off or vice versa, a seamless on/off mode control is required due to noise in such transient region in the time domain, which will be discussed in Section 3.2. 3.1 WSOLA-Based Time-Scale Modification The WSOLA-based TSM algorithm, or simply WSOLA algorithm, uses a waveform similarity measure to eliminate the phase distortion of overlap-and-add [8]. The

High-Quality and Low-Complexity Real-Time Voice Changing

23

WSOLA algorithm determines the best spot between an input frame and a reference frame to be overlapped in order to maintain the natural continuity of the input signal. The synthesis equation of the WSOLA algorithm is defined as [8] ∑ v(n − k ⋅ D) x(n + k ⋅ D ⋅ α − k ⋅ D + Δk ) y(n) = k ∑ v( n − k ⋅ D ) k

(1)

where x(n ), y (n ), and v (n ) are an input signal, its corresponding time-scaled output signal, and a window signal, respectively. In addition, D indicates the overlap-andadd (OLA) length and α indicates a time-scale factor. If α is less than 1, the output signal is time-expanded. Otherwise the output signal is time-compressed. In Eq. (1), Δ k represents an optimal shift of the k-th frame. The optimal shift is measured by the following equation of

Δ k = arg max[corr ( R (n ), CΔ ( n ))] Δ

(2)

where corr ( R (n ), CΔ (n )) represents the normalized cross-correlation between the reference signal, R(n), and a candidate signal, CΔ (n ), for a search range, − Δ max ≤ Δ ≤ + Δ max . That is, the normalized cross-correlation is represented as 2 L −1

corr( R( n ), CΔ ( n )) =

∑ R( n )CΔ ( n )

n =0 2 L −1 2

2 L −1

n =0

n =0

∑ R (n)

∑

(3) CΔ2 ( n )

In the above equation, R (n ) = v (n − (k − 1) D ) x (n + (k − 1) ⋅ D ⋅ α + Δ k −1 + D ) and CΔ ( n ) = v (n − k ⋅ D ) x( n + k ⋅ D ⋅ α + Δ ). 3.2 Noise Reduction between on/off Transition

In order to remove the noise the voice changing mode is switched from on to off or vice versa, a Hanning window is applied to both the original signal stored in the buffer and the voice changed signal by the WSOLA algorithm. Then the two signals are merged by using the following equation of

⎧s (n ) w( N / 2 + n ) + sBUF ( n ) w( n ) , n < N / 2 s' (n ) = ⎨ VE n ≥ N /2 ⎩s BUF (n ),

(4)

where w(n ) = 0.5(1 − cos( N2π−n1 )) . In addition, the window length, N , is set as the half

length of an output frame length. In Eq. (4), sVE (n ) is the n-th sample of voice

24

S.D. Jo et al.

changed signal processed by the WSOLA algorithm, s BUF (n ) is the n-th sample of the signal stored in a buffer, and s' (n ) is the n-th sample of the output signal. Fig. 2 compares the spectrogram of a voice changed speech signal processed by a conventional method and that by the proposed method with the noise reduction technique in the transient regions. As shown in Fig. 2 (a), we can observe the noise spectrum at the boundary that the voice changing mode is changed from on to off. However, the kind of noise has been disappeared in Fig. 2 (b).

Fig. 2. Comparison of spectrograms processed by (a) the conventional method and (b) the proposed method

4 Complexity Reduction for Efficient Implementation on a Resource-Constrained Device In this section, we discuss a method of reducing the computational complexity for real-time implementation of the proposed voice changing method on a resourceconstrained device. Especially, we control the processing step in the WSOLA algorithm when the voice changing mode is switched on or off. Fig. 3 shows a flowchart of reducing the complexity reduction method. First of all, input signals are segmented into frames and then the proposed voice changing method is initialized for every frame. After that, one frame signal is stored in a buffer, where the buffer size is equal to the algorithmic delay. When the voice changing mode is on, the WSOLA algorithm is fully applied to the input signal. So, the cross-correlation between the signals from a present frame and the next frame is calculated and the optimal shift spot is found to maintain the natural continuity. However, when the voice changing mode is off, we skip the computations for both the cross-correlation calculation and the optimal shift finding in the WSOLA algorithm. But just the overlap-and-add technique is applied using the input frame signal stored in a buffer. This procedure enables us to reduce the computational complexity when the voice changing is off.

High-Quality and Low-Complexity Real-Time Voice Changing

25

Fig. 3. Flowchart for reducing the computational complexity of the proposed voice changing method with an on/off control

5 Performance Evaluation In this section, we evaluated the performance of the proposed voice changing method in terms of processing time and speech quality after implementing it on a resourceconstrained device. Table 1 describes the specifications of the device. That is, it is equipped with an embedded operating system (OS) and an ARM processor with a CPU clock of 133 MHz and a RAM size of 17 MB. Table 2 compares the computational complexity between the conventional method and the proposed method with the voice changing mode. Here, the average processing time was measured by using stereo speech data of 300 seconds long, sampled at a rate of 32 kHz. As shown in the table, the proposed method yielded a processing time reduction of 84.5% when the mode is off, compared to when the mode is on. Table 3 shows the preference test results of the proposed method over the conventional method. To this end, five people with no hearing disabilities participated in the test. Two speech files processed by the conventional method and the proposed

26

S.D. Jo et al.

method were presented to the participants, and the participants were asked to choose their preference. If they felt no difference between two files, they were guided to select ‘no difference.’ As shown in the table, the proposed method was preferred by 76%, compared to the conventional method. Table 1. Specifications of a resource-constrained device in which the proposed voice changing method is implemented Item Embedded OS CPU Clock RAM

Specification ARM 133 MHz 17 MB

Table 2. Computational complexity comparison between the conventional method and the proposed method when the voice changing mode is on or off Item Average processing time at on mode Average processing time at off mode Percentage of complexity reduction

Conventional method 11.82 ms 11.81 ms 0.08%

Proposed method 11.81 ms 1.83 ms 84.5%

Table 3. Preference test results

Preference

Conventional method

No difference

Proposed method

4%

20%

76%

6 Conclusion In this paper, we proposed a voice changing method to provide a seamless switchable function with a low computational complexity for digital imaging devices. The proposed method combined the WSOLA algorithm with a sampling rate changing technique that operated in the time domain. In order to reduce the computational complexity, we controlled the processing step in the WSOLA algorithm according to the voice changing mode. In addition, the proposed method was designed to include a noise reduction technique in the region where the voice changing switching mode changed from on to off or vice versa. We compared the performance of the proposed method with that of a conventional one in terms of the processing time and speech quality. It was shown from the experiments that the proposed voice changing method gave a relative complexity reduction of 84.5% on an ARM processor and was more preferred than the conventional method by 76%. Acknowledgments. This work was supported in part by Samsung Electronics Co., and by the Global Frontier project (No. 2010-0029751) of MEST in Korea.

High-Quality and Low-Complexity Real-Time Voice Changing

27

References 1. Salor, Ö., Demirekler, M.: Dynamic programming approach to voice transformation. Speech Communication 48(10), 1262–1272 (2006) 2. Moulines, E., Laroche, J.: Non-parametric techniques for pitch-scale and time-scale modification of speech. Speech Communication 16(2), 175–205 (1995) 3. Stylianou, Y.: Voice transformation: a survey. In: Proceedings of ICASSP, pp. 3585–3588 (2009) 4. Benesty, J., Sondhi, M., Huang, Y.: Handbook of Speech Processing. Springer, Heidelberg (2007) 5. Vergin, R., O’Shaughnessy, D., Farhat, A.: Time domain technique for pitch modification and robust voice transformation. In: Proceedings of ICASSP, pp. 947–950 (1997) 6. Roucos, S., Wilgus, A.: High quality time-scale modification for speech. In: Proceedings of ICASSP, pp. 493–496 (1986) 7. Wayman, J., Wilson, D.: Some improvements on the synchronized-overlap-add method of time scale modification for use in real-time speech compression and noise filtering. IEEE Transactions on Acoustics, Speech, and Signal Processing 36(1), 139–140 (1988) 8. Verhelst, W., Roelands, M.: An overlap-add technique based on waveform similarity (WSOLA) for high quality time-scale modification of speech. In: Proceedings of ICASSP, pp. 554–557 (1993) 9. Hardam, E.: High quality time scale modification of speech signals using fast synchronized overlap add algorithms. In: Proceedings of ICASSP, pp. 409–412 (1990) 10. Moulines, E., Charpentier, F.: Pitch-synchronous waveform processing techniques for textto-speech synthesis using diphones. Speech Communication 9(5-6), 453–467 (1990) 11. Keogh, E., Pazzani, M.: Derivative dynamic time warping. In: Proceedings of 1st SIAM International Conference on Data Mining, pp. 1–11 (2001)

Complexity Reduction of Virtual Reverberation Filtering Based on Index-Based Convolution for Resource-Constrained Devices Kwang Myung Jeon1, Nam In Park1, Hong Kook Kim1, Ji Woon Kim2, and Myeong Bo Kim2 1

School of Information and Communications Gwangju Institute of Science and Technology (GIST), Gwangju 500-712, Korea {kmjeon,naminpark,hongkook}@gist.ac.kr 2 Camcorder Business Team, Digital Imaging Business Samsung Electronics, Suwon-si, Gyeonggi-do 443-742, Korea {jiwoon.kim,kmbo.kim}@samsung.com

Abstract. Virtual reverberation effects are a vital part of virtual audio reality. Reverberation effects can be directly applied by implementing a convolution process between the input audio and a reverberation filter response that characterizes a virtual space. In order to apply reverberation effects, however, additional or dedicated processors are required for practical implementation due to the excessively long impulse response of the reverberation filter. In this paper, we propose a fast method for applying virtual reverberation effects based on a reverberation filter approximation and an index-based convolution process. Throughout exhaustive experiments, we attempt to optimize the proposed method in terms of satisfaction of the reverberation effect and its computational requirements. We then implement three different types of virtual reverberation functions in a resource-constrained digital imaging device. It is shown that the virtual reverberation effects implemented by the proposed approach are able to operate in real-time with less than 5ms latency, with an over 80% overall satisfaction score in the subjective preference test. Keywords: Virtual reverberation, sparseness of impulse response, index-based convolution, audio effects.

1 Introduction Reverberation is a very common phenomenon in our life. Whether we speak in a classroom or listen to musical performances in a concert hall, the sounds we hear contain delayed reflections from many different directions based on the characteristics of the room. Hence, virtual reverberation effects, which reflect physical room characteristics, are a vital part of virtual audio reality. Using reverberation, an unaffected recorded sound can be transformed to the sound as if it was recorded in a large room, a musical hall, or a wet bathroom. This ability to apply the reverberation effects of desired rooms is especially useful in audio productions T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 28–38, 2011. © Springer-Verlag Berlin Heidelberg 2011

Complexity Reduction of Virtual Reverberation Filtering

29

for movies, PC games, and in virtual reality applications, where most source sounds are recorded in a studio with no inherent reverberation effects. Reverberation can be generated by multiple feedback delay circuits to create echo signals to make artificial reverberations. In digital signal processing, the multiple feedback delay circuits are realized using a reverberation filter whose impulse response contains delayed responses by considering the positions of sound sources, listening spots, and the characteristics of the room we want to realize. Then, a conventional convolution process between the input audio and that reverberation filter response is performed [1]. Many digital signal processing algorithms, including the image method [2], have focused on the design of the reverberation filter response. However, they cannot be directly implemented in most resource-constrained devices due to the high computational requirement which is derived by the convolution process with excessively long impulse response of the reverberation filter. For this reason, reverberation effects are commonly applied via additional processors or by using a hardware dedicated to fast convolution of reverberation processes [3][4]. It should be noted, however, that the implementation of these strategies is still limited in resource-constrained devices due to both cost and implementation issues. To resolve the computational problem of applying reverberation effects using the convolution process, we propose an approximation approach of a reverberation filter and a new convolution process, so-called index-based convolution, in order to reduce the overall computational requirements.

2 Review of Virtual Reverberation Filtering The application of virtual reverberation effects is based on two major steps; a reverberation filter generation and a convolution process between the input audio and the generated reverberation filter. The reverberation filter generation is based on the simulation of room acoustics [2][6]. A more sophisticated way of modeling reverberation has previously been proposed by considering several factors such as the positional information of sound sources in a room and the acoustic absorption of a room surface, the humidity, and the air temperature [5]. However, the reverberation characteristics of a well-modeled filter can be distorted by the following approximation step. Depending on the computational capability of the device to be implemented, the degree of the approximation can be varied. Since implementing reverberation effects on a resource-constrained device requires a high approximation ratio, the well-modeled reverberation filter is approximated in a way of preserving the strong characteristics of the well-modeled filter. Fig. 1 shows an example of the virtual sound source in a 2-dimensional space derived by the image method. The cross and the asterisk symbol in the figure represent the source sound position and the listening spot of the virtual listener, respectively. In addition, the black circles represent the virtual sound positions perceived as a reverberation. In this process, a main issue regarding the reverberation filter generation is how to calculate the virtual sound positions to reflect the positional information to the filter.

30

K.M. Jeon et al.

Fig. 1. Example of the virtual sound source in 2-dimensional space derived by the image method

Fig. 2. Illustration of calculating the virtual sound position in a 1-dimensional space

Fig. 2 shows the simplified method of calculating the virtual sound position in a 1dimensional space. In the figure, the circle, x s , and x r represent the recording position, the distance between the recording position and the source sound position, and the distance between the recording position and the wall of the modeled room, respectively. Then, the i-th virtual sound position in a 1-dimensional space is denoted as

⎛ 1 − ( −1)i xi = ( −1)i xs + ⎜ i + ⎜ 2 ⎝

⎞ ⎟ x r − xm ⎟ ⎠

(1)

where xm is the distance between the recording and listening positions. Note that the concept describe above can also be extended to find the virtual sound position in the y- and z-direction. In other words, we can obtain the j-th and k-th virtual sound positions in the y- and z-direction as y j and zk , respectively, by using Eq. (1). Thus,

the distance between the recording position and the ijk-th virtual sound source in a 3dimensional space, d ijk , is represented as

d ijk = xi2 + y 2j + zk2 .

(2)

In Eq. (2), d ijk is used to derive the unit impulse response for the reverberation filter. First of all, the impulse response of each virtual sound position is represented as

Complexity Reduction of Virtual Reverberation Filtering

⎧1, if d ijk = tc aijk = ⎨ ⎩0, otherwise

31

(3)

where t is the time delay of the echo and c is the speed of sound which is given by the conditions of the room’s medium. aijk is the unit impulse response at time t . Second, magnitude of each unit impulse response is computed by taking into account the wall’s reflectivity and d ijk . For the given distance, d ijk , and position (i, j, k), the magnitude of each impulse response, eijk , is calculated as

eijk = rijk bijk i+ j+k

where rijk = rw

(4)

and bijk ∝ 1 d ijk . In other words, bijk is distance coefficient

which is inversely proportional to d ijk , and rijk is the room’s reflection coefficient by assuming that every wall surrounding the room has the same reflection coefficient defined as rw . Finally, the reverberation filter containing the characteristics of the virtual room is represented as h (t ) = ∑ ∑ ∑ aijk eijk .

(5)

i j k

After the reverberation filter is generated, a linear convolution process between the input data and the reverberation filter is carried out to generate the reverberation effects for the output data. However, the major problem in this simple process is the excessive computational requirement, which comes from the long filter response range. To overcome this problem, a fast method that reduces the computational requirements is proposed in the following section.

3 Fast Implementation of Virtual Reverberation The proposed method for applying reverberation effects consists of two steps such as filter generation and filter application step. Fig. 3 shows an overall procedure of applying reverberation effects using the proposed method. Note that the filter generation step includes reverberation filter generation and its approximation. In this section, we assume that a reverberation filter is generated by using the procedure described in Section 2.

Fig. 3. Procedure of applying reverberation effects using the proposed method

32

K.M. Jeon et al.

3.1 Reverberation Filter Approximation

The impulse response of the generated reverberation filter has significantly long duration. Typically, around a hundred thousand or more durational impulse response is required to achieve the desired reverberation effects. Thus, the conventional convolution process between the input data and the reverberation filter causes significantly computational burden. To reduce computational burden while maintaining the desired reverberation effects, we propose a new convolution process when the data are sparse, which is called an index-based convolution process in this paper and will be discussed in the next subsection. The index-based convolution process is originated from the fact that the convolution should be actually done only for non-zero data. Therefore, we can approximate the generated reverberation filter such that the number of non-zero values in the response is as small as possible. Reverberation filter approximation is performed by clipping non-zero values into zero by using a predefined threshold. In other words, the approximated reverberation filter, H a ( z ), is obtained as N −1

N −1

i =0

i =0

H a ( z ) = ∑ h (i ) w(i ) z − i = ∑ ha (i ) z − i

(6)

where h(i ) is the i-th value of the impulse response or the i-th filter coefficient of the generated reverberation filter, and N is the duration of the filter. In addition, ha (i ) is the filter coefficient obtained from h(i ) by applying the weighting value defined as ⎧1, w(i ) = ⎨ ⎩0,

if h(i ) ≥ Thr else

, for i = 0,1,L ,n

(7)

where Thr is a predefined threshold and used to control an approximation ratio. Fig. 4(a) shows the impulse response of the generated reverberation filter. For the filter generation, the room size was set to 135m × 180m × 16m, and the source spot and the listening spot was set in the position of (67m, 90m, 10m) and (67m, 45m, 10m), respectively. For the given positions of source and listening spot, ( x r =0, x s =0, x m =0), ( y r =90, y s =45, y m =0), ( z r =0, z s =0, z m =0) in Eq. (1). In addition, rw =0.9 in Eq. (4). It is known from the figure that the generated reverberation filter consists of densely distributed responses with small amplitude and sparsely distributed responses with a distinctive large amplitude in between the small ones. Due to such distribution characteristics of the generated reverberation filter, applying the filter approximation method described in Eqs. (6) and (7) can dramatically reduce the non-zero filter coefficients of the reverberation filter. Fig. 4(b) shows the impulse response of an approximated reverberation filter when Thr =0.18. As mentioned earlier, the approximation does not hurt the performance of the generated reverberation filter. That is, the effectiveness of the virtual reverberation by the generated reverberation filter is perceptually identical to that by the approximated filter. We performed exhaustive informal listening tests and thus it was found that the approximated filter provided somewhat better reverberation effects than the generated filter, which will be discussed in Section 4.1.

Complexity Reduction of Virtual Reverberation Filtering

33

Fig. 4. Comparison of impulse responses; (a) impulse response of the generated reverberation filter and (b) impulse response of an approximated reverberation filter

l := 0; k := 0 repeat if w( k )! = 0 then hI (l ) := ha ( k ); I (l ) := k ; l := l + 1; until k = N − 1 M := l ; Fig. 5. Pseudo-code for obtaining hI (n ) from ha (n )

3.2 Index-Based Convolution

In this subsection, we propose a modified approach of the linear convolution designed to reduce complexity of a filter whose non-zero impulse response is sparsely distributed, which is here referred to as index-based convolution. A key idea of the

34

K.M. Jeon et al.

index-based convolution is to skip the computation at the time when the impulse response of the filter is equal to zero. Therefore, the index-based convolution is very computationally efficient and thus it can be applied to the approximated reverberation filter described in Section 3.1. The index-based convolution can be derived from the linear convolution with the approximated reverberation filter. To begin with, the linear convolution is applied to the approximated filter, ha (n ), from Eqs. (6) and (7), as N −1

N −1

k =0

k =0

y ( n ) = ∑ ha ( k ) x ( n − k ) = ∑ h(k ) w( k ) x( n − k )

(8)

where h (n ) is the impulse response of the generated reverberation filter and N is the duration of the filter. Also, x(n) and y (n) are input and output audio respectively. By properly selecting the threshold in Eq. (8), the number of the actual summation in Eq. (8) can be performed much less then N times. If we ignore the summation in Eq. (8) when w( n ) = 0, then we obtain hI (n ) from ha (n ) as shown in Fig. 5. In the figure, M is the duration of hI (n ), and I (n ) is a position of the n-th non-zero value if the impulse response such that hI ( n ) = h( I ( n )). Thus, Eq. (8) can be rewritten as M −1

y (n ) = ∑ hI (l ) x (n − I (l )) . l =0

(9)

In Eq. (9), we need ( M − 1) additions and M multiplications for each y (n ). Consequently, the computation of the index-based convolution is dominated by M . In total, we need N ( M − 1) additions and NM multiplications for the index-based convolution, which are smaller than N ( N − 1) additions and N 2 multiplications for the conventional convolution. Since M θ 3 and SSER > θ 4 ) ⎧speech, ⎪ ACC (n ) = ⎨mixed signal, if ( SRER θ 2 ) and ( SNER >θ 3 or SSER > θ 4 ) (1) ⎪non - speech, otherwise ⎩

where signal-to-residual energy ratio (SRER), low-to-high band energy ratio (LHER), signal-to-non-speech energy ratio (SNER), and signal-to-speech energy ratio (SSER) are defined as E R ( n) / E ( n), E L (n ) / E H (n), E (n ) / E N (n ), and E ( n) / E S (n ), respectively. In the definition, E N (n) and ES (n) are average energies corresponding to E N (n) and ES (n ), respectively. They are estimated from the frames classified as non-speech and speech, respectively. In other words, if the n-th frame is classified as speech, then ES (n) is updated as

E S ( n ) = α E S ( n − 1) + (1 − α ) E ( n )

(2)

where 0 < α < 1. However, ES (n ) = ES ( n − 1) if the n-th frame is non-speech or mixed-signal. Similarly, E N (n) is updated as E N ( n ) = α E N ( n − 1) + (1 − α ) E ( n ) if the n-th frame is non-speech, but E N (n ) = E N (n − 1) if the n-th frame is speech or mixed-signal. Next, sub-band scaling is conducted to enhance the speech band signals as well as suppress the non-speech band signals. That is, the low band and high band signals obtained from the 2-band QMF analysis are scaled by g L (n ) or g H (n ), respectively. They are estimated using ES (n ) and E N (n) as

g H ( n) = 1 − β

E S ( n) E N (n)

g L (n ) = 2 − g H (n)

(3) (4)

42

J.A. Kang et al.

where β is a constant and used to control the degree of the gain variation. Subsequently, the scaled sub-band signals are synthesized through 2-band QMF synthesis. Finally, a scale factor in the time domain is obtained to further enhance or suppress the synthesized signals according to the ACC result. This scale factor is used to decrease the energy of audio signals classified as mixed signal or non-speech in order to further suppress the noise signals. Otherwise, for audio signals as classified as speech, this scale factor is used to match the energy of synthesized audio signals to the energy of original audio signals in order to maintain the energy of the audio signals enhanced by sub-band scaling as that of the original audio signals. Thus, we re-compute the signal energy from the synthesized signal after applying the sub-band scale factors of Eqs. (3) and (4), which is here denoted as Eˆ (n). The scale factor is then computed as ⎧ E ( n) , if ACC (n) = speech ⎪ˆ ⎪ E ( n) ⎪⎪ E (n) g ( n) = ⎨ S , if ACC (n) = mixed signal ˆ ⎪ k m E (n) ⎪ E S ( n) , if ACC (n) = non - speech ⎪ ˆ ⎪⎩ k n E (n)

(5)

where k m or k n is the estimated SNR from the frames declared as mixed signal or non-speech, respectively. In other words, when the n-th frame is classified as mixed signal or non-speech, g (n ) is determined so that the energy of the output signals decreases up to ES (n) / k m or ES ( n) / kn . On the other hand, when the n-th frame is classified as speech, g (n) is determined so that the energy of the output signals becomes equal to that of the input signals. Consequently, by multiplying g (n) to the signals synthesized by the 2-band QMF, we generate the output signals that highlight speaker’s voice against background noise.

3 Service Scenario Using the Proposed Audio Effect Algorithm for Portable Digital Imaging Devices Fig. 2 shows the procedure for a service scenario using the proposed AE algorithm for portable digital imaging devices. In order to activate the service, a user should select a video file and command a portable digital imaging device to play back the selected file. During the service, a user can also select the proposed AE algorithm optionally by enabling an on/off function implemented on the device. In other words, if the function is on, the proposed AE algorithm operates once every frame and the processed audio data are sent to the output device. Otherwise, audio data are directly brought to the output device.

Audio Effect for Highlighting Speaker’s Voice Corrupted by Background Noise

User

43

System a. Read the unit frame data from audio content in video file

A. Select video file B. Commend to play back

b. Verify on/ off function on

C. Enable on/off function

off

c. Process audio data by the proposed AE algorithm d. Send the processed audio data to the output device

e. Send audio data to the output device

Fig. 2. Service scenario using the proposed AE algorithm for portable digital imaging devices

4 Performance Evaluation In this section, we first discuss how the proposed AE algorithm works by showing the waveforms and parameters for each processing step and then perform an informal listening test for the audio contents before and after applying the proposed AE algorithm. We prepared audio contents which were recorded by a digital camcorder in a stereo format of MPEG AAC at a sampling rate of 32 kHz with a 16 bit resolution. In order to set the thresholds in Eq. (1), we performed exhaustive preliminary experiments with a number of audio contents recorded in noise environments, and we found it out that setting θ1 , θ2 , θ 3, and θ 4 in Eq. (1) to 0.25, 2.0, 1.8, and 0.35, respectively, provided the best performance of the proposed AE algorithm. In addition, other constants were also set as α = 0.5 in Eq. (1), β = 0.05 in Eq. (3), and km = 1.5 and kn = 3.5 in Eq. (5). Fig. 3 shows an example how the proposed AE algorithm highlights speaker’s voice. That is, Fig. 3(a) displays the input audio waveform, where speaker’s voice was corrupted by babble and music noises. Then, we performed the ACC and the result was shown in Fig. 3(b), where 0, 1, and 2 indicated non-speech, mixed signal, and speech, respectively. Next, we estimated sub-band scale factors, g L (n) or g H (n ), and depicted them as a solid line and a dotted line in Fig. 3(c), respectively. By multiplying the sub-band scale factors to the low and high band signal followed by applying the 2-band QMF synthesis, we obtained the signal displayed in Fig. 3(d). After that, we estimated a time-domain scale factor, g (n ) , defined in Eq. (5), from the waveform of Fig. 3(d), which was shown in Fig. 3(e). Finally, we obtained the output signal after multiplying g (n ) to the waveform of Fig. 3(d), which was shown in Fig. 3(f). It was shown from the figure that the proposed AE algorithm classified each frame of the input audio signals to speech, non-speech, or mixed signal as expected, thus background noise was suppressed effectively in the output audio signal.

44

J.A. Kang et al.

(a)

(b)

(c)

(d)

(e)

(f) Fig. 3. Example of the experimental results by the proposed AE algorithm: (a) input audio signal, (b) the classification result, (c) sub-band scale factors (solid line: low band scale factor, dotted line: high band scale factor), (d) synthesized audio signal after multiplying the sub-band scale factors, (e) time-domain scale factor, and (f) output audio signal

Audio Effect for Highlighting Speaker’s Voice Corrupted by Background Noise

45

Finally, to demonstrate the perceptual effectiveness of the proposed AE algorithm, we carried out an informal listening test. Ten participants including seven males and three females participated in this test, and each listener voted a preferred audio content between the original audio content and its processed version by the proposed AE algorithm. Table 1 shows the average preference score for 10 audio contents. As shown in the table, the audio contents processed by the proposed AE algorithm were significantly preferred rather than the original audio contents. Table 1. Average preference score for the audio contents processed by the proposed AE algorithm Original audio contents Preference (%)

5

Audio contents processed by the proposed AE algorithm 95

5 Conclusion In this paper, an audio effect (AE) algorithm was proposed to enhance speech signals corrupted by background noise in audio content and applied to a portable digital imaging device for highlighting speaker’s voice. First of all, the proposed AE algorithm classified each short segment of audio signals as speech, non-speech, or mixed signal, on the basis of the parameters such as signal energy, sub-band energy, and linear prediction residual signal energy. Then, a scale factor for the output signal was adaptively determined depending on the classification and the estimated signalto-noise ratio. It was shown from an informal listening test that audio contents processed by the proposed AE algorithm outperformed the original audio contents. Acknowledgments. This work was supported in part by the Global Frontier project (No. 2010-0029751) of MEST in Korea, and by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the MEST in Korea (2010-0023888).

References 1. http://www.youtube.com/ 2. Boll, S.: Suppression of acoustic noise in speech using spectral subtraction. IEEE Transactions on Acoustics, Speech and Signal Processing 27(2), 113–120 (1979) 3. Lim, J.S., Oppenheim, A.V.: Enhancement and bandwidth compression of noisy speech. Proceedings of the IEEE 67(12), 1587–1604 (1979) 4. ISO/IEC 13818-7: Information Technology - Generic Coding of Moving Pictures and Associated Audio Information - Part 7: Advanced Audio Coding, AAC (December 2004)

Detection of Howling Frequency Using Temporal Variations in Power Spectrum Jae-Won Lee1 and Seung Ho Choi2 1 Graduate School of NID Fusion Technology Seoul National University of Science and Technology, Seoul 139-743, Korea [email protected] 2 Department of Electronic and Information Engineering Seoul National University of Science and Technology, Seoul 139-743, Korea [email protected]

Abstract. Indoor audio feedback is a common problem in many audio amplification systems. The audio feedback can be out of control in some indoor conditions, which results in howling. Also, the howling frequency is subject to variation by the changes of indoor environments. Most conventional methods attempt to eliminate the howling, but they do not provide a way to predict the occurrence of howling. This paper presents a novel method to predict the howling frequency using temporal variations in power spectrum. Keywords: Howling, acoustic feedback circuit, moving average, power spectrum.

1 Introduction Audio amplification systems such as mobile devices and hearing aids include both speaker and microphone, and howling can be generated at some specific frequency due to acoustic feedback circuit (AFC) as shown in Fig. 1 [1-3]. As a positive feedback circuit, AFC can diverge at a particular frequency in some phase condition, and this howling phenomenon restricts the amplitude gain of audio system [2-4]. Conventional methods to suppress the howling control the frequency gain after howling is detected [1]. Least mean square method that is generally used for the howling suppression works after the howling occurs [5-8]. The howling may vary depending on the overall environment such as the positions of microphones and loudspeakers, room shape and arrangement, the position and movement of talker, room temperature, etc. Therefore, it is difficult to predict the howling [9]. This paper presents a novel method to predict the howling frequency using temporal variations in power spectrum. This paper is organized as follows. Howling phenomena and transfer function characteristics are described in Section 2. In addition, relations between indoor environment and howling frequency are described. In Section 3, we describe the proposed howling prediction method that detects howling frequency using temporal variations in power spectrum before howling occurrence. And we conclude this article in Section 4. T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 46–51, 2011. © Springer-Verlag Berlin Heidelberg 2011

Detection of Howling Frequency Using Temporal Variations in Power Spectrum

47

2 Howling Phenomena 2.1 Effects of Indoor Reflections In an electric amplifier system, as shown in Fig. 1(a), the microphone input signal x(t) is amplified by the gain g and then the speaker output signal y t is generated. The y t makes the multiple reflection signals { y t } [2]. And then, the { y t } are summed to the feedback signal y t that is again entered to the microphone, where τ is the delay time of y t and α is the attenuation factor of the indoor wall [10]. The AFC of the single reflection path is shown in Fig. 1(b) and the transfer function H ω is given by Eq. (1). From the magnitude response |H ω | in Eq. (2), we can see that the transfer function is a comb filter that changes periodically [9]. As in Eq. (3), the frequency at H ω 2πm becomes a potential howling frequency (PHF) where m is an integer [10]. In other words, a howling is likely to occur at the PHF.

yi (t) = αi y0 (t − τi )

Y (ω )

X (ω )

y0 (t)

− jτω

αe y ( t ) = ∑ α i y 0 (t − τ i ) n

x(t)

i= 0

(a)

(b)

Fig. 1. AFC by indoor reflections; (a) generation of indoor reflections and (b) transfer function model

H ω

Y ω X ω

g·

|H ω |

H ω

1

αgcos τω j αgsin τω 1 2αgcos τω α g

(1)

g 1 tan

2αgcos τω

α g

αgsin τω . αgcos τω 1

(2)

(3)

The transfer function can be represented by Eq. (4). However, it is difficult to determine the parameters since the transfer function is affected by the environment such as the indoor conditions, the positions of microphone and loudspeaker, and the directivity characteristics [10]. H ω

Y ω X ω

αe

.

(4)

48

J.-W. Lee and S.H. Choi

magnitude[dB]

In fact, the howling frequency changes continuously according to moving devices and circumferential changes. Therefore it is difficult to predict a howling occurrence in advance. As shown in Fig. 2 that shows the PHF simulation result by 4 reflections, the PHF changes with time.

phase[degree]

frequency[Hz]

frequency[Hz]

Fig. 2. A PHF simulation result by 4 reflections

2.2 Magnitude Change of Howling Signal The AFC can easily diverge since the AFC is a positive feedback circuit. When a howling occurs, the signal magnitude increases continuously over time at a specific frequency as shown in Fig. 3(a) [9]. The spectrogram in Fig. 3(b) shows a howling phenomenon on an actual music signal.

y(t)

{

n −1

} +∑y

y n (t) = x ( t-nΔt ) * g s ( t ) * g f ( t )

n

m =0

m

(t)

y1 (t) = x ( t-Δt ) * g s ( t ) * g f ( t ) + y 0 (t) y0 (t) = x(t ) * g s (t ) time(s)

(a)

(b)

Fig. 3. An example of howling phenomenon; (a) magnitude of a howling signal and (b) spectrogram

2.3 Experiments on Indoor Condition Changes As shown in Fig. 4, after changing circumferential condition, the howling frequency has shifted from 380 Hz to 747 Hz. Therefore, the transition of reflective sound leads

Detection of Howling Frequency Using Temporal Variations in Power Spectrum

49

to the change of howling frequency. Also, room temperature is related to sound velocity and phase shift so that the howling frequency is affected by the room temperature. Fig. 5 shows that the howling frequency has shifted from 380 Hz at 21°C to 384 Hz at 23°C.

(a)

(b)

Fig. 4. An example of howling frequency change by different circumferential conditions; (a) the howling at 380 Hz and (b) The howling at 747Hz

0.5 10kHz 0.5kHz

0.4 0.3

phase[rad]

0.2 0.1 0 -0.1 -0.2 -0.3 -0.4 -0.5

0

5

10

15

20

25

30

35

40

45

50

temperature[o C]

(a)

(b)

Fig. 5. An example of howling frequency change by different temperature; (a) Phase shift and (b) Howling at 384 Hz

3 Proposed Howling Frequency Detection Method 3.1 Observation of Spectral Change Using Moving Average Detection of howling occurrence is difficult since acoustic signals change in both time and frequency domains. In this research work, we used a moving average (MA) filter in order to examine spectral changes as in Eq. (5), where X(ω,n) is the Fourier transform of x(n) at ω rad/sec and MA n, ω is the MA value of X(ω,n). MA n, ω

∑

X ω, n

i .

(5)

Figs. 6(a) and 6(b) show the MA values with p=50 and p=100, respectively. As shown in Fig. 6(b), the spectral change is minimized except the spectrum at howling frequency.

50

J.-W. Lee and S.H. Choi

(a)

(b)

Fig. 6. Examples of MA filtering in spectral domain; (a) p = 50 and (b) p = 100

Fig. 7. Decision for howling occurrences; (a) MA and dMA, (b) VA, and (c) DH

3.2 Detection of Howling Using Moving Average Filtering Considering the signal energy at howling frequency continuously increases, the MA value also increases over the time. Therefore, if the MA value continuously increases at a specific frequency, we can predict that a howling is likely to occur at that frequency as in Fig. 7. The dMA n, ω in Eq. (6) is used to decide whether MA value increases or not. The VA n, ω in Eq. (7) is 1 if dMA n, ω is positive and is 0 if dMA n, ω is not positive. If VA n, ω values are 1 for m times, DH n, ω in Eq. (8) is set to 1 as shown in Fig. 7. Therefore, we can detect the howling using the DH values. Fig. 8 shows the block diagram for the howling detection method. dMA n, ω

MA n, ω

VA n, ω

DH n, ω, m

1, 0,

MA n

dMA n, ω dMA n, ω

VA n

i, ω

1, ω

(6)

0 0

(7)

(8)

Detection of Howling Frequency Using Temporal Variations in Power Spectrum

51

Fig. 8. Block diagram for the proposed howling detection method

4 Conclusion In this paper, we explained the howling phenomenon by the acoustic feedback circuit and the potential howling was related to indoor conditions and environmental changes by the experiments. We presented a novel method to predict the howling frequency using temporal variations in power spectrum. And, we showed that the proposed moving average filtering method could be successfully used for howling detection.

References 1. Loetwassana, W., Punchalard, R., Lorsawatsiri, A., Koseeyaporn, J.: Adaptive howling suppressor in audio amplifier system. In: Proceedings of Asia-Pacific Conference on Communications, pp. 445–448 (2007) 2. Lee, J.W., Kang, S.H., Choi, S.H.: Prediction of potential acoustic gain considering directivity of microphone and loudspeaker. In: Proceedings of Spring Meeting of the Acoustical Society of Korea, vol. 26, pp. 275–278 (2007) 3. Schroeder, M.R.: Improvement of acoustic feedback stability by frequency shifting. Journal of the Acoustical Society of America 36, 1718–1724 (1964) 4. Nyquist, H.: Regeneration theory. Bell Syst. Tech. J. 11, 126–147 (1932) 5. Gil-Cacho, P.: Regularized adaptive notch filters for acoustic howling suppression. In: Proceedings of EUSIPCO, pp. 2574–2578 (2009) 6. Antman, H.S.: Extension to the theory of howlback in reverberant rooms. Journal of Acoustical Society of America (Letters to the Editor) 39(2), 399 (1966) 7. Yasukawa, H., Furukawa, I., Ishiyama, Y.: Acoustic echo control for high quality audio teleconferencing. In: Proceedings of ICASSP, pp. 2041–2044 (1989) 8. Ibaraki, I., Furukawa, H., Noono, H.: Pre-howling howlback detection method. In: Proceedings of ICASSP, pp. 941–944 (1986) 9. Lee, J.W., Kang, S.H., Choi, S.H.: The effect of room environments on howling frequency. In: Proceedings of Spring Meeting of the Acoustical Society of Korea, pp. 53–56 (2010) 10. Davis, D., Davis, C.: Sound System Engineering, 2nd edn., pp. 424–426. Focal Press (1997)

Differential Brain Activity in Reading Hangul and Hanja in Korean Language Hyo Woon Yoon and Ji-Hyang Lim Department of art therapy, Daegu Cyber University, Daegu, Korea [email protected]

Abstract. Even though the Korean words (Hangul) are characterized as phonemes like other alphabetic languages, their shape resembles much more morphemes like Chinese characters (Hanja). The use of functional magnetic resonance imaging permits the collection of brain activation patterns when native Korean speakers (12 persons as subjects) read Hangul and Hanja. The Korean language uses both alphabetic Hangul and logographic Hanja in its writing system. Our experimental results show that the activation patterns obtained for reading Hanja by Korean native speakers involve neural mechanisms that are similar to Chinese native speakers; i.e. strong left-lateralized middle frontal cortex activation. For the case of Korean word reading, the activation pattern in the bilateral fusiform gyrus, left middle frontal gyrus, left superior temporal gyrus, right mid temporal gyrus, precentral gyrus, and insula was observed Keywords: Word perception, brain activity, frontal cortex.

1 Introduction It is generally known that perceiving or reading visually presented words encompasses many processes that collectively activate several specialized neural systems to work in concert. Functional imaging techniques such as Positron Emission Tomography (PET) or functional Magnetic Resonance Imaging (fMRI) have provided meaningful insights into the neural systems that underlie word recognition and reading process in the human brain. In the proposed model of written word perception [1,2], it is proposed that a large-scale distributed cortical network, including the left frontal, temporal, and occipital cortices, mediates the processing of visuo-orthographic, phonologic, semantic, and syntactic constituents of alphabetic words. For example, the posterior fusiform gyri are relevant to visual processing, whereas the inferior frontal lobe emphasizes their role in semantic processing [3,4]. Regarding various written languages or writing systems, the question of how the surface form of words influences the neural mechanisms of the brain during word recognition is of interest. One of the most different writing systems from alphabetic words is the Chinese character. Alphabetic systems are based on the association of phonemes with graphemic symbols and linear structure, whereas Chinese characters are based on the association of meaningful morphemes with graphic units, the configuration of which is square and nonlinear. Previous studies using visual hemifield paradigms demonstrated T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 52–59, 2011. © Springer-Verlag Berlin Heidelberg 2011

Differential Brain Activity in Reading Hangul and Hanja in Korean Language

53

that the right cerebral hemisphere is more effective in processing Chinese characters than the left cerebral hemisphere [5]. This leads to a Chinese character-word dissociation hypothesis for a lateralisation pattern, since word perception is regarded to be left-lateralised. This conclusion has been disrupted, because some more current results of brain activation based on fMRI experiments suggest that the reading of Chinese characters is bi-lateralized. In particular, the left inferior frontal cortex (BA 9/45/46) emphasized the importance of the semantic generation or processing of Chinese characters [6,7]. Chinese characters are used not only in the writing systems of Chinese language, but are also widely used in the Japanese and Korean languages. The Korean writing system consists of the mixture of the pure Korean words and Chinese characters. The Korean words are characterized as phonemic components similar to the alphabetic words used in English or German. However, the shape of Korean words is nonlinear. The composition of its symbols is shaped into a square-like block, in which the symbols are arranged left to right and top to bottom. Its overall shape makes Korean more similar to Chinese than other alphabetic orthographies [8]. Furthermore, unlike alphabetic words, these phonemic symbols are not arranged in a serial order, but are combined into a single form to represent a syllable. These syllabic units are spatially separated from each other. Each Korean syllable is constructed of two to four symbols that in various combinations represent each of 24 phonemes. Thus, in a sense, Korean words, Hangul, can also be regarded as syllabograms (Figure 1). Hangul is the name of the Korean alphabet. In addition, the Korean vocabulary consists of pure Korean words (24.4 %), Chinese-derivative words (69.32 %), and other foreign words (6.28 %). Chinese derivative words can be written either in the form of Chinese ideogram or its corresponding Korean words [9]. In the current Korean writing system, e.g. daily newspapers or boulevard magazines in South Korea, the use of Chinese characters is relatively sparse. According to the statistics of the year 1994 [10], the proportion of Chinese characters in the body of daily newspapers are about 10 % and since then this has continuously diminished. Surprisingly, although these are unique and interesting characteristics, the neural mechanisms involved in reading Korean words have been rarely studied, at least with modern functional imaging techniques. Using functional magnetic resonance imaging technique, we investigated the neural mechanism involved in reading these two different writing systems by Korean native speakers. In doing so, we hope to identify specific neural mechanisms that are involved in reading Korean words (phonemes) and Chinese characters (morphemes).

2 Materials and Methods 2.1 Subjects Seven male and five female right-handed subjects (mean age: 22 years, S.D. 1.5 years) participated in the study. All were native Korean speakers who has been educated for Chinese characters for more than 6 years in school. They did not have any medical, neurological or psychiatric illness at past or present, and they did not take medication. All subjects consented to the protocol approved by the Institutional Ethics and Radiation Safety Committee.

54

H.W. Yoon and J.-H. Lim

2.2 Experimental Design As stimuli, two-character Chinese words and Korean words with equivalent phonetic as well as semantic components were chosen (Figure 1). There were 60 words for each category. All words were nouns. Half consisted of abstract meanings and the other half concrete meanings. Stimuli were presented using custom-made software on a PC and projected via an LCD projector onto a screen at the feet of the subjects. The subjects viewed the screen via a homemade reflection mirror attached on the head RF coil. Each stimulus was presented for 1.5 seconds long, followed by a blank screen for 500 ms. Ten different items of this stimulus pattern were presented, including a blank screen for one second prior to the first stimulus within a block. These stimuli blocks were alternated with the baseline task. During the baseline task, a fixation point was projected on the middle of the screen for 21 seconds. Two kinds of stimuli (Korean words and Chinese characters) and baseline task blocks lasted equally for 21 seconds each. A total of six blocks of Korean words and six blocks of Chinese characters were presented, and these were intermixed at random. During the experiment, the subjects were instructed to press the right button for nouns with an abstract meaning and the left button for those with a concrete meaning. Simultaneously, they should respond covertly to the stimuli presented. 2.3 Data Acquisition and Analysis Images were acquired by using 1.5 Tesla MRI scanner (Avanto, Siemens, Erlangen, Germany) with a quadrature head coil. Following a T1-weighted scout image, highresolution anatomic images were acquired using an MPRAGE (Magnetization-Prepared RApid Gradient Echo) sequence with TE = 3.7 ms, TR =8.1 ms, flip angle = 8°, and image size of 256 x 256. T2*-weighted functional data were acquired by using echo planar imaging (EPI) with TE = 37 ms, flip angle = 80°, TR = 3000 ms, and image size of 64 x 64. We obtained 30-slices EPI images with slice thickness of 5 mm and no gaps between slices for the whole brain. Total 172 volumes were acquired per an experimental run. For each participant, the first four volumes in each scan series were discarded, which were collected before magnetization reached equilibrium state. Image data were analyzed using SPM2 (Wellcome Department of Cognitive Neurology, London). The images of each subject were corrected for motion and realigned using the first scan of the block as a reference. T1 anatomical images were coregistered with the mean of the functional scans and then aligned to the SPM T1 template in the atlas space of Talairach and Tournoux [11]. Finally, the images were smoothed by applying Gaussian filter of 7 mm full-width at half-maximum (FWHM). In order to calculate contrasts, the SOA (stimulus onset asynchrony) from the protocol was defined as events and convolved with the hemodynamic response function (HRF) to specify the appropriate design matrix. The general linear model was used to analyze the smoothed signal at each voxel in brain. Significant changes in hemodynamic response for each subject and condition were assessed using t-statistics. For the group analysis, contrast images of single subject were analyzed using a random effect model. Activations were reported if they exceeded a threshold P < 0.05, corrected on the cluster level (P < 0.0001 uncorrected at the single voxel level). Significance on the cluster level was calculated in consideration of peak activation and extent of the cluster.

Differential Brain Activity in Reading Hangul and Hanja in Korean Language

55

Fig. 1. a) indicates the activation map “Korean word reading” minus “Chinese character reading” in 12 subjects (threshold at p < 0.0005, uncorrected at a single voxel level). b) indicates the activation map of Korean words minus baseline (left two images) and Chinese character minus baseline (right two images). Threshold p-value for b) is 0.0001 (uncorrected at the single voxel level).

a)

b) Fig. 2. Brain areas showing the repetition suppression effect. The anterior portion of the left fusiform gyrus showed reduced activation in case of the cross script condition a), whereas a more posterior portion of the fusiform gyrus is responsible for the repetition suppression in the case of the same script condition b).

56

H.W. Yoon and J.-H. Lim

3 Results The mean reaction time for subjects during Korean word reading was 1.01 sec (S.D.: 325 ms), whereas, for Chinese character reading, it was 1.24 sec (S.D.: 367 ms). A paired t-test verified the significance between these two reaction times (p < 0.00001). Significant signal changes for Korean words reading vs. baseline were detected bilaterally in the fusiform gyrus (BA 19/37) and in the left middle frontal area (BA 46/6). In addition, right hemispheric activation was observed in the medial frontal gyrus (BA 8). For Chinese characters vs. baseline, the activation patterns appeared to be slightly different. In the region, responsible for the visual stimuli per se, we observed bi-hemispheric activation for the Chinese character reading vs. baseline task. In the frontal (superior, BA 8 and inferior area, BA 9) and parietal (superior, BA 7) cortices, only left hemispheric activation was significant in contrast with the baseline task. Brain areas responsible for the repetition suppression are summarized in Table 2. Repetition suppression effect in the case of prime Hanja and target Hanja was clearly seen in the area of bilateral middle fusiform gyrus. In contrast, the suppression effect of priming was found in the area of left medial fusiform gyrus (x = -36, y = -40, z = -12) in the case of prime word Hanja, target word Hangul condition. According to the further analysis this area also showed a repetition priming suppression in the case of the cross-script condition. This was done by doing computerizing a linear combination of prime Hanja-target Hangul and prime Hangul-target Hanja conditions (inclusively masked by both contrast at p < .05). This suggests that the left medial portion of fusiform gyrus seemed to exhibit a significant effect of repetition suppression, irrespective of the direction of script alternations. To the contrary, the repetition priming suppression effect in the case of the same script conditions (linear computerizing of prime Hanja-target Hanja and prime Hangul-target Hangul) showed also in the area of left fusiform gyrus, but more posterior to coronal section and lateral to sagittal section (x = -42, y = -46, z = -16).

4 Discussion In the different contrast of Korean words minus Chinese character conditions, significant positive signal changes were observed in the right superior gyrus of the frontal lobe (BA 8), the left superior temporal lobe (BA 41), and the right midtemporal lobe (BA21), precentral gyrus (BA 6) and insula (BA 13) and for the condition of Chinese character minus Korean words, activation was observed in the bi-hemispheric visual area (BA 19). In terms of behavioral data, significantly longer reaction times were observed for Chinese character reading compared to Korean word reading. Since very simple Characters were used as stimuli, it would appear that the reaction time advantage for reading Korean words is not derived from a familiarity effect. Rather, it might rely on differences in characteristics of phonological processing between these two writing systems. The phonological processing in Chinese character recognition is at the syllable-morpheme phonology level. This is the fundamental difference regarding the role of phonology between Chinese and alphabetic writing systems. The concept of

Differential Brain Activity in Reading Hangul and Hanja in Korean Language

57

pre-lexical phonology is misleading for Chinese character reading [8]. However, in processing Korean words, pre-lexical phonology is activated rapidly and automatically. Reading Korean words for meaning involves pre-lexical information processing [12]. In the functional imaging data, the activated area for the condition of Chinese character vs. baseline reading was found to be in the left hemispheric inferior and superior gyri of the frontal lobe (BA 6/9). This demonstrates the left lateralized pattern of the frontal cortex during Chinese character reading. This activation can be attributed to the unique square configuration of Chinese characters [13,14]. Chinese characters consist of a number of strokes that are packed into a square shape according to stroke assembly rules, and this requires a fine-grained analyses of the visual-spatial locations of the strokes and subcharacter components [15]. In addition, it is known that the left middle frontal cortex (BA 6/9) is the area of spatial and verbal working memory by which the subject maintains a limited amount of spatial and verbal information in an active state for a brief period of time [16,17]. More precisely, this area may play a role as a central executive system for working memory, which is responsible for coordination of cognitive resources [18]. In our experiment, even though a working memory process was not involved in the subjects’ decision, they indeed needed to coordinate the semantic (or phonological) processing of the Chinese characters. These two processes of coordination of cognitive resources and semantic processing were explicitly required by the experimental task and the intensive visuospatial processing of the Chinese characters. It seems that the activation of the left middle frontal gyrus should be involved in these two cognitive processes. This left frontal activation pattern is consistent with other studies, in which functional imaging techniques of Chinese character reading by native Chinese speakers were used, especially the activation of BA 9 [7,19,20]. Left hemispheric middle frontal activations (BA 46/6) were also observed for the condition of Korean words vs. baseline and this appears to be correlated with similar mechanisms associated with the reading of other alphabetic words [21]. Since our subjects were ask to respond after seeing and the covert speaking of Korean words (forced choice option), which is connected with semantic processing, the activation of middle frontal area seems to underlie this cognitive process. We propose that this might be the reason for why the left frontal area is strongly activated during this experimental task. Occipital lobe activation was observed for Chinese character reading in contrast with baseline as well as a direct comparison with Korean words (Table 1). The activated occipital areas, such as the fusiform gyrus, are thought to be relevant to the visual processing of Chinese characters. Interestingly, we observed right hemispheric dominant occipital activation, even though two-character Chinese words were presented as stimuli. There were some indications that the reading of two-character Chinese word is left lateralized [19,20], but our results did not support the dissociation hypothesis of single and two-character Chinese word perception. Bilateral activation of occipito-temporal area was also observed for the Korean word reading. It is generally thought that this area is relevant to the processing of the visual properties of Korean words. The activation pattern is bilateral, but the left hemispheric activity was relatively weaker (Table 1). This is not in agreement with previous studies with alphabetic words [22].

58

H.W. Yoon and J.-H. Lim

Another interesting imaging result of the present study is that the more anterior, medial area (x = -36, y = -40, z = -12) of the left fusiform gyrus is involved in the repetition suppression in case of cross script condition. This cross script related site of the present study lies further along 14 mm in the posterior temporal region. A study of Thompson-Schill et al. (1999)[23] reported that there was activation reduction of left temporal region (averaged coordinates x = -40, y = -28, z = -12, with standard deviation of y-axis 17 mm) involved in the repeated retrieval of semantic knowledge. The decreased activity in this area during semantic retrieval of primed items may be related to the phenomenon of repetition suppression observed in neurons in anteriorventral inferotemporal cortex of nonhuman primates [23]. Furthermore, Devlin et al. (2004) [24] reported that reduced activation of the left middle temporal gyrus was found for word pairs with a high degree of semantic overlap. Moreover, according to a previous study of cross script masked priming effect in Japanese by Nakamura et al. (2005)[10], the left middle temporal cortex (x = -48, y = -43, z = -2) was found to be involved in the repetition suppression. They suggested that this activation may correspond to a progressive abstraction process, as is also proposed for object recognition, whereby the raw visual features of stimuli are progressively transformed from perceptual to conceptual. In fact, this part of the left temporal gyrus was reported to be associated with the semantic network. In addition, in the behavioral results of masked priming experiment of Kim and Davis (2002)[25], they have shown priming effect of Hangul-Hanja prime-target relations. It was also suggested that this priming effect was occured at the semantic level. They further postulated that the facilitation due to priming depends upon the semantic process, which should be happening with the lexical information simultaneously (Kim and Davis, 2002). Taken together, the result of the present study due to cross script repetition suppression in the anterior part of fusiform gyrus seems to be related to the semantic representation influenced by subliminal primes, even though the activity reduction area of the present study lies closer to the occipital region compared to aforementioned previous studies. It seems that this region is related with the anterior-posterior progression in word processing. In terms of this interpretation one point should be considered. As Nakamura et al. (2005) [10] have mentioned in their study, some cautious approaches should be made due to this interpretation. The posterior temporal activity reduction due to repeated items could include the effect of motor congruity as a potential confounding variable, which is inflated by response association learned through repeated exposure to the same items). The result of the present study indicating that the stronger priming effect in case of prime Hanja, target Hangul condition may lead to an similar interpretation of them, since the subliminal primes in Hanja and their orthography-to-lexicon route seem to be activated ultimately by the phonological route of visible target words in Hangul.

References 1. Price, C.J.: The anatomy of language: contributions from functional neuroimaging. J. Anat. 3, 335–359 (2000) 2. Demonet, J.E., Chollet, F., Ramsay, S., Cardebat, D., Nespoulus, J.N., Wise, R., Rascol, A., Frackowiak, R.: The anatomy of phonological and semantic processing in normal subjects. Brain 115, 1753–1768 (1992)

Differential Brain Activity in Reading Hangul and Hanja in Korean Language

59

3. Bookheimer, S.: Functional MRI of language: New approaches to understanding the cortical organization of semantic processing. Ann. Rev. Neurosci. 25, 151–188 (2002) 4. de Zubicaray, G.I., Wilson, S.J., McMahon, K.L., Muthiah, S.: The semantic interference effect in the picture-word paradigm: An event-related fMRI study employing overt responses. Human Brain Mapping 14, 218–227 (2001) 5. Tzeng, O., Hung, D., Cotton, B., Wang, W.S.-Y.: Visual lateralisation effect in reading Chinese characters. Nature 282, 499–501 (1979) 6. Ding, G., Perry, C., Peng, D., Ma, L., Li, D., Xu, S., Luo, Q., Xu, D., Yang, J.: Neural mechanisms underlying semantic and orthographic processing in Chinese-English bilinguals. NeuroReport 14, 1557–1562 (2003) 7. Tan, L.H., Spinks, J.A., Gao, J.-H., Liu, H.-L., Perfetti, C.A., Xiong, J., Stofer, K.A., Pu, Y., Liu, Y., Fox, P.T.: Brain activation in the processing of Chinese characters and words: a functional MRI study. Human Brain Mapping 10, 16–27 (2000) 8. Wang, M., Koda, K., Perfetti, C.A.: Alphabetic and nonalphabetic L1 effects in English word identification: a comparison of Korean and Chinese English L2 learners. Cognition 87, 129–149 (2003) 9. Kim, H., Na, D.: Dissociation of pure Korean words and Chinese-derivative words in phonological dysgraphia. Brain and Language 74, 134–137 (2000) 10. Nakamura, K., Dehaene, S., Jobert, A., Le Bihan, D., Kouider, S.: Subliminal Convergence of Kanji and Kana Words: Further Evidence for Functional Parcellation of the Posterior Temporal Cortex in Visual Word Perception. J. Cog. Neurosci. 17(6), 954– 968 (2005) 11. Gusnard, D., Raichle, M.: Searching for a baseline: functional imaging and the resting human brain. Nat. Rev., Neurosci. 2, 685–694 (2001) 12. Kuo, W., Yeh, T.C., Duann, J.-R., Wu, Y.-T., Ho, L.-W., Hung, D., Tzeng, O.J.L., Hsieh, J.-C.: A left-lateralized network for reading Chinese words: a 3 T fMRI study. NeuroReport 12, 3997–4001 (2001) 13. Tan, L.H., Liu, H.-L., Perfetti, C.A., Spinks, J.A., Fox, P.T., Gao, J.-H.: The neural system underlying Chinese logograph reading. NeuroImage 13, 836–846 (2001) 14. Chee, M., Tan, E., Thiel, T.: Mandarin and English single word processing studies with functional magnetic resonance imaging. J. Neurosci. 19, 3050–3056 (1999) 15. Chee, M.W., Weekes, B., Lee, K.M., Soon, C.S., Schreiber, A., Hoon, J.J., Chee, M.: Overlap and dissociation of semantic processing of Chinese characters, English words, and pictures: evidence from fMRI. NeuroImage 12, 392–403 (2000) 16. Zhang, W., Feng, L.: Interhemispheric interaction affected by identification of Chinese characters. Brain and Cognition 39, 93–99 (1999) 17. Mathews, P.M., Adcock, J., Chen, Y., Fu, S., Devlin, J.T., Rushworth, M.F.S., Smith, S., Beckmann, C., Iversen, S.: Towards understanding language organization in the brain using fMRI. Human Brain Mapping 18, 239–247 (2003) 18. Courtney, S.M., Petit, L., Maisog, J.M., Ungeleider, L.G., Haxby, J.V.: An area specialized for spatial working memory in human frontal cortex. Science 279, 1347–1351 (1998) 19. Owen, A.M., Doyon, J., Petrides, M., Evans, A.C.: Planning and spatial-working memory: A positron emission tomography study in humans. Eur. J. Neurosci. 8, 353–364 (1996)

Computational Neural Model of the Bilingual Stroop Effect: An fMRI Study Hyo Woon Yoon Department of art therapy, Daegu Cyber University, Daegu, Korea [email protected]

Abstract. A functional MRI was used to investigate the computational neuronal model of differential processing patterns of two languages (Korean as a mother language, L1 and English as L2) in the late Korean-English bilingual subjects during the performance of a Stroop task during overt production of words. The Stroop paradigm experiment was done separately in L1 and L2 and the imaging results of these different conditions were compared. In the case of L1, the activation of the bilateral anterior cingulate gyrus was observed among others. L1 related activation was also observed in middle frontal and inferior parietal lobule. In the case of L2, frontal and parietal as well as superior temporal activation was observed, but the absence of ACC activation was reported. This difference led to an argument that the differential information processing (automatization, inhibition control) mechanisms between L1 and L2. Keywords: automatization; inhibition control; anterior cingulate gyrus.

1 Introduction A number of studies on bilingual subjects have investigated the neural mechanisms associated with the processing of L1 and L2, especially with the aid of modern imaging techniques such as PET (Positron Emission Tomography) and fMRI (functional Magnetic Resonance Imaging). The focus of most of the studies was on whether the same or different locations of the brain are activated during the processing of L1 and L2. Their assumption was that the activation of the same site for L1 and L2 indicated that the same module was shared for the processing of both languages [1,2]. While another line of studies provide evidence for different cortical organization for L1 and L2 for late bilinguals [3,4,5] If the use of two languages would be explained in terms of the concepts of implicit and explicit memory, this model can also explain the reason why bilingual aphasic patients show paradoxical recovery. Even though the damage interferes with the implicit memory, the explicit knowledge, which relies on the different neural mechanism, could be used. In the case where L2 was recovered better than L1, it seemed to be related to the broader regions involved in the explicit knowledge of L2 could compensate the damaged language area [6,7]. The loss of L2, which has been repeatedly reported among the patients with Alzheimer’s disease, can generally be explained by the fact that they lost their declaritive memory mechanism at first. In the present study, the functional MRI was used to investigate the neural correlates of the language processing differences in T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 60–65, 2011. © Springer-Verlag Berlin Heidelberg 2011

Computational Neural Model of the Bilingual Stroop Effect: An fMRI Study

61

L1 and L2 during the performance of the Stroop task in late Korean-English bilingual subjects. The main focus of the study was to identify differences in the processing mechanisms between L1 and L2 during the performance of the Stroop task. It was thought that the inhibition of automatic processing is needed for the Stroop task in L1, whereas L2 processing should show less activity in such inhibition related cortical areas. Rather, L2 processing should correspond more to the activity in the memory related cortical regions, or those responsible for color perception.

2 Materials and Methods 2.1 Subjects Eighteen undergraduate students (9 males and 9 females with normal vision Mean=21.3, SD = 2.1, Range=19~26) participated in this study. All were native Korean speakers who had received English education in school for more than 6 years in school. They were right-handed, had no psychiatric history, and did not regularly take any medicine. All subjects consented to the protocol approved by the Institutional Ethics and Radiation Safety Committee. They were paid for participating in the study. 2.2 Experimental Design The independent variables of the experiment were divided largely into the kind of task (congruent, incongruent), stimulus language (Korean, English), and response language (Korean, English). The box-car design was used. The materials were presented using E-Prime. As stimuli, three Korean color words and three English color words (RED, GREEN, BLUE) were used. During the experiment, the participants were asked to loudly speak the colors of presented words in L1 and L2 as accurately and quickly as possible. There were four types of experimental tasks and each was repeated three times. In the experimental tasks ICKK and ICEE, subject spoke the color of the presented word using the same language (Korean-Korean, English-English), where the meaning of the word was incongruent with the color of the word. 2.3 Data Acquisition and Analysis Images were acquired by using 1.5 Tesla MRI scanner (Avanto, Siemens, Erlangen, Germany) with a quadrature head coil. Following a T1-weighted scout image, highresolution anatomic images were acquired using an MPRAGE (Magnetization-Prepared RApid Gradient Echo) sequence with TE = 3.7 ms, TR =8.1 ms, flip angle = 8°, and image size of 256 x 256. T2*-weighted functional data were acquired by using echo planar imaging (EPI) with TE = 37 ms, flip angle = 80°, TR = 3000 ms, and image size of 64 x 64. We obtained 30-slices EPI images with slice thickness of 5 mm and no gaps between slices for the whole brain. Total 172 volumes were acquired per an experimental run. For each participant, the first four volumes in each scan series were discarded, which were collected before magnetization reached equilibrium state.

62

H.W. Yoon

Image data were analyzed using SPM2 (Wellcome Department of Cognitive Neurology, London). The images of each subject were corrected for motion and realigned using the first scan of the block as a reference. T1 anatomical images were coregistered with the mean of the functional scans and then aligned to the SPM T1 template in the atlas space of Talairach and Tournoux [8]. Finally, the images were smoothed by applying Gaussian filter of 7 mm full-width at half-maximum (FWHM). In order to calculate contrasts, the SOA (stimulus onset asynchrony) from the protocol was defined as events and convolved with the hemodynamic response function (HRF) to specify the appropriate design matrix. The general linear model was used to analyze the smoothed signal at each voxel in brain. Significant changes in hemodynamic response for each subject and condition were assessed using t-statistics. For the group analysis, contrast images of single subject were analyzed using a random effect model. Activations were reported if they exceeded a threshold P < 0.05, corrected on the cluster level (P < 0.0001 uncorrected at the single voxel level). Significance on the cluster level was calculated in consideration of peak activation and extent of the cluster.

3 Results The imaging data of 18 subjects were analyzed. In order to observe the typical Stroop effect the incongruent condition (ICKK) was subtracted from the congruent condition (CKK). Bilateral anterior cingulate (BA 24/32), middle frontal gyrus (BA 9/10), left lentiform nucleus (putamen), inferior parietal lobule (BA 40), caudate, right precentral gyrus (BA 6), superior frontal gyrus (BA 6), and thalamus activation were observed (Table 2 & Figure 1). The direct comparison of L1 and L2 during Stroop task performance shows the following cortical activations. This comparison was done in order to find out the differential pattern of language processing in L1 and L2 (automatization, control inhibition). ICKK minus ICEE shows the activation of the bilateral putamen, thalamus, anterior cingulate (BA 32), middle frontal gyrus (BA 6/8/9), caudate, left posterior cingulate (BA 23), right inferior frontal gyrus (BA47), and middle temporal gyrus (BA 21). The inverted comparison, ICEE minus ICKK, shows the activation of the bilateral superior temporal gyrus (BA 38), parahippocampal gyrus (BA 30/36), precentral gyrus (BA 6), caudate, left fusiform gyrus (BA 36), cuneus (BA 17), right inferior parietal lobule (BA 40), medial frontal gyrus (BA 8), and insula (BA 13). The comparison of ICEK minus ICKK should confirm of the hypothesis that L2 word processing depends on the controlled processes, since the subjects should verbally respond in Korean in the ICEK condition despite the words being presented in English. This contrast shows the activation of the bilateral parahippocampal gyrus (BA 36), right insula (BA 13), superior temporal gyrus (BA 22), left posterior cingulate (BA 23), and caudate.

4 Discussion In order to identify the typical cortical activation areas during performance of the Stroop task, the incongruent condition was subtracted from the congruent conditions

Computational Neural Model of the Bilingual Stroop Effect: An fMRI Study

63

Fig. 1. Activation map of the contrast ICKK minus CKK in upper three images and ICEK minus ICKK in below three images. The differential activation patterns can be seen, especially in ACC.

for Korean use (ICKK minus CKK). The activation of the bilateral anterior cingulated (BA 24/32), middle frontal gyrus (BA 9/10), left lentiform nucleus (putamen), inferior parietal (BA 40), caudate, right precentral gyrus (BA 6), superior frontal gyrus (BA 6), and thalamus was observed in this contrast. In the Stroop task, the recognition of real color meaning is interfered with by the word, which does not correspond with the presented colors [9]. In general, the processing of word reading is regarded as more automatized than the color recognition [3], thus the successful performance of color recognition in the Stroop task requires the inhibition of the automatized processing of accompanied word reading. The capability of such inhibition is one of the characteristics of working memory executive functions. The executive function is known to be mostly related to the involvement of the brain area dorsolateral prefrontal cortex (DLPFC) and anterior cingulate cortex [10,11]. Especially anterior cingulate activation is responsible for the monitoring of such inhibition functions in the Stroop task [12]. A number of neuroimaging studies have reported commonly the activation of ACC during the Stroop task performance [13,14,15]. The activation of ACC and DLPFC in the present study was consistent with the results of previous studies, which indicated that these areas were important for task-relevant control of conflicting information or of competing sources of information. On the basis of literature evidences including the present paper’s results, the DLPFC and ACC are typical brain regions involved in the Stroop task performance, but involvement of the parietal region in performing the task has also been reported [14,15,1718]. Bench et al. (1993) [14] and Peterson et al. (1999) [17] reported inferior parietal activation during Stroop task performance, and they have interpreted this activation is related to the retaining

64

H.W. Yoon

attention, which is needed for task performance. Adleman et al. (2002) [16] reported that inferior parietal activation is also connected with the role in attending to the related features while inhibiting the unrelated features in the stimulus. The left inferior parietal activation in the present study seemed to be related to one of the general attention processing mechanisms demanded by other tasks such as the Simon task, which involves the inhibition of incongruent spatial features [17,19]. According to the previous literature evidences, brain activations of the thalamus, pulvinar, superior colliculus, posterior parietal regions, prefrontal regions, anterior cingulate cortex, and basal ganglia have been known to be the neural correlates of the attention processes [13,20]. The involvement of the thalamus among them in the case of attention processing is particularly meaningful, since this brain area seems to be related to the selection of input information for the further processing of attention. In a PET study by LaBerge & Buchsbaum (1990) [20], thalamus activation was more involved during the task of searching for a letter among 8 different items, compared to the simple recognition of one. They interpreted this to mean that thalamic activity was related to the selection process of input information to attend. In the present study, the cognitive load was larger in the case of the incongruent condition, compared to the congruent one. Therefore, the activation of the thalamus as well as basal ganglia (putamen, caudate) seemed to serve this attention process during the incongruent condition of Stroop task performance. The precentral gyrus plays a role involving sensoric and motoric processes. Pinel et al. (1999) [21] have postulated that the activation of this area is related to visual identification and appropriate response. Schneider & Chein (2003) [12] have postulated that the precentral gyrus is responsible for suppressing movement, which is needed for the reaction delay during interference tasks such as Stroop. The results of the present study seemed to be consistent with these previous evidences. The behavioral data in this study indicated that there were significant differences between congruent and incongruent conditions. This means that the incongruent task should be more difficult compared to the congruent and this difficulty should include the interference effect of task performance as well.

References 1. Klein, D., Milner, B., Zatorre, R.J., Zhao, V., Nikelski, J.: Cerebral organization in bilinguals: A PET study of Chinese–English verb generation. NeuroReport 10(13), 2841– 2846 (1999) 2. Hernandez, A.E., Martinez, A., Kohnert, K.: In search of the language switch: An fMRI study of picture naming in Spanish–English bilinguals. Brain and Language 73(3), 421– 431 (2000) 3. Cohen, J.D., Dunbar, K., McClelland, J.L.: On the control of automatic process: A parallel distributed processing account of the Stroop effect. Psychological Review 97(3), 332–361 (1990) 4. Perani, D., Dehaene, S., Grassi, F., Cohen, L., Cappa, S., Paulesu, E., Dupoux, E., Fazio, F., Mehler, J.: Brain processing of native and foreign languages. NeuroReport 7(15-17), 2439–2444 (1996) 5. Pallier, C., Dehaene, S., Poline, J.-B., LeBihan, D., Argenti, A.-M., Dupoux, E., Mehler, J.: Brain imaging of language plasticity in adopted adults: Can a second language replace the first? Cerebral Cortex 13(2), 155–161 (2003)

Computational Neural Model of the Bilingual Stroop Effect: An fMRI Study

65

6. Aglioti, S., Fabbro, F.: Paradoxical selective recovery in a bilingual aphasic following subcortical lesions. NeuroReport 4(12), 1359–1362 (1993) 7. Fabbro, F., Paradis, M.: Acquired aphasia in bilingual child. In: Paradis, M. (ed.) Aspects of bilingual aphasia, pp. 67–83. Pergamon Press, Oxford (1995) 8. Talairach, J., Tournoux, P.: Co-planar stereotaxic atlas of the human brain. Thime, New York (1988) 9. MacLeod, C.M.: Half a century of research on the Stroop effect: An integrative review. Psychological Bulletin 109(2), 163–203 (1991) 10. Roberts, A.C., Robbins, T.W., Weiskrantz, L.: The prefrontal cortex: Executive and cognitive functions. Oxford University Press, Oxford (1998) 11. Stuss, D.T., Shallice, T., Alexander, M.P., Picton, T.W.: A multidisciplinary approach to anterior attentional functions. Annals of the New York Academy of Science 769(11), 191– 211 (1995) 12. Schneider, W., Chein, J.M.: Controlled & automatic processing: Behavior, theory, and biological mechanisms. Cognitive Science 27(3), 525–559 (2003) 13. Banich, M.T., Milham, M.P., Atchley, R., Cohen, N.J., Webb, A., Wszalek, T., Kramer, A.F., Liang, Z.P., Wright, A., Shenker, J., Magin, R.: fMRI studies of Stroop tasks reveal unique roles of anterior and posterior brain systems in attentional selection. Journal of Cognitive Neuroscience 12(6), 988–1000 (2000) 14. Bench, C.J., Frith, C.D., Grasby, P.M., Friston, K.J., Paulesu, E., Frackowiak, R.S.J., Dolan, R.J.: Investigations of the functional anatomy of attention using the Stroop test. Neuropsychologia 31(9), 907–922 (1993) 15. Pardo, J.V., Pardo, P.J., Janer, K.W., Raichle, M.E.: The anterior cingulate cortex mediates processing selection in the Stroop attentional conflict paradigm. Proceedings of the National Academy of Sciences of the USA 87(1), 256–259 (1990) 16. Adleman, N.E., Menon, V., Blasey, C.M., White, C.D., Warsofsky, I.S., Glover, G.H., Reiss, A.L.: A developmental fMRI study of the Stroop color-word task. NeuroImage 16(1), 61–75 (2002) 17. Peterson, B.S., Kane, M.J., Alexander, G.M., Lacadie, C., Skudlarski, P., Leung, H.C., May, J., Gore, J.C.: An event-related functional MRI study comparing interference effects in the Simon and Stroop tasks. Cognitive Brain Research 13(3), 427–440 (2002) 18. Taylor, S.F., Kornblum, S., Lauber, E.J., Minoshima, S., Koeppe, R.A.: Isolation of specific interference processing in the Stroop task: PET activation studies. NeuroImage 6(2), 81–92 (1997) 19. Fink, G.R., Dolan, R.J., Halligan, P.W., Marshall, J.C., Frith, C.D.: Space-based and object-based visual attention: Shared and specific neural domains. Brain 120(11), 2013– 2028 (1997) 20. LaBerge, D., Buchsbaum, M.S.: Positron emission tomographic measurements of pulvinar activity during an attention task. Journal of Neuroscience 10(2), 613–619 (1990) 21. Pinel, P., Le Clec’H, G., van de Moortele, P.F., Naccache, L., Le Bihan, D., Dehaene, S.: Event-related fMRI analysis of the cerebral circuit for number comparison. NeuroReport 10(7), 1473–1479 (1999)

The Eye Movement and Data Processing Due to Obtained BOLD Signal in V1 : A Study of Simultaneous Measurement of EOG and fMRI Hyo Woon Yoon1,*, Dong-Hwa Kim2, Young Jae Lee3, Hyun-Chang Lee4, and Ji-Hyang Lim1 1

Department of Art Therapy, Daegu Cyber University, Daegu, Republic of Korea Tel.: +82-53-850-4081, Fax: +82-53-850-4019 [email protected] 2 Department of Electrical Engineering, Hanbat University, Daejeon, Republic of Korea 3 Department of Multimedia, Jeonju University, Jeonju, Republic of Korea 4 Division of Information and Electronic Commerce, Wonkwang University, Iksan, Republic of Korea

Abstract. The simultaneous measurement of EOG and fMRI has been done in the present study in order to investigate the influence of eye movement (blinking mechanism) on the functional magnetic resonance imaging (fMRI) signal response in the primary visual cortex. The conventional Echo-Planar Imaging (EPI, T2*-weighted) with concurrent electrooculogram (EOG) was obtained in four subjects while they viewed a fixation point and a checkerboard with a flickering rate of 8Hz. The division of two different conditions in the whole experimental blocks can be done with the help of EOG information: fixation and moving eye. The fMRI data comparison of these two conditions has been achieved. The results of this study indicated that there are no differential signal changes between these two conditions. This suggests that eye blinking does not affect BOLD signal changes in the primary visual cortex. In terms of data processing the eye blinking can be ignored according to the results of the present study. Keywords: simultaneous measurement of fMRI and EEG, Artifact correction.

1 Introduction For measuring correlates of neuronal activation in the brain several methodological approaches are currently used, e.g., Positron Emission Tomography (PET) or functional magnetic resonance imaging (fMRI). One of the preferred methods for this purpose is functional MRI which can detect the increased blood flow related to neuronal activity with relatively high spatial resolution. T2*-weighted MR imaging can reveal changes in blood oxygenation in activated brain areas [1]. Fast imaging sequences such as echo-planar imaging (EPI) can capture stimulus-evoked transient *

Corresponding author.

T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 66–71, 2011. © Springer-Verlag Berlin Heidelberg 2011

The Eye Movement and Data Processing Due to Obtained BOLD Signal in V1

67

changes in blood oxygenation level-dependent (BOLD) contrast, which likely reflects hemodynamic responses [2-4]. The majority of fMRI studies are based on visual stimuli and eye movement could influence signal intensity of functional MR data, especially in the primary visual cortex. It has been reported that local eye movements influence whole-head motion correction procedures, resulting in inaccurate movement parameters and potentially lowering the activation detection sensitivity [5]. Eye movement also affected the magnitude of fMRI response in the extrastriate cortex during visual motion perception [6] and eye blinking in a dark environment evoked primary visual cortex’s activation [7]. In this study, we have measured electrooculogram (EOG) data simultaneously with EPI data acquisition to detect the eye blinking signal while visual stimuli were being presented. The aim of our study is to find out if there are differences or correlations in the activation patterns of the primary visual cortex between a large and small frequency of eye blinking. However, hemodynamic response is only indirectly linked to the energy consumption of the neural population, and takes place on a timescale of more than 3 seconds after giving stimulus input. In order to compensate for temporal resolution and indirect measurement of neural activity, a direct measuring method of the electrical activity of neurons has been suggested. This can be done with using electroencephalogram (EEG) and this method features the millisecond timescale, which might represent underlying cognitive processes. The combination of EEG and fMRI or the simultaneous recording of the EEG and fMRI might be a promising tool for functional brain mapping, which provides physiological characteristics with both high spatial and temporal resolution. In the present study, we aimed to complete the experiment using this combination. In particular, the acquisition of EOG is done for measuring the frequency of eye blinking in each subject.

2 Materials and Methods 2.1 Subjects Four right-handed healthy volunteers (mean age of 26 years with SD of 1.2 years, all males) participated as subjects in this study after giving their written informed consent. They were free of any neurological antecedent and had good vision. Before starting the scanning session, subjects put on the cap for recording EEG. 2.2 EEG (Electroencephalogram) Recording A commercially available MR-compatible system (Brain Amp MR, Brain Products GmbH, Germany) was used for EEG recording and used along with a speciallydesigned electrode cap (BrainCap-MRI). The electrode cap contains 32 EEG channels and three additional channels dedicated to electocardiogram (ECG) and electrooculogram (EOG) acquisition. The amplifier was designed to be placed inside the magnet bore of the scanner and was connected to the host computer outside the MR room via a fiber optic cable. The resolution and dynamic range of the amplifier was 100nV and ±3.2 mV, respectively. The EEG and EOG waveforms were recorded

68

H.W. Yoon et al.

with a sampling rate of 500 samples/s. Band pass filtering from 0.5 to 80 Hz was applied along with 60 Hz notch filtering. We recorded EOG in order to monitor the subject’s blinking and eye movement. The gradient-induced artifacts on EEG signal were removed by digital filtering (8-10). Detecting the blink signal on the EEG recording monitored eye movements. 2.3 Experiment Design Visual stimuli were produced using custom designed software. Stimuli were presented with the aid of a video projecting on a screen to the subjects via a mirror. In experimental sessions visual stimuli had concentric circles shape. The fixation point at the center was static, while the surrounding field was flickered. These were circle type checkerboards which had a flickering rate of 8 Hz. Stimuli were presented to subjects with a viewing angle of 13° vertically, 17° horizontally and a viewing distance 5 m. During the experiment, subjects performed 24 blocks of condition with a duration of 30 sec alternating with visual center point fixation. Images were acquired during three experimental conditions (natural eye movement, fixed eye, and rest). During the rest condition, subjects were asked to look at the fixation point. During natural eye movement’s condition, they also should look at the fixation point and simultaneously the flickering checkerboard’s pattern which was being presented. During the fixed eye condition, subjects were asked to keep eyes open and try to not to blink their eyes during stimuli presentation. Checkerboard stimuli were presented as the natural eye movement condition. 2.4 fMRI Scanning fMRI data was acquired with a 3 Tesla MR scanner (Varian Console, built up by ISOL Tech., Korea). Echo-planar images (128x128 matrix, over a 160mm field of view) consisted of 15 consecutive axial sections (3mm thickness, no gap, repetition time/echo time = 3000/37 milliseconds, flip angle 70). An experiment session consisted of 24 blocks of 30 seconds. The remaining 240 sec (40 volumes) were analyzed. Using a quadrature head coil (anatomic scan), high-resolution anatomic images were acquired using an MPRAGE sequence (echo time TE = 3.7 ms, TR = 8.1 ms, flip angle = 8°, FOV = 256 x 256 mm). 2.5 Data Analysis Before scanning, we recorded each subject’s EOG reference data and measured the amplitude, shape, and rate of reference data for detecting the eye’s movement and blinking. During functional MR imaging, the switching of gradient magnetic field induced a series of artifacts into EEG whose amplitudes are 10 to 100 times lager than that of the EEG, data which was measured outside the MR-room. This makes the monitoring of EOG waveforms difficult when MR imaging is being simultaneously performed. The correction method of gradient-induced artifact is to average the intervals in which the gradient changes of the scanner have taken place. The averaged gradient-induced artifact is then subtracted from the original EEG data in the affected intervals. The gradient-induced artifacts are generally not completely eliminated in

The Eye Movement and Data Processing Due to Obtained BOLD Signal in V1

69

this way. For this reason, different filters then were applied to the data in the corrected ranges. After gradient-induced artifacts on recorded EOG were removed by these methods [2], we detected the blink signal being similar to the each subject’s reference EOG signal which has over 150μV and about 2~4 Hz frequency. The EOG signals typically showed sustained spike wave during eye blinking as shown in Figure 1(C).

Fig. 1. The Electrooculography. The top is reference data, middle is acquired during fMRI scan and Bottom is removed gradient-induced artifacts on the middle. Calibration = 1 second, 1500μV.

To analyze fMRI data in adequate blocks, the first slice scan time within a volume was used as a reference for alignment by linear interpolation of the following slice of that volume to correct for the temporal slice scan time shifts. Data analysis and visualization were performed with the fMRI software package Brain Voyager (Brain Innovation, Maastricht, The Netherlands) [11]. Before the main analysis the following processing steps were carried out 1) motion correction, 2) spatial smoothing of EPI images with full width at half-maximum of 4 mm, 3) transformation into Talairach [12] coordinate space. The cortical sheets of individual subjects and a template brain were reconstructed as polygon meshes based on high-resolution T1-weighted structural three-dimensional recording. The white-gray matter boundary was segmented, reconstructed, smoothed, and morphed. A morphed surface always possesses a link to the folded reference mesh so that functional data can be correctly projected onto partially inflated representations. The ON and OFF phase were used for linear regression analysis of the BOLD signal time course. Using an empirically founded model [13] of the temporal dynamics of the fMRI signal, hemodynamic predictors were computed from the ON and OFF phase and a general linear model (GLM) was computed for every voxel.

70

H.W. Yoon et al.

3 Results As shown in Figure 2, the results of one of four subjects indicates that the activation of primary visual cortex for both the fixed eye (FE) and natural eye movement (ME) conditions was observed. The indicated activation area was observed in the other three subjects. The averaged blinking frequency, which was detected by EOG, for the FE is condition 0.8 and 2.9 for the ME condition. Even though a larger blinking frequency can be observed by the ME condition compared to FE condition, the activation of the contrast ‘ME minus FE’, or ‘FE minus ME’ indicates that there were very few significantly activated voxels.

Fig. 2. Processing results of blinking eye (BE)’s condition. (A) is activated voxel on ‘FE-FP’s contrast (p < 3.66E-34). (B) is time course of MR signal intensity as a function of time in activated area. Green, blue, and red columns signify FE, BE, and FP’s stimulation periods. The signal look like typical BOLD signal in all of BE (red columns) and FE (blue columns)’s stimulation periods.

We proceed to correlation and consider that each data point consists of two measurements, the averaged blinking and BOLD-signal time course data in each block of the FE and ME conditions. The correlation between those two values is given by the covariance and variance of the two conditions’ blocks. Correlation is dimensionless and takes values limited to the range of –1 to +1. As we observe, correlation between the number of blinks and the average MR-time course data are smaller. Therefore, eye blinking doesn’t affect primary visual cortex activity in our experiment.

4 Discussion Eye movements, both voluntary and involuntary, are the main focuses of this study. Due to their intrinsic nature, they can be possible sources of artifact and confound when interpreting functional MRI data. The blinking process is reported to be controlled in the orbitofrontal and visual cortices, including the anterior portion of the visual cortex and the primary visual cortex [7]. Local eye movements influence whole-head motion correction procedures, resulting in inaccurate movement parameters and potentially lowering the sensitivity to detect activations [5]. For this reason, during fMRI

The Eye Movement and Data Processing Due to Obtained BOLD Signal in V1

71

experimentation, infrared visible light video cameras or fMRI echo planar image data can be used for monitoring eye movement [14, 15]. While these methods appear simple to realize at first view, it is unfortunately not so simple to realize for technical reasons (i.e., synchronization of eye movement with pulse sequences and limited echo planar imaging time). Based on this, an MR-compatible EEG recorder might be a promising tool for monitoring the eye movement. Our results indicate that eye blinking doesn’t affect primary visual cortex and shows that the general fMRI study need not consider the effect of the eye blinking mechanism in data processing. It appears to be enough to instruct the subject to fix their eye on the stimuli in a general fMRI study.

References 1. Ogawa, Lee, T.M., Kay, A.R., Tank, D.W., et al.: Brain magnetic resonance imaging with contrast dependent on blood oxygenation. Proc. Natl. Acad. Sci. 87(24), 9868–9872 (1990) 2. Bandettini, Wong, E.C., Hinks, R.S., Tikofsky, R.S., Hyde, J.S.: Time course EPI of human brain function during task activation. Magn. Reson. Med. 25(2), 390–397 (1992) 3. Ernst, Hennig, J.: Observation of a fast response in functional MR. Magn. Reson. Med. 32(1), 146–149 (1994) 4. Menon, Ogawa, S., Hu, X., Strupp, J.P., Anderson, P., Uğurbil, K.: BOLD based functional MRI at 4 Tesla includes a capillary bed contribution: echo-planar imaging correlates with previous optical imaging using intrinsic signals. Magn. Reson. Med. 33(3), 453–459 (1995) 5. Tregellas, Tanabe, J.L., Miller, D.E., Freedman, R.: Monitoring eye movements during fMRI tasks with echo planar images. Hum. Brain Mapp. 17(4), 237–243 (2002) 6. Freitag, Greenlee, M.W., Lacina, T., Scheffler, K., Radü, E.W.: Effect of eye movements on the magnitude of functional magnetic resonance imaging responses in extrastriate cortex during visual motion perception. Exp. Brain Res. 119(4), 409–414 (1998) 7. Tsubota, Kwong, K.K., Lee, T.Y., Nakamura, J., Cheng, H.M.: Functional MRI of brain activation by eye blinking. Exp. Eye Res. 69(1), 1–7 (1999) 8. Allen, Polizzi, G., Krakow, K., Fish, D.R., Lemieux, L.: Identification of EEG events in the MR scanner: the problem of pulse artifact and a method for its subtraction. Neuroimage 8(3), 229–239 (1998) 9. Kruggel, Wiggins, C.J., Herrmann, C.S., von Cramon, D.Y.: Recording of the eventrelated potentials during functional MRI at 3.0 Tesla field strength. Magn. Reson. Med. 44(2), 277–282 (2000) 10. Allen, Josephs, O., Turner, R.: A method for removing imaging artifact from continuous EEG recorded during functional MRI. Neuroimage 12(2), 230–239 (2000) 11. Muckli, Kriegeskorte, N., Lanfermann, H., Zanella, F.E., Singer, W., Goebel, R.: Apparent motion: event-related functional magnetic resonance imaging of perceptual switches and States. J. Neurosci. 22(RC219), 1–5 (2002) 12. Talairach, Tournoux, P.: Co-Planar Stereotaxic Atlas of the Human Brain. Thieme Medical Publishers, New York (1988) 13. Boynton, Engel, S.A., Glover, G.H., Heeger, D.J.: Linear systems analysis of functional magnetic resonance imaging in human V1. J. Neurosci. 16(13), 4207–4221 (1996) 14. Kimming, Greenlee, M.W., Huethe, F., Mergner, T.: MR-Eyetracker: a new method for eye movement recording in functional magnetic resonance imaging. Exp. Brain Res. 126(3), 443–449 (1999) 15. Beauchamp: Detection of eye movements from fMRI data. Magn. Reson. Med. 49(2), 376–380 (2003)

Face Tracking for Augmented Reality Game Interface and Brand Placement Yong Jae Lee1 and Young Jae Lee2,* 1

Tongmyong University [email protected] 2 College of Culture and Creative Industry, Jeonju University, Korea [email protected]

Abstract. This paper proposes the AR game interface which is more faster and emotive by using an intelligent autonomous agent. As the operation of AR game has designed in accordance gamer’s face tracking, this study applied the movements on game. Since the nature of the game requires the real-time interaction, CBCH algorithm has been selected for face recognition. In case of failed face tracking, the interface agent has been used to provide the gamer with the sound information to be helped with situation perception. Furthermore, retracking has been enabled so that the proposed algorithm could help the gamer to be able to effectively react to the attack. This paper also looked at the design for the new beneficiary model for the game industry through interdisciplinary research between game and advertising. In conclusion, application to the 3D ping-pong game brought about effective and powerful results. The proposed algorithm might be used as fundamental data for developing the AR game interface. Keywords: AR game interface, face tracking, agent, 3D ping-pong game.

1 Introduction With advanced technologies in computer hardware and software, technologies involving game production and game interface are rapidly developing. Recently various and natural interfaces which do not use keyboard or mouse have been developed and in use. Among these, experience-based interface using gamer's motions is in wide application for AR or functional games [1-11]. The AR game allows the gamer to enjoy the exercising effect since it is based on the motions of the body. This kind of game has the advantage in that the gamer could stretch the stiff body after hours of computer use while enjoying the game [1]. However, most of the games which have now been developed for PC use are designed to use limited hardware such as keyboard, mouse and joystick, which is the main cause for confinement to limited space and less sense of reality. Other interface *

Corresponding author.

T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 72–78, 2011. © Springer-Verlag Berlin Heidelberg 2011

Face Tracking for Augmented Reality Game Interface and Brand Placement

73

gadgets such as haptic gloves or glasses and HMD interface have also their own drawbacks several ways: they cause discomfort since they have to be worn on the user's body; they use wires which can bother free movements of the user; they are expensive to be bought for the purpose of playing the game. [1] If we can only use human movements to control the game without the need for keyboard or sensormounted gadgets, we could make the computer user-friendly with more human interface, not to mention enhancing the sense of reality for the game. [2] Thus, for the purpose of this research 3D ping pong game has been developed with no other interfaces except for the webcam and the paper proposes the game interface using gamer's face recognition. When face recognition fails, the interface agent appears and tells the gamer about the current situation with accompanying message. The algorithm for re-recognizing the face has also been proposed and the following experiments proved its performance and validity.

2 AR Games and Agent AR(Augmented Reality) focuses on enhancing fun through the interactive experiences within the environment which combines the elements of the virtual reality and the real-world. There has been steady attempts to introduce the augmented reality into the game technology, as we can see in the examples of bowling game and car racing on the table. "Eye of the Judgement", the card game for PlayStation 3 by Sony is the AR game released in 2008, which shows 3D virtual characters on the table by using card recognition technology[2-6]. There is also a game which is played interactively in a limited space with many markers installed on the ceiling recognizing the hands of the user [1]. Interface agent is also called user interface. Interface agent is the agent which is autonomous and with reinforced learning ability designed to provide more convenient computer-using environment. Interface agent also presents more familiarity to the computer user by introducing human or animal images. Through technologies such as 3D animation, sound synthesis and lip animation, the interface agent provides the environment in which the user might feel as if he or she is conversing with other people or personified animals while using the computer. Peedy of MS and Genie of Argo are the examples. Peedy, the Microsoft agent character, can understand human speech and react to it. Multimodal agent interfaces provide the user with the environment enabling easy input [7-8].

Fig. 1. Eye of Judgement of SONY Corp

74

Y.J. Lee and Y.J. Lee

3 Proposition of the Algorithm and Experiments 3.1 Proposition of the Algorithm CBCH(Cascade of Boosted Classifier Working with Haar-like Feature Algorithm)[911], which is proposed here, works like this. The image input through the webcam is recognized by using CBCH and the coordinate values of the face is changed into the coordinate values for the game interface. In the 3D ping-pong game presented here, the bars for offense and defense are matched with the coordinates and the bar movements are controlled by the face movements. The collision sensing algorithm can detect the collision of the ball and the bar whose action and reaction is to be realized in consideration of the directions of the movement vector. If the face recognition fails in the next frame, the agent notifies the gamer of the situation with sound and text more than once. 3.2 Ping-pong Game For the purpose of the research, the 3D ping-pong game has been custom-produced by using Irrlicht engine, Visual C++, and SAPI(speech engine). It is to be noted that the agent has been used to notify the gamer about the situation regarding face recognition. The background of the game is 800 * 600 sized and 32-bit graphics. Two bars for the offense are based on the faces of the gamer 1 and the gamer 2 respectively. To make the ball three-dimensional, we used the cube scene node and the spare scene node functions of the Irrlicht engine. We made the ball and the bars spin toward the y axis to enhance the sense of reality and to attract the attention of the gamer respectively. Square collision method has been selected for the collision algorithm of the bars and the ball since it assures rapid detection. Background music and sound effect have been added for the game immersion effect. The moment of the collision has been given special auditory effect to emphasize that the 'spinning' ball hits the bar. The agent used for the game is Genie of Argo(Figure 3.) 3.3 Experiment 1 Frame Design The Experiment 1 is about the frame design for the game image using the webcam image and the resulting image has been checked. In Figure 2, we modified the coordinates for the webcam and the game image by using the constant values of alpha and beta and tested the locations on Experiment 1 image. To apply the designed frame in the game, Experiment 1 tested the correspondence between the bar and the face interface. Fig. 3(b) shows the resulting movement of the bar to the corresponding left bottom on the game scene after we had moved the face(game interface)to the left bottom in Fig. 3(a). In Fig. 3(c), the face(game interface) moves to the right top which makes the bar in Fig. 3(d) move to the corresponding right top on the game scene. The two cases confirm that the movement of the face triggers the reaction of the bar, which moves to the corresponding location on the game scene. Therefore, the Experiment 1 verifies that gamer's face could be used as the valid game interface.

Face Tracking for Augmented Reality Game Interface and Brand Placement

75

Fig. 2. Frame Image

(a)

(b)

(c)

(d) Fig. 3. Experiment 1 Image

3.4 Experiment 2 Experiment 2 is to identify the function of the agent in notifying the gamer when CBCH face recognition algorithm fails. In Fig. 3(a), the face recognition fails and the Genie in Fig. 3(b) appears and tells, with the accompanying text, the gamer to adjust. The gamer, upon hearing the message, adjusts the face(the game interface) and moves the face(Fig. 3(c)) to defend not to lose score(Fig. 3(d)). Fig. 3(e) shows defense and offense and Fig. 3(f) is the resulting image. As shown, the role of the agent is very important in scoring and defending. 3.5 Experiment 3. Brand Placement on Game Experiment 3 showed the possibilities of interdisciplinary studies between game and advertising by placing particular brand on the AR game interface. Recently, interdisciplinary studies are drawing scholar’s attention in many fields. That's why the

76

Y.J. Lee and Y.J. Lee

(a)

(b)

(c)

(d)

(e)

(f) Fig. 3. Experiment 2 Image

convergence is getting more important concept in multimedia. In game industry, a new beneficiary model such as brand placement can be a successful way to guarantee their business. And also, prior researches have found out the synergy effects could be maximized by integrating these two academic fields. In other words, advertising tools including brand placement can be developed as an innovative beneficiary model for game industry. In summary, brand exposure on game can serve as an opportunity to raise the brand awareness and preference for the sponsor and the game company can reap profits. It can also raise the enjoyment level for the game user, making its realization highly probable.

Face Tracking for Augmented Reality Game Interface and Brand Placement

77

Fig. 4. Experiment 3 Image

4 Conclusion In this paper, there has been attempt through experiments to apply information regarding gamer's face recognition to the 3D ping-pong game, to find out about the performance of the interface functions. In case of failed face recognition, we used the agent to notify the gamer with speech and text of the current situation. The results of the experiments showed that the effective reaction of the gamer was possible. And we could also realize various roles of the agent through speech and text which include ordering tactics, expressing emotions, cheering and etc. This paper also looked at the design for the new beneficiary model for the game industry through interdisciplinary research between game and advertising. In conclusion, this research verifies that this kind of approach is conducive to increasing fun factor as well as providing extra assistance to the gamer.

References 1. Kim, K.Y., et al.: ARPushPush: Augmented Reality Game in Indoor Environment. In: KHCI, pp. 354–359 (2005) 2. Handheld Augmented Reality Game System Using Dynamic Environment Kang, Won 3. http://www.comp.dit.ie/bmacnamee/papers/MixedRealityGames.pdf 4. http://www.eyeofjudgement.com 5. http://www.google.co.kr/search?hl=ko&newwindow=1&complete=1& q=Eye+of+Judgement&btnG=%EA%B2%80% 20%20%20%20%20%20%20%25%20EC 6. Hyung: Handheld Augmented Reality Game System Using Dynamic Environment. KAIST, Thesis for Master’s Degree (2007) 7. Lee, Y.J., Lee, Y.J.: Interface of Augmented Reality Game Using Face Tracking and Its Application to Advertising. Security-Enriched Urban Computing and Smart Grid Communications in Computer and Information Science 78, 614–620 (2010)

78

Y.J. Lee and Y.J. Lee

8. Lee, Y.J., Lee, Y.J.: The Application of Agent and Advertising in 3D Sports and Game. Journal of the Korea Institute of Marine Information and Communication Sciences 14(10), 2269–2276 (2010) 9. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: IEEE Conf. on Computer Vision and Pattern Recognition, Kauai, Hawaii, USA (2001) 10. http://www.lienhart.de/ICIP2002.pdf 11. http://cafe.naver.com/opencv.cafe?iframe_url=/ ArticleRead.nhn%3Farticleid=1328

On-line and Mobile Delivery Data Management for Enhancing Customer Services Hyun-Chang Lee1, Seong Yoon Shin2, and Yang Won Rhee2 1

Division of Information and e-Commerce, Wonkwang University 344-5, Sinyoung-Dong, Iksan-Si, Jeonbuk, Korea [email protected] http://www.wku.ac.kr 2 Department of Computer Information Eng., Kunsan University {s3397220,ywrhee}@kunsan.ac.kr http://www.kunsan.ac.kr

Abstract. With the growing trend toward the use of supply chain and ecommerce, logistic service providers for product warehousing, transportation and delivery are placing great emphasis on information technology (IT) to be competitive globally. Realizing the current service tracking system merely supports order status tracking within a service provider, applies mobile agent technology for customer satisfaction index through online delivery tracking across the logistic alliances. Therefore, in this paper, we propose a system that utilizes three-tier architecture for mobile agent technology and develops a prototype system for logistic delivery service tracking to satisfy customers who want to know the location of their things in detail. Also, we demonstrate the concept and technology proposed with mobile and internet. The online service tracking services enable customers to monitor the real-time status of their service requests through internet and therefore becomes key tool for modern enterprises to compete successfully in a global marketplace.

1 Introduction In a context of growing market globalization, rapid diffusion of information and communication technologies and increasingly widespread de-localization of manufacturing activities, product mobility is preferred to their accumulation in warehouses, involving a continuous flow of materials through the supply chain [1]. Therefore, firms have to supply distant markets from their own warehouses and plants with always-greater frequency. This also involves relevant expenditures. [2] for instance states that transportation costs on average impact for about 50 percent of total logistic costs. This requirement will become even more critical with the progressive diffusion of e-commerce activities [4]. The technique for customer contentment enhances needs also from the perspective of e-commerce activation. Such factors make the distribution logistics increasingly important and often critical for competitiveness of companies [7] justifying major efforts to reduce logistic T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 79–89, 2011. © Springer-Verlag Berlin Heidelberg 2011

80

H.-C. Lee, S.Y. Shin, and Y.W. Rhee

expenditures to the greatest extent possible in order to delivery goods at a reduced price and to enhance a customer contentment to customers. According to statistics from Council of Logistics Management, 20 to 30 percent of total production costs are directly attributed to distribution and logistics management [8]. Investment in efficient distribution and logistics management systems can considerably improve enterprise competitiveness. Further, logistics management can significantly affect the efficiency of production, distribution and the quality of total customer services. Distribution and logistics management consist of a series of activities including warehousing, order manipulation, goods picking and dispatching, transportation and inventory control [9]. Since a vast amount of information is generated across the various logistic activities and participants, efficiency and quality of enterprise logistic services are hard to monitor, control and also provide with efficient and high-quality customer services. Moreover, since many participants such as freighters, distribution centers, manufacturers, and distributors are involved in the logistic service industry, in this paper we present them with a model to enhance customer services by providing a part containing real-time monitor function and show the prototype system to implement for service enhancement to customers. The rest of this paper is organized as follows. In section 2, a short review of on-line service tracking and logistics management is discussed. We explain a logistic service system analysis and design in Section 3. In Section 4, we present our technical prototype system for enhancing customer services. Finally, we conclude with some comments in section 5.

2 Logistics Management of Customer Service Perspective This paper proposes an approach for customer service enhancement through online global logistic services using mobile agents. In this section, the literature covering online service tracking, agent-based techniques and logistics management is reviewed to formulate the research questions and basis for the approach. 2.1 Online Service Tracking Along with the tremendous development in industrial engineering and management applications, including supply chain management (SCM), customer relationship management (CRM), and global logistics, various computer-aided applications have been developed to assist implementation enterprises. The online logistic service tracking system is one of the more effective solutions that have been developed to support efficient customer service response. In the competitive global market, enterprises should efficiently respond to customer requests to gain market advantage. Service status tracking is the fundamental offer to provide customers a means to realize the status of their requests and to anticipate and plan actions. For a manufacturer down stream in the supply chain, this service provides real-time information that enhances the effectiveness of raw material planning and scheduling.

On-Line and Mobile Delivery Data Management for Enhancing Customer Services

81

Since the service tracking system provides the order and delivery status of the products and services, users of the system can make decisions based on the actual status. Unlike the traditional approach, the Internet-based technique including wireless environment has the advantage that information exchange and transmission are not geographically restricted. Realizing the importance of efficient response to customers, traditional, non-Internet-based approaches for business transaction and communication have gradually been replaced by Internet means [10]. With the development of the Internet and wireless technique, numerous service tracking systems have gone online. Though users can easily access real-time status information via Web -based service tracking systems, most tracking systems cannot confirm an accurate place where his or her material is after departure. Therefore, this research aims at developing an Internet and Wireless-based logistic service tracking system for efficient feedback of service status. 2.2 Logistics Management The distribution center does not only take charge of a role as a hub for the storage, transportation to customers and logistic service providers, but also play a part as a control center of the distribution as well as the management of all of the logistics for the successful operations [13]. The well-known logistics companies such as UPS and FedEx have better organized distribution center than others. Furthermore, because the efficient delivery from the supplier to the customer relies on the information management of transportation and warehousing and so on, the real time service system like a RSSOL must have a control algorithm of the data from the diversified logistic activities and show the accurate position of customer items to enhance customer services.

3 Design and Analysis for Real-Time Delivery Service In this section, the business analysis, architecture and functions of the proposed logistic service tracking system are discussed. The objectives of the proposed approach are to provide customers with an intelligent mechanism to track the service. The system is designed for the following two participants to meet their operational requirements and to enhance their business competitiveness: (1) Customers. Expect their requests for products or services from distributors and manufacturers to be fulfilled on time. (2) Logistic service providers. Provide logistic services with a range of information support provided to buyers in order to enhance the business efficiency of the customers. 3.1 Coordination Model Analysis A general procurement processes within a supply chain are described as shown in figure 1[13]. At first, a consumer places an order on the retailer and then the retailer may request the logistic service provider to deliver the goods, and vice versa in the delivery process from the logistic service provider to retailer, finally to the consumer.

82

H.-C. Lee, S.Y. Shin, and Y.W. Rhee

As shown in figure 1, the transportation plays an essential role in distributing and delivering the merchandise to the consumers, the retailers, the manufacturers and the raw material suppliers. However, in case of the international commerce including ecommerce, the delivery must take advantage of intermediary distribution centers and overseas service providers of the logistics between suppliers and foreign customers in addition to the domestic logistic services.

Fig. 1. Configuration of the Supply Chain

Fig. 2. Three-tire architecture for supply chain

On-Line and Mobile Delivery Data Management for Enhancing Customer Services

83

In the delivery process from an order to the customer, the real time localization of customer’s deliver items are critical for the satisfied services and more efficient Logistic Services than before. Therefore, in this paper, we propose a RSSOL prototype system using the facility of logistics. 3.2 System Design The configuration of the RSSOL in figure 2 shows three-tire architecture of three parts: the agent center as a service-tracking kernel, the user interactive and real time logistic service and the delivery operation center as a carrier. The RSSOL consists of three parts proposed in figure 2 and we explain two points of view to enhance customer services as follows: At the customer side: 1. 2.

Ordering phase: the customer can order through the internet after logging on. Delivery status tracking phase: - Log on internet, and then - Lists of the serviced items with delivers charging of the items are displayed on the web page. - A customer can search for the location of a deliverer by pushing the last specific hyperlink text. - The delivered status of the items is displayed at the bottom side of the screen. - The customers can estimate the arrival time by confirming the location of the deliverer on the internet, which is one of the enhanced services, and finding algorithm of the short cut path.

At the Internet service provider (ISP) side: 1. 2. 3. 4.

A deliverer decides which item to deliver the next based on the displayed data. The deliverer logs on the server using mobile device and update the items data after delivering, also one of the services which make customer take services. The deliverer location and item is updated simultaneously by sending mobile information with signature of the customer to the server after step 2. If any items to be delivered are remained, then continue from 2 to 3, or finish LSP side activity.

4 Implementation and Evaluation of RSSOL In this section, we verify the superiority of the proposed method by implementation and evaluation. 4.1 Environment for Delivering To implement the RSSOL to enhance customer services, we first look into the environment of delivering system through postal operations. The addressing system of

84

H.-C. Lee, S.Y. Shin, and Y.W. Rhee

Korea is composed of rather complex than that of other countries. The addressing system of Korea is configured as shown in figure 3(a). We can also mention that the addressing system is similar to other countries. For example, 4 or at most 5 items (nation, state, city, street, address) among items in figure 3(a) are enough information to deliver in the addressing system of America. Nonetheless, ISPs only use the information of city or station level to deliver any product or things to customers. This means that customers through internet can get the information of their products delivered till the city or station level of that city although street information exist. Therefore, in this paper, we emphasize and use that information to enhance customer services. It is also one of main contributions of the paper. For instance, a customer who does not receive his or her product yet can not estimate time to receive the product, whereas the customer using this proposed system, RSSOL, can estimate the time to receive products because the customer can know roughly the distance between the customer location and the last delivered location of deliverer through internet.

Nation

State

City

district

street

address

a) Components of addressing system 100-101 Sangrok apartment, Ilwon-dong, Gangnam-gu, Seoul, Korea b) Example for postal system to deliver in Korea Fig. 3. Korean record for addressing system

For RSSOL, we design and show the implementation result. First, we implemented RSSOL by dividing it into two parts: one is customer side and the other is service provider one as discussed in section 3. For these directions, we can first consider the following service provider side. 4.2 Service Provider Side to Deliver ISPs take the responsibility of delivering customer’s products safely and get handwrite information of customers after delivering by deliverer using PDA in realtime. However, because of the sophisticated contents of the address record until to the address, they can give an exact localization as well as an estimation of the arrival time for their items through internet accesses of customers. Figure 4 shows a delivering example of a deliverer using PDA that accesses to RSSOL. As for first step, deliverer like figure 4(a) can login to connect the server, RSSOL. Then in the figure 4(b), the receiver’s names and addresses to be delivered for a day are displayed in the PDA screen after clicking the left lower button, “Bring”

On-Line and Mobile Delivery Data Management for Enhancing Customer Services

85

가져오기

(data from server, “ ” in korean), in 4(b). If the deliverer wants to know more information of a customer chosen, then by clicking the center button “Detail”(“ ” in Korean), the deliverer could get the result like figure 4(c). At step 2, e.g. after delivering the customer’s product, deliverer clicks the right lower button, “send” (“ ”) to the RSSOL as a last step for confirming and storing the delivering information. So we can see the PDA screenshot to be deleted the last customer information in figure 4(d).

자세히 보내기

(a) deliverer login

(b) item list to be delivered

(c) customer information (d) update after delivering Fig. 4. A series of operations at a deliverer side using mobile device

4.3 Customer Side Any customers who want to localize their items or know to distribution status can take advantage of this RSSOL, confirm the deliverer’s location and estimate the time to get their items through internet easily. Figure 5 shows the processes of customer

86

H.-C. Lee, S.Y. Shin, and Y.W. Rhee

side: figure 5(a) for ordering phase of a customer and 5(b) for the neighboring delivery list of deliverer within ordering information of customers that is including facts whether customer items are delivered or not. If a customer connects to RSSOL through web browser, the customer could see the deliverer position by confirming whether the previous customers (first to be delivered) of the customer get their items or not. If the previous other customers have not got their items, the customer could estimate the time to get his or her item like figure 6(a) to 6(c) only by clicking the last field and by confirming the status at the bottom side as well. The status of delivering items is displayed on the bottom side as shown in figure 6(a) for the first delivery. Next order is shown in a figure 6(b) for the second delivery and so on. For example, in case of “hclee” (in Korean “ ”) which will be delivered from now on, any sign is not displayed at the bottom side and only he is able to estimate the time to get his items by the previously explained procedure. Figure 7 shows a comparison result of RSSOL with conventional logistics methodology for the delay in logistic service. We see that the more there are customers, the better we get the result of delay unit.

이현창

a) ordering phase

b) delivery list of deliverer Fig. 5. Steps through the internet at a customer side

On-Line and Mobile Delivery Data Management for Enhancing Customer Services

a) 1st delivery state

b) 3rd delivery state

c) state for a specific customer “hclee” Fig. 6. Monitoring states of being delivered

87

88

H.-C. Lee, S.Y. Shin, and Y.W. Rhee

Fig. 7. Real-time service delay according to the number of customers

5 Conclusion According to the popularities of supply chain and e-commerce, the ISPs recognize importance of the information technology (IT) in their business as real time monitoring tools. The customer’s satisfaction must be more important than before and critical in leading competitiveness of companies through the IT. Hence, in this paper we propose the RSSOL in the logistics and implemented it by the PDA combined with web pages, and show the usefulness of the RSSOL in real time localization. Especially, the ISPs can cut down the Logistic expenditures because of monitoring the materials flows in real time and scheduling their products, and also the clients can get a chanced services through the real time localization and estimation to get their items.

References 1. Zografos, K.G., Giannouli, I.M.: Emerging trends in logistics and their impact on freight transportation systems: a European perspective. Transportation Research Record (1790), 36–44 (2002) 2. Weil, M.: Moving more for less. Manufacturing Systems 16, 90–94 (1998) 3. Frank, L.: Architecture for integration of distributed ERP systems and e-commerce systems. Industrial Management & Data Systems 104(5), 418–429 (2004) 4. Caputo, A.C., Cucchiella, F., Fratocchi, L., Pelagagge, P.M., Scacchia, F.: Analysis and evaluation of e-supply chain performances. Industrial Management & Data Systems 104(7), 546–557 (2004) 5. Ho, D.C.K., Au, K.F., Newton, E.: The process and consequences of supply chain virtualization. Industrial Management & Data Systems 103(6), 423–433 (2003) 6. Soliman, F., Youssef, M.A.: Internet-based e-commerce and its impact on manufacturing and business operations. Industrial Management & Data Systems 103(8), 546–552 (2003) 7. Lai, K.H., Ngai, E.W.T., Cheng, T.C.E.: An empirical study of supply chain performance in transport logistics. International Journal of Production Economics 87, 321–331 (2004) 8. Ministry of Economic Affairs (MoEA), Yearbook of Logistic Service Industry in Taiwan, Taipei (2000) 9. Aitken, J.: Supply chain integration within the context of a supplier association. Cranfield University, Cranfield, PhD thesis (1998)

On-Line and Mobile Delivery Data Management for Enhancing Customer Services

89

10. AIMO: Artificial Intelligence Modelling Organisation, AIMO (1999), http://aimo.kareltek.fi/ 11. Anton, J.: Call Center Benchmark Report, Purdue University, West Lafayette, IN (2000) 12. Amy, J.C., Charles, V., Hou, J.-L., Chen, B.J.G.: Mobile agent technology and application for online global logistic services. Industrial Management & Data Systems 104(2), 169– 183 (2004) 13. Huang, S.-M., Kwan, I.S.Y., Hung, Y.-C.: Planning enterprise resources by use of a reengineering approach to build a global logistics management system. Industrial Management & Data System 101(9), 483–491 (2001)

Design of LCL Filter Using Hybrid Intelligent Optimization for Photovoltaic System Jae Hoon Cho1, Dong-Hwa Kim2, Mária Virčíková3, and Peter Sinčák3 1

Chungbuk National University, Cheongju Chungbuk, Korea 2 Hanbat National University, Daejeon, Korea 3 Dept. of Cybernetics and Artificial Intelligence, FEI TU of Košice, Slovak Republic {jhcho1975,koreahucare}@gmail.com, {maria.vircikova,peter.sincak}@tuke.sk

Abstract. This paper proposes new design method of LCL filter parameters by using hybrid intelligent optimization. Usually, the voltage-source inverters (VSI) have been used for control of both link voltage and power factor in the grid connected topologies. But, since VSI can cause high-order harmonics by high switching frequency, it is important to choose the filter parameters to achieve good filtering effect. Compared with traditional L and LC filter, LCL filter is known as an effective method on reducing harmonic distortion at the switching frequency of VSI. However, design of the LCL filter is complex, and the selection of the initial values of the inductors is difficult at the start of the design process. This paper proposes an approach for designing LCL filter parameters by hybrid optimization method with both genetic algorithm and clonal selection. Simulation results are provided to prove that the proposed method is more effective and simple than conventional methods. Keywords: LCL filter, voltage-source inverters, genetic algorithm, clonal selection.

1 Introduction Renewable energy sources such as photovoltaic, micro turbines, small wind turbines, and fuel cell have received plenty of global attention due to depletion of the Fossil fuel and environment pollution. In particular, photovoltaic system has received great attention because it has many advantages such as simplicity of allocation, nonpolluting production, silent operation, long life time, low maintenance and absence of fuel cost [1-2]. Photovoltaic system, however, need to adopt means of storage to meet sustained load demands since the output power of photovoltaic system is usually dependent on the weather conditions and other external factors. The generation system with photovoltaic array can be operated in stand-alone mode and grid-connected mode. In grid-connected mode, this system need power conditioning units for interfacing with grid and load. Since PV sources are dc power supply, so an inverter is required for dc/ac conversion. Voltage-source inverter(VSI) are now commonly used in many power conversion applications including renewable source and distributed generation systems. The VSI has many advantages such as T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 90–97, 2011. © Springer-Verlag Berlin Heidelberg 2011

Design of LCL Filter Using Hybrid Intelligent Optimization for Photovoltaic System

91

bidirectional power flow, controllable power factor and sinusoidal input current. The inverter with an L filter or LC filter is simple in control, but the ability of highfrequency harmonics current suppression is limited. A PWM inverter with higher switching frequency will result in smaller LC filter size. However, the switching frequency is generally limited in high power applications. In order to solve this problem, LCL filter is proposed to decrease the inductance of L without depreciate filter effect[3]. The LCL filter has advantages compared with LC filter in a great attenuation with high-frequency harmonics, and can meet the requirements of harmonics attenuation even in a lower switching frequency and smaller inductors [45]. In spite of such advantages, LCL filter has its own drawbacks. A modern recursive method and some constraints were suggested in by trial and error method to decide on LCL parameters of a grid-connected voltage source inverter. But the method is complicated and selection of the initial values of the inductors is difficult at the start of the design process [6-7]. For the best performance of an LCL filter, the design of filter parameters by intelligent optimization methods have been developed [8-9]. Artificial intelligence methods has become a popular search technique used in computing to find exact or approximate solutions to optimization and search problems. Genetic algorithms are a particular class of evolutionary algorithms inspired by biology such as inheritance, mutation, selection, and crossover. To improve the performance of simple GA, various hybrid methods have been proposed and these obtained better performance than simple GA in most case [10-11]. On the other hand, Castro and Zuben [12] presented a clonal selection algorithm to solve complex problems such as learning and multi-modal optimization. The clonal selection algorithm inspired by immune system has the ability of getting out local minima, operates on a population of points in search space simultaneously, not on just one point, and does not use the derivative or any other information. This paper proposes an approach for designing LCL filter parameters by hybrid optimization method with both GA and clonal selection. For the simulation of the proposed method, PV system and LCL filter are used by Matlab/Simpowersystem. Simulation results are provided to prove that the proposed method is more effective and simple than both traditional methods and other intelligent optimization methods.

2 Grid-Connected PV System and Simplified Design Principle of LCL Filter 2.1 Grid-Connected PV System with VSI The power circuit of the grid connected PV system, as shown in Fig.1, consists of a PV array, ultracapacitor(UCB) as storage unit, a dc link capacitor that keeps dc voltage constant, a 3 phase voltage source inverter(VSI), LCL filter and utility grid. The LCL-filtered VSI topology has been reported for utility interfacing of renewable source or distributed generation and provides higher harmonic attenuation than that of conventional filter such as L and LC filter. The LCL filter, however, requires a proper filter parameters to avoid resonance problem which deteriorate stability of the system[13].

92

J.H. Cho et al.

Fig. 1. Topology of three phase VSC with LCL filter

The three-phase voltage source inverter connected grid through LCL filter is shown in Fig.1, where i1 is the inverter-side current, i2 is the grid-connected current, iC is the filter-capacitor current. The traditional design LCL filter can be calculated by the following equations[8]. L1 =

Vg 2 6 f sw iripple, peak

C2 ≤ 0.05Cb 1 (ωn Zb )

Cb =

Zb =

2 VgLL

Pn

(1) (2) (3)

(4)

Vg : the RMS value of grid voltage, fsw : inverter switching frequency. Cb : the base capacitance, Z b : base impedance, VgLL : grid line voltage Pn : inverter rated power. iripple, peak denotes peak value of harmonic current, it used to be 15% of peak value of fundamental current. C2 = 0.025Cb

(5)

For a further attenuation of current ripple of 20dB at the switching frequency, the grid side inductance of LCL filter L2 is computed as follow. L2 = 0.8L1

(6)

Design of LCL Filter Using Hybrid Intelligent Optimization for Photovoltaic System

93

3 Design of LCL Filter Parameters by Hybrid Intelligent Optimization 3.1 Clonal Selection Algorithm De Castro and Von Zuben [12] presented a clonal selection algorithm to solve complex problems such as learning and multi-modal optimization. A clonal selection algorithm is primarily focused on mimicking the clonal selection principle which is composed of the mechanisms; clonal selection, clonal expansion, and affinity maturation via somatic hypermutation[15] and is used to describe the basic features of an immune response to an antigenic stimulus. When an antigen encounters the immune system, its epitopes eventually will react with a B-lymphocyte with B-cell receptors on its surface that more or less fit and this activates that B-lymphocyte. This process is known as clonal selection. The main features of the clonal selection theory are as follows [12]: 1) generation of new random genetic changes, subsequently expressed as diverse antibody patterns by a form of accelerated somatic mutation. 2) phenotypic restriction and retention of one pattern to one differentiated cell (clone). 3) proliferation and differentiation on contact of cells with antigens. The overall procedure of clonal selection is schematically shown in Fig. 2.

Fig. 2. Clonal selection theory

3.2 Constraints for Designing the Parameters of LCL Filter Some constraints are considered to insure the performance of LCL filter[8].

a. b.

To insure a high power factor, a constraint of the capacitor is needed : C2 ≤ 0.05Cb Consider the stability of the control system and the loss of power switch devices, the total inductance should be lower than 0.1 pu(Per Unit) that is L1

+ L2 < 0.1Lb

94

J.H. Cho et al.

c.

To avoid the resonance at lower order or high order frequency, the resonance frequency should be 10 f n ≤ f res ≤ 0.5 f sw ig ( hsw ) i ( hsw )

=

2 Z LC 2 2 ωres − ωsw

2

L Z 2 ωres = T LC , LT = L1 + L2 , Z LC = L1

(7)

1 L2C f

2 ωsw = (2π f sw )

where ig ( hsw ) / i ( hsw ) is harmonic attenuation rate.

These constraints can be used for the object function of intelligent optimization algorithm to achieve the required filter effort with inductor and capacitor value as small as possible. In this paper, the object function of the proposed algorithm is calculated by Eq. (8) [16]. Object _ F = w * Har + (1 − w) * Total _ L + penalty _ fun.

⎧if ⎨ ⎩

f res is 10 f n ≤ f res ≤ 0.5 f sw , penalty _ func = 0 otherwise

, penalty _ func = 10

(8)

(9)

The first term in (8) is harmonic attenuation rate (Har) with a weight function w and the next term is total inductance. The penalty function can be used for the above mentioned constraint c.

4 Simulation and Results In order to verify the proposed design method, after LCL fitler parameters are evaluated by the proposed method, the parameters are applied to the simulation model based on MATLAB/ SimPowerSystems. Table1 shows the system parameters and the initial parameters of the proposed method. Figure 3 shows current output of VSI with LCL fitler parameters selected by general method and the proposed method. Figure 4 provides a zoomed-in view of part of A in figure 3. As shown in Table 2, the total inductance of the proposed method are lower than others and the fitness of the proposed method is higher than others which means the harmonic is lower than others. As shown in figure 3 and 4, the LCL filter with a low harmonic distortion can be obtained by the proposed method.

Design of LCL Filter Using Hybrid Intelligent Optimization for Photovoltaic System Table 1. The system parameters and initial parameter of the proposed method PV array

Values

PV Power (Pn)

12kW

Utility line (VgLL) Utility frequency(fn) Switching frequency (fsw) DC rated voltage (Vdc)

380V 60Hz 10kHz 700V

Ultracapacitor bank

Values

Capacitance Internal series resistance Voltage Short-circuit current

165 [F] 6.1[mΩ] 48.6[V] 4800[A]

The Proposed algorithm

Values

Population size Probability of crossover of gene Probability of mutation Iteration for the proposed algorithm The No. of clones Probability of crossover of clone Weight for object function

100 0.65 0.05 200 5 0.25 0.5

Table 2. Simulation results for the proposed method and others

Methods

L1 (mH)

L2 (mH)

Cf (uF)

Normal

1.28

1.12

3.89

3.28

0.78

GA

1.22

0.98

4.65

4.66

0.83

The proposed method

1.19

0.77

4.65

4.87

0.87

fres (kHz)

Fig. 3. Current waves by general method and the proposed method

fitness

95

96

J.H. Cho et al.

Fig. 4. Zoomed plot of current waves in figure 3

5 Conclusion Usually, the voltage-source inverters (VSI) have been used for full control of both dclink voltage and power factor in the grid connected topologies. But, the VSI can cause high-order harmonics disturbing other sensitive loads/equipment on the grid. This paper proposes an approach for designing LCL filter parameters by hybrid optimization method with both GA and clonal selection. In order to verify the proposed design method, after LCL fitler parameters are evaluated by the proposed method, the parameters are applied to the simulation model based on MATLAB/ SimPowerSystems. Simulation results are provided to prove that the proposed method is more effective and simple than both traditional methods and other intelligent optimization methods. Acknowledgments. The work developed in this paper has been supported by the DAEDEOK INNOPLIS (“R&D Hub Cluster project”).

References 1. Molina, M.G., Mercade, P.E.: modeling and control of grid-connected photovoltaic energy conversion System used as a Dispersed Generator. In: 2008 IEEE PES Transmission and Distribution Conference and Expositon, pp. 1–8 (2008) 2. Fakham, H., Degobert, P., Francois, B.: Control system and power management for a PV based generationunit including batteries. In: International Aegean Conference on Electrical Machines and Power Electronics, pp. 141–146 (2007) 3. Lindgren, M., Svensson, J.: Connecting Fast Switching Voltage Source Converters to the Grid-Harmonic Distortion and its Reduction. In: IEEE Stockholm Power Tech Conference, pp. 18–22 (1995) 4. Halimi, B., Dahono, P.A.: A Current control method for phase-controlled rectifier that has an LCL filter. In: 4th IEEE International Conference on Power Electronics and Drive Systems, pp. 20–25 (2001)

Design of LCL Filter Using Hybrid Intelligent Optimization for Photovoltaic System

97

5. Yin, J.J., Duan, S., Zhou, Y., Liu, F., Chen, C.: A novel parameter design method of dualloop control strategy for grid-connected inverters with LCL filter. In: IPEMC 2009, pp. 712–715 (2009) 6. Liserre, M., Blaabjerg, F., Hansen, S.: Design and Control of an LCL-filter based Threephase Active Rectifier. IEEE Trans. Industry Applications 47, 1281–1291 (2005) 7. Tavakoli Bina, M., Pashajavid, E.: An efficient procedure to design passive LCL-filters for active power filters. Electric Power Systems Research 79, 606–614 (2009) 8. Sun, W., Chen, Z., Wu, X.: Intelligent optimize design of LCL filter for three phase voltage-source PWM rectifier. In: IEEE 6th IPEMC 2009, pp. 970–974 (2009) 9. Chen, Y.M.: Passive filter design using genetic algorithms. IEEE Trans. Industrial Elec. 50, 202–207 (2003) 10. Grimaccia, F., Mussetta, M., Zich, R.E.: Genetical Swarm Optimization: Self-Adaptive Hybrid Evolutionary Algorithm for Electromagnetics. IEEE Trans. on Antennas and Propagation 55, 781–785 (2007) 11. Kao, Y.T., Zahara, E.: A hybrid genetic algorithm and particle swarm optimization for multimodal functions. Applied Soft Computing 8, 849–857 (2008) 12. De Castro, L.N., Von Zuben, F.J.: Learning and optimization using the clonal selection principle. IEEE Trans. Evol. Comput. 6, 239–251 (2002) 13. Lee, K.J., Park, N.J., Kim, R.Y., Ha, D.H., Hyun, D.S.: Design of an LCL filter employing a symmetric geometry and its control in grid-connected inverter applications. In: 39th IEEE Power Electronics Specialists Conference, pp. 963–966 (2008) 14. Wang, T.C.Y., Ye, Z., Sinha, G., Yuan, X.: Output filter design for a grid-interconnected three-phase inverter. In: 34th IEEE Power Electronics Specialists Conference, pp. 779–784 (2003) 15. Brownlee, J.: Clonal selection algorithms. Technical report id: 070209a, Victoria, Australia: ComplexIntelligent Systems Laboratory (CIS), Centre for Information Technology Research (CITR), Faculty of Informationand Communication Technologies (ICT), Swinburne University of Technology (2007) 16. Cho, J.H., Yoon, J.H., Cho, Y.I., Chun, M.G.: LCL filter Design for Renewable Energy Sourceusing Advanced Bacterial Foraging Optimization. In: SCIS & ISIS 2010, pp. 618– 623 (2010)

Distributed Energy Management for Stand-Alone Photovoltaic System with Storages Jae Hoon Cho1 and Dong Hwa-Kim2 1

Chungbuk National University, Cheongju Chungbuk, Korea 2 Hanbat National University, Daejeon, Korea {jhcho1975,koreahucare}@gmail.com

Abstract. This paper presents a distributed energy management system (DEMS) for stand-alone photovoltaic(PV) system with storage devices. Usually, batteries can be used as the storage device for PV system to compensate PV output power changed by irradiation and temperature. Recently, Ultracapacitor has been adopted for the better dynamic characteristic of a whole system. The control of these systems, called hybrid system, often is complicated and need to design a proper energy management system(EMS). DEMSs are more effective than conventional centralized EMS, but it is difficult to design the DEMS for a power system because of various states of the system during operation of standalone mode. To verify the performance of the proposed DEMS, the stand-alone photovoltaic(PV) system is designed by Matlab/SimPowerSystems and Matlab/StateFlow tool is used for designing the proposed DEMS. Keywords: distributed energy management systems(DEMSs), centralized EMS, stand-alone photovoltaic(PV) system.

1 Introduction Because of the ever increasing energy consumption, the soaring cost and the exhaustible nature of fossil fuel, many researches focus on renewable sources such as wind turbine, photovoltaic(PV) energy, fuel cell and micro-turbine, etc. Particularly, because of some advantages such as abundance, pollution free, simplicity of allocation, absence of fuel cost, PV system are becoming popular as a promising alternative source[1-4]. However, the stand-alone PV system need to add certain energy storage devices since the output power generated by a PV system is highly dependent on weather conditions. For example, during cloudy periods and at night, a PV system would not generate any power. In addition, it is difficult to store the power generated by a PV system for future use. To overcome this problem, a PV system can be integrated with other alternate power sources and/or storage systems, such as battery and UC bank [5-6]. For the integrated system with more than two storage systems, a proper energy management system(EMS) are usually required[7]. The energy management system can be categorized into two types: centralized EMS(CEMS) and distributed EMS(DEMS). DEMS are more effective than conventional CEMS, but it is difficult to design the DEMS for power system because of the complex control strategy according to several distributed controller. T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 98–106, 2011. © Springer-Verlag Berlin Heidelberg 2011

Distributed Energy Management for Stand-Alone Photovoltaic System with Storages

99

This paper proposes distributed energy management system(DEMS) for standalone PV system with storages. To verify the performance of the proposed DEMS, the stand-alone photovoltaic(PV) system is designed by Matlab/SimPowerSystems and Matlab/StateFlow tool is used for designing the proposed DEMS.

2 System Description and Modeling The system model, as shown in Fig. 1, is described for the proposed DEMS. Here, to sustain the power demand and solve the energy storage problem, battery and ultracapacitor is connected in parallel through bi-directional DC-DC converter. The system composed of PV system, ultracapacitor bank(UCB), and battery and each component is connected to the same DC Bus through the DC-DC converter with distributed controller. Both UCB and Battery have bi-directional DC-DC converter for charging and discharging, respectively. The boost DC-DC converter connected in PV system is used for maximum power tracking(MPPT). In this system, IncCond MPPT algorithm is adopted [8].

Fig. 1. The stand-alone PV system with storages

2.1 PV System A PV system consists of many cells connected in series and parallel to provide the desired output terminal voltage and current. The PV system can be modeled by various mathematical methods [9-10]. In this paper, the PV system with a 2D-Lookup table and a controlled current source is used for simulation time and computational efficiency. The irradiance data and the I-V characteristic curve of PV array can be used for modeling the PV output power. Fig. 2 shows PV system Matlab/simulink model using the 2D-Lookup table and the controlled current source. The input data for 2D Look-up table are irradiance data and output voltage of PV system.

100

J.H. Cho and D.-H. Kim

Fig. 2. Matlab simulink model for PV array Table 1. Parameters of the PV array PV system Parameters The number of series cells per string The number of parallel cells per strings Ideality or completion factor Boltzmann’s constant PV cell temperature Electron charge Short-circuit cell current (A) PV cell reverse saturation current (A) Series resistance of PV cell (Ω)

Values 105 148 1.9 1.3805e23[J/K] 298 [K] 1.6e-19 C 2.926 0.00005 0.0277

2.2 Ultracapacitor Bank (UCB) Model The ultracapacitor is an energy storage device that is able to handle fast fluctuations in energy level. When comparing ultracapacitors with batteries, they have a significantly lower energy density than the batteries but they have a higher power density compared with batteries[11]. For practical applications, the terminal voltage determines the number of capacitors which must be connected in series to form a bank and the total capacitance determines the number of capacitors which must be connected in parallel in the bank. The detailed model including various characteristic of ultracapacitor can be used in modeling ultracapacitor. But, for the simplicity of simulation, a simplified equivalent model that is only used for principle verification has been used in many papers [9,12]. The parameters used in the mathematical modeling of the UC bank are as follows[9]: C CUC-total ESR, R EUC ns np RUC-total Vi Vf

capacitance [F] the total UC system capacitance [F] equivalent series internal resistance [U] the amount of energy released or captured by the UC bank[W/s] the number of capacitors connected in series the number of series strings in parallel the total UC system resistance [U] the initial voltage before discharging starts [V] the final voltage after discharging ends [V]

Distributed Energy Management for Stand-Alone Photovoltaic System with Storages

101

The amount of energy drawn from the UC bank is directly proportional to the capacitance and the change in the terminal voltage , given by EUC =

(

1 C Vi 2 − V f2 2

)

(1)

The total resistance and the total capacitance of the UC bank may be calculated as

RUC −total = ns

ESR np

(2)

CUC −total = n p

C ns

(3)

Table 3. Ultracapacitor module parameters UC parameters

Value

Capacitance [F]

165

Internal series resistance (dc) [mΩ]

6.1

Leakage current [A]

0.0052, 72 h, 25 C

Operating temperature

40 C to 65 C

Voltage [V]

48.6

Short-circuit current [A]

4800

Power density [W/kg]

7900

Energy density [Wh/kg]

3.81

2.3 Battery Model In this paper, electric circuit-based battery model is used for representing state of charge (SOC) estimations of battery packs. The battery can be modeled by using a simple controlled voltage source in series with a constant resistance. This model assumes the same characteristics for the charge and the discharge cycles and the open voltage source that can be calculated with a non-linear equation based on the actual SOC of the battery [13]. Fig. 3 shows the Matlab/SimPowersystem model for battery. The controlled voltage source is described by equation (4): E = E0 − K

Q

∫

Q − idt

Vbatt = E − Ri

E = no-load voltage(V) E0 = battery constant voltage(V) K =polarization voltage(V)

∫

+ A exp(− B ⋅ idt )

(4)

(5)

102

J.H. Cho and D.-H. Kim

Q =battery capacity(Ah)

∫ idt =actual battery charge(Ah) A =exponential zone amplitude(V) -1 B =exponential zone time constant inverse(Ah) Vbatt =battery voltage(V) R =internal resistance( Ω ) i =battery current(A)

Fig. 3. The SimPowerSystems model of battery

3 Distributed Energy Management System (DEMS) for PV System In the stand-alone PV system, since demanded load power must be only satisfied by PV system and storage devices, a proper energy management is essential. The storage devices such as battery and ultracapacitor can be introduced into PV system as an energy buffer to balance the input power and the output power. In order to draw the best performance of such hybrid systems, it is known that DEMS is better reliability than CEMS[14]. Fig. 4 represents the proposed distributed energy management system. In this system, for designing strategies of DEMS, the rated voltage of battery and UCB is considered. So, the battery and UCB voltage will be maintained within the safe range that can be represented by their state of charge (SOC). The proposed DEMS consists of three distributed controllers: PV controller, Battery controller and UCB controller. The battery controller and UCB controller operates two modes: DC-Voltage control mode and power control mode. But, both of them can't have the same mode at a same time. Therefore, only one controller at a time must regulate the DC-link voltage. Also, in the DC-voltage control mode, the use of UCB has priority because UCB is capable of providing the short-term peak load demand. The PV system executes two strategies: Permission for charging and allocation of excess power. The PV system has to track the maximum power point (MPP) by controlling a dc/dc converter to ensure efficient operation and allocates available power to recharge the battery and UCB. If the total output power of PV system is higher than the demand power, the remaining power will be used to charge the battery

Distributed Energy Management for Stand-Alone Photovoltaic System with Storages

103

Information and request

PV Controller

Battery Controller

Information

UCB Controller

Mode

and request

change

DC-voltage Control Mode Power Control Mode

Mode change

DC-voltage Control Mode Power Control Mode

Fig. 4. The proposed distributed energy management system

and the UCB. Each controller has a right to permit or refuse a request of others. When battery or UCB requests the power for charging SOC of battery and UCB, PV system decides whether or not to accept their requests according to its own state. This behavior also is similar to the operation between Battery and UCB controller.

4 Simulation and Results The PV system developed in this paper is capable of delivering 12kW of power at the best radiation conditions and the maximum current of UC power is 750A. Also, the rated voltage and current of the battery model are 200V and 6.5A, respectively. To show as clearly as possible the response of the EMS, the simulation time was set 20s and the external power is assumed to be 10kW. Table 4 shows the power allocation by PV controller during DC-link voltage control mode of UCB. When UCB controller executes DC-link voltage control mode, it mean that UCB is working in the range of between maximum of SOC and minimum of SOC. In this paper, the favorable SOC for the UCB is set between 80% and 40%. Similarly, the suitable SOC of battery is set between 80% and 30%. In Table 4, PPV denotes the remaining power of PV system after providing the demand power. Fig. 5 shows PV output power by MPPT and the demand load power. Table 4. Power allocation during DC-link voltage control mode of UCB UCB charge power

Batter charge power

SOC = Ok

-

-

DC-link Mode

SOC = Ok

-

-

State 1_3

DC-link Mode

SOC = Not

-

PPV

State 1_4

DC-link Mode

SOC = Not

-

0.8PPV

State

UC SOC

Battery SOC

State 1_1

DC-link Mode

State 1_2

104

J.H. Cho and D.-H. Kim

Fig. 5. PV MPPT output power and load power

Fig. 6. The current and voltage of UCB

Fig. 6 and 7 show the current and voltage of UCB and battery, respectively. During low power demand period (from 0 to 2s), the UCB operates charge mode. Thus, the UCB current is negative in this period. During high power demand period (from 2 to 6s), for sustaining demand load power, both the UCB and battery operate discharge mode and the UCB and battery current is positive. As shown in Fig. 6 and 7, the UCB and battery properly operate according to the constantly varying the load power and PV output power.

Distributed Energy Management for Stand-Alone Photovoltaic System with Storages

105

Fig. 7. The current and voltage of battery

5 Conclusion This paper presents a distributed energy management system (DEMS) for stand-alone photovoltaic(PV) system with storage devices. DEMS are more effective than conventional CEMS, but it is difficult to design the DEMS for power system because of the complex control strategy according to several distributed controller. To verify the performance of the proposed DEMS, the stand-alone photovoltaic(PV) system is designed by Matlab/SimPowerSystems and Matlab/StateFlow tool is used for designing the proposed DEMS. Acknowledgments. The work developed in this paper has been supported by the DAEDEOK INNOPLIS (“R&D Hub Cluster project”).

References 1. Borowy, B.S., Salameh, Z.M.: Optimum photovoltaic array size for a hybrid wind/PV system. IEEE Transactions on Energy Conversion. 9, 482–488 (1994) 2. Markvart, T.: Sizing of hybrid photovoltaic-wind energy systems. Solar Energy 57, 277– 281 (1996) 3. Martínez, J., Medina, A.: A state space model for the dynamic operation representation of small-scale wind-photovoltaic hybrid systems. Renewable Energy 35, 1159–1168 (2010) 4. Hwang, J.J., Lai, L.K., Wu, W., Chang, W.R.: Dynamic modeling of a photovoltaic hydrogen fuel cell hybrid system. International Journal of Hydrogen Energy 34, 9531– 9542 (2009) 5. Xie, J., Zhang, X., Zhang, C., Wang, C.: Research on Bi-Directional DC-DC Converter For a Stand-Alone Photovoltaic Hybrid Energy Storage System. In: 2010 Asia-Pacific Power and Energy Engineering Conference (APPEEC), pp. 1–4 (2010) 6. Gee, A.M., Dunn, R.W.: Novel battery/supercapacitor hybrid energy storage control strategy for battery life extension in isolated wind energy conversion systems. In: 2010 45th International Universities Power Engineering Conference (UPEC), pp.1–6 (2010)

106

J.H. Cho and D.-H. Kim

7. Katiraei, F., Iravani, R., Hatziargyriou, N., Dimeas, A.: Microgrids management. IEEE Power and Energy Magazine 6, 54–65 (2008) 8. Kim, S.K., Jeon, J.H., Cho, C.H., Ahn, J.-B., Kwon, S.H.: Dynamic Modeling and Control of a Grid-Connected Hybrid Generation System With Versatile Power Transfer. IEEE Trans. on Industrial Electronics 55, 1677–1688 (2008) 9. Zue, A.O., Chandra, A.: Simulation and stability analysis of a 100 kW grid connected LCL photovoltaic inverter for industry. In: IEEE 2006 Power Engineering Society General Meeting, pp. 1–6 (2006) 10. Uzunoglu, M., Onar, O.C., Alam, M.S.: Modeling, Control and simulation of a PV/FC/UC based hybrid power generation system for stand-alone applications. Renewable Energy 34, 509–520 (2009) 11. Johansson, P.: Comparison of Simulation programs for Supercapacitor Modeling. Chalmers university of technology, master of science thesis (2008) 12. El-shark, M.Y., Rahman, A., Alam, M.S., Byrne, P.C., Sakla, A.A., Thomas, T.A.: dynamic model for a stand-alone PEM fuel cell power plant for residential applications. Journal of Power Sources 138, 199–204 (2004) 13. Tremblay, O., Dessaint, L.-A., Dekkiche, A.-I.: A generic Battery Model for the Dynamic Simulation of Hybrid Electric Vehicles. In: Vehicle Power and Propulsion Conference, pp. 284–289 (2007) 14. The application of Multi Agent System in Microgrid coordination control. In: International conference on Sustainable Power Generation and Supply, pp. 1–6 (2009)

Framework for Performance Metrics and Service Class for Providing End-to-End Services across Multiple Provider Domains Chin-Chol Kim1, Jaesung Park2,*, and Yujin Lim3 1

Digital Infrastructure Division, National Information Society Agency, 77 Mugyo-Dong, Jung-Gu, Seoul, 100-775, Korea [email protected] 2 Department of Internet Information Engineering, University of Suwon, 2-2 San, Wau-ri, Bongdam-eup, Hwaseong-si, Gyeonggi-do, 445-743, Korea 3 Department of Information Media, University of Suwon, 2-2 San, Wau-ri, Bongdam-eup, Hwaseong-si, Gyeonggi-do, 445-743, Korea {jaesungpark,yujin}@suwon.ac.kr

Abstract. Developing a unified solution that enables the end-to-end delivery of services over multiple provider domains at a guaranteed quality level is challenging. International standard organizations offer their own definition of performance metrics and service classes. In this paper, we define the unified performance metrics and service classes for interworking of various types of networks. Keywords: Quality-of-service, service-level agreement, performance metric, service class.

1 Introduction The Internet is moving from being a simple monolithic data service network to a ubiquitous multi-service network in which different stakeholders including content providers, service providers, and network providers. They require to co-operate for offering value-added services and applications to content consumers [1]. The problem of how to extend QoS (Quality-of-Service) capabilities across multiple provider domains for providing end-to-end services, has not been solved satisfactorily to-date. Furthermore, developing a unified solution that enables the end-to-end delivery of services over various types of networks at a guaranteed quality level is more challenging. The current practice in service offering is using of Service Level Agreements (SLAs). This paper presents the solution for end-to-end QoS-enabled service delivery over heterogeneous networks. To do this, we compare the performance metrics defined by international standard organizations and re-define the unified metrics. Then we analyze *

Corresponding author.

T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 107–113, 2011. © Springer-Verlag Berlin Heidelberg 2011

108

C.-C. Kim, J. Park, and Y. Lim

the service classes presented by the organizations and re-define the unified service class. Besides, we consider how to allocate performance to multiple provider domains. This paper is organized as follows. Section 2 highlights related works and our proposed performance metrics. Section 3 describes how to achieve the interworking of heterogeneous networks. Finally, conclusions are presented in Section 4.

2 Performance Metrics 2.1 Related Works Generally, performance metrics are defined to evaluate the end-to-end QoS provisioning. International standard organizations lay out several criteria for the metrics. We introduce performance metrics defined by thee organizations; IETF, ITU-T, and GSMA. First, IETF develops and promotes Internet standards and it proposes 5 metrics to maximize the service quality and reliability of end-to-end path, as below [2-7]. ‹

Packet Delay For a real number dT, “the delay from Src (source) to Dst (destination) at T is dT” means that Src sent the first bit of a packet to Dst at time T and that Dst received the last bit of that packet at time T+dT.

‹

Packet Delay Variation The variation in packet delay is sometimes called "jitter”. The packet delay variation is defined for two packets from Src to Dst as the difference between the value of the delay from Src to Dst at T2 and the value of the delay from Src to Dst at T1. T1 is the time at which Src sent the first bit of the first packet, and T2 is the time at which Src sent the first bit of the second packet.

‹

Packet Loss “The loss from Src to Dst at T is 0” means that Src sent the first bit of a packet to Dst at time T and that Dst received that packet. “The loss from Src to Dst at T is 1” means that Src sent the first bit of a packet to Dst at time T and that Dst did not receive that packet.

‹

Packet Reordering If a packet s is found to be reordered by comparison with the NextExp value, its “Packet-Reordered” = True; otherwise, “Packet-Reordered” = False if s >= NextExp then /* s is in-order */ NextExp = s + 1; Packet-Reordered = False; else /* when s < NextExp */ Packet-Reordered = True;

‹

Packet Duplication The packet duplication is a positive integer number indicating the number of (uncorrupted and identical) copies received by Dst in the interval [T, T+T0] for a packet sent by Src at time T.

Framework for Performance Metrics and Service Class for Providing End-to-End Services

109

Second, ITU-T coordinates standards for telecommunications. It provides 8 metrics for QoS monitoring across heterogeneous provider domains to provide end-to-end services as below [8]. ‹

IP Packet Transfer Delay (IPTD) IPTD is the time, (t2 – t1) between the occurrence of two corresponding IP packet reference events, ingress event at time t1 and egress event at time t2, where (t2 > t1) and (t2 – t1) ≤ Tmax.

‹

IP Packet Error Rate (IPER) IPER is the ratio of total errored IP packet outcomes to the total of successful IP packet transfer outcomes plus errored IP packet outcomes.

‹

IP Packet Loss Rate (IPLR) IPLR is the ratio of total lost IP packet outcomes to total transmitted IP packets.

‹

Spurious IP Packet Rate Spurious IP Packet Rate is the total number of spurious IP packets observed at that egress point during a specified time interval.

‹

IP Packet Reordered Ratio (IPRR) IPRR is the ratio of the total reordered packet outcomes to the total of successful IP packet transfer outcomes.

‹

IP Packet Severe Loss Block Ratio (IPSLBR) IPSLBR is the ratio of the IP packet severe loss block outcomes to total blocks.

‹

IP Packet Duplicate Ratio (IPDR) IPDR is the ratio of total duplicate IP packet outcomes to the total of successful IP packet transfer outcomes minus the duplicate IP packet outcomes.

‹

Replicated IP Packet Ratio (RIPR) RIPR is the ratio of total replicated IP packet outcomes to the total of successful IP packet transfer outcomes minus the duplicate IP packet outcomes.

Third, GSMA is an association of mobile operators and related companies devoted to supporting the standardizing, deployment and promotion of the GSM mobile telephone system. GSMA defines the metrics requested to IPX providers by wireless service providers as below [9]. ‹

Max Delay Max Delay is the maximum value of the one-way transit delays across an IP transport network.

‹

Max Jitter Max Jitter is the maximum value of delay variations across an IP transport network.

110

C.-C. Kim, J. Park, and Y. Lim

‹

Packet Loss Packet Loss is the ratio of total lost packets to total transmitted packets via an IP transport network.

‹

SDU Error Ratio SDU Error Ratio is the ratio of total errored packets to total transmitted packets via an IP transport network.

‹

Service Availability Service availability is a proportion of the time that the service is considered available to service providers on a monthly average basis.

Fig. 1. The comparison of performance metrics

2.2 Definition of Performance Metrics We summarize performance metrics mentioned in the previous subsection, as shown in Fig 1. We select the common metrics among the metrics defined by the standard organizations, such as delay, jitter, and packet loss. The delay and jitter seriously affect the quality of real-time streaming multimedia applications such as voice over IP, online games and IPTV. In some cases, excessive delay can render the application unusable. Some network transport protocols such as TCP provide for reliable delivery of packets. In the event of packet loss, the receiver asks for retransmission or the sender automatically resends any segments. Although TCP can recover from packet loss, retransmitting missing packets causes the throughput of the connection to decrease. In addition, the retransmission possibly causes severe delay in the overall transmission. Thus, when the end-to-end QoS is evaluated across multiple providers, at least three metrics should be considered. However, the definitions of the metrics are different among the organizations. We re-define the metrics consistently as below. ‹

Delay The arithmetic mean of one-way packet transit delays between source and destination.

Framework for Performance Metrics and Service Class for Providing End-to-End Services

‹

Jitter The arithmetic mean of differences between successive packet delays.

‹

Packet Loss The ratio of total lost packets to total transmitted packets.

111

3 Interworking of Multiple Providers 3.1 Definition of Service Class Service providers offer their own service class and the quality of these services are different. When the traffic enters into the provider domain and the traffic is mapped into the higher level of service class than the requested level, the network resources are wasted. Whereas, when the traffic is mapped into the lower level of service class than the requested level, the service quality is not guaranteed. Thus we define the unified service classes to solve the mapping problem between service classes of different providers.

Fig. 2. The comparison of service classes

Fig. 3. Definition of service classes

112

C.-C. Kim, J. Park, and Y. Lim

Fig. 2 summarizes the service classes proposed by IETF, ITU-T, and GSMA [1012]. IETF divides the service classes based on the services offered. ITU-T divides the classes based on the service characteristics and GSMA focuses on the services used. We re-define the service class for interworking of heterogeneous networks, as shown in Fig. 3. 3.2 Allocation of Performance Even though the performance metrics and service classes are re-defined consistently, another challenge to achieve the end-to-end QoS is presented: how can QoS classes, e.g., network performance, be assured for users? There are two basic approaches to solve the problem [13]. One involves allocating performance to a limited number of network segments, which allows operators to contribute known levels of impairments per segment, but restricts the number of operators that can participate in the path. The other approach is impairment accumulation, which allows any number of operators to participate in a path. On the surface, this may appear too relaxed, but assuming operators in a competitive environment will actively manage and improve performance. The static allocation approach divides the end-to-end path into a fixed number of segments and budgets the impairments such that the total objective is met in principle. It requires that individual segments have knowledge of the distance and traffic characteristics between the edges of their domains, as these properties of the segment affect the resulting allocations. For example, the delay budget allocated to a network segment depends on whether it is access or transit, and whether the transit distance is metro or regional. Similarly, packet loss and delay variation will have to be allocated according to whether the segment is access or transit, as the traffic aspects can differ significantly. An important aspect of the static allocation is its dependence on the number of providers, as the allocation has to be done accordingly. This can result in undershooting or overshooting the objective because paths can have a different number of network segments than designed for. Accumulation approach is defined as those that include requests of what performance level each provider can offer, followed by decisions based on the calculated estimate of end-to-end performance. The requester may be the customer-facing provider only or include all the providers along a path. The responder may be a provider or their proxy. However, there are several weaknesses of the approach. First, users' segment impairments are not taken into account. Second, if the initial process fails, multiple passes of request/estimation cycle may be required. Third, it requires customer or customer proxy involvement. Finally, commitments for each network segment must be pre-calculated taking distance into account.

4 Conclusions In this paper, we focus on how to extend QoS capabilities across multiple provider domains for providing end-to-end services. To solve the problem, we define the unified performance metrics and the service classes for interworking of multiple provider domains. For the future work, we need to solve another problem of how to allocate performance to the domains, by using IETF PDB (Per-Domain Behavior) concept.

Framework for Performance Metrics and Service Class for Providing End-to-End Services

113

References 1. Ahmed, T., Asgari, A., Mehaoua, A., Borcoci, E., Berti-Equille, L., Georgios, K.: End-toEnd Quality of Service Provisioning through an Integrated Management System for Multimedia Content Delivery. Elsevier Computer Communications 30, 638–651 (2007) 2. Paxson, V., Almes, G., Mahdavi, J., Mathis, M.: Framework for IP Performance Metrics. IETF internet-standard: RFC2330 (1998) 3. Almes, G., Kalidindi, S., Zekauskas, M.: A One-way Delay Metric for IPPM. IETF internet-standard: RFC2679 (1999) 4. Almes, G., Kalidindi, S., Zekauskas, M.: A One-way Packet Loss Metric for IPPM. IETF internet-standard: RFC2680 (1999) 5. Demichelis, C., Chimento, P.: IP Packet Delay Variation Metric for IP Performance Metrics (IPPM). IETF internet-standard: RFC3393 (2002) 6. Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, S., Perser, J.: Packet Reordering Metrics. IETF internet-standard: RFC4737 (2006) 7. Uijterwaal, H.: A One-Way Packet Duplication Metric. IETF internet-standard: RFC5560 (2009) 8. ITU-T Recommendation Y.1540. Internet Protocol Data Communication Service – IP Packet Transfer and Availability Performance Parameters (2007) 9. GSMA IR.34. Inter-Service Provider IP Backbone Guidelines (2008) 10. Babiarz, J., Chan, K., Baker, F.: Configuration Guidelines for DiffServ Service Classes. IETF internet-standard: RFC4594 (2006) 11. ITU-T Recommendation Y.1541. Network Performance Objectives for IP-based Services (2006) 12. 3GPP TS 23.107. Quality of Service (QoS) Concept and Architecture (2010) 13. ITU-T Recommendation Y.1542. Framework for achieving end-to-end IP performance objectives (2006)

Design of a Transmission Simulator Based on Hierarchical Model Sang Hyuck Han and Young Kuk Kim* Department of Computer Science & Engineering, Chungnam National University 220 Gung-dong, Yuseong-Gu, Daejeon 305-764, South Korea [email protected], [email protected]

Abstract. Recently, there are increasing efforts to use IT technologies on power domains like as power automation systems, simulation systems, etc. Especially, simulation technologies are necessities for systems to improve their qualities and verify their errors with low costs. But, it's limited to use them because most of power simulation systems are focus on simulating small substations and commercial solutions need expertise knowledge to use them and their cost is very high. In this paper, we describe a design of HBTS(Hierarchical Based Transmission Simulator) that simulates Korea's substations. HBTS outputs each substation's currents, voltages, phase angles. Keywords: Transmission simulator, industrial process control.

1

Introduction

Recently, IT technologies play a key role for risk management and quality control of power automation systems. Requests for convergence of IT and Power Industries are increasing at an alarming rate. Especially, Real-time data processing technologies and power simulation technologies are most essential ones among those technologies. Power simulation technologies are useful to improve system’s qualities and verify system’s errors with low costs and educate inexperienced operators. Also, the data that are generated from the simulation can be used for power automation systems like SCADA(Supervisory Control And Data Acquisition), RTDB(Real-Time DataBase) to verify their functionalities and enhance their performance. But, it's limited to use them because most of power simulation systems are focus on simulating small substations and commercial solutions need expertise knowledge to use them and their cost is very high[1][2]. In this paper, we describe a design of HBTS(Hierarchical Based Transmission Simulator) that similarly simulates Korea's substations. HBTS describes Korea's substations with IEEE 4 Node Test Feeder and a hierarchical tree structure. Accordingly, Using real power usage capacity that is provided in KPX(Korea Power eXchange), HBTS regularly can output each substation's currents, voltages, phase angles. HBTS can be used on various ways. First, it can both check and enhance *

Corresponding author.

T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 114–122, 2011. © Springer-Verlag Berlin Heidelberg 2011

Design of a Transmission Simulator Based on Hierarchical Model

115

performance of the RTDB that collects and manages data from a lot of substations. Second, with detail description of HBTS, you can considerably decrease your development time when you make similar simulators. This paper is organized as follows. Chapter 2 describes related works that are about Korea's transmission overview and main electric theories that are used in HBTS, and Chapter 3 describes a design of HBTS in detail, and Chapter 4 describes a conclusion and future works.

2 2.1

Related Works Korea Transmission Overview

Fig. 1 is a conceptual figure of Korea's transmission power system, and it is mainly consist of generators, ultra high voltage substations(745kV), high voltage substations (345kV), medium voltage substations(154kV), distribution substations(154kV). Each generator produces power, and changes it with high voltage power, and sends it to high voltage substations. High voltage substations distribute some of power some factories or train substations, and transmit power that is remained to medium voltage substations. Medium voltage substations distribute some of power to broadcasts, big buildings, stadiums, and transmit power that is remained to distribution substations. Distribution substations distribute some factories, buildings, and pole transformers change power with low voltage that is 22.9kV and distribute it to small factories, markets, customers[3].

Fig. 1. Conceptual diagram of transmission power system

Fig. 2 is the picture of Korea's power system[4]. It is consist of generators and substations. Korea's generators have some features. First, most of generators produce power with water, steam, nuclear. Second, most of hydro power plants are located on Han river, and steam or nuclear plants are located on forest and sea area because they import raw materials from abroad. Third, power generation size of each power plant is bigger than before because of difficulty on selecting right area. With these reasons, also, submission lines are longer and have massive capacity. Most of substations are located on client area or big industrial sector. Especially, clients area near metropolitan have difficulty with power send and receive because

116

S.H. Han and Y.K. Kim

Fig. 2. Korea Power System

preventing building of large capacity plant near metropolitan. For sending and receiving in metropolitan, transmission lines of 765kV power plant are used for decreasing power loss[4][5]. 2.2

The Main Technologies

The main technologies for development of HBTS are PU(Per Unit system)[4], KCL(Kirchhoff's Current Law)[6], IEEE 4 Node Test Feeder[7]. A per-unit system is the expression of system quantities such as voltage, current, power, reactive power, real power, impedance, admittance as fractions of a defined base unit quantity. Generally, we use PU as symbol of a per-unit system. Table 1. PU(Per-Unit System) Items Voltage

Formula V PU

(1)

P PU Power

Q PU W PU I

(2)

A

∆

Current √

I PU

∆

√

∆

√

(3) (4)

Line to line voltage V [kV] of a power system is represented as (1) when base [kV], and power-reactive power is represented each of line to line voltage is V power P[MW] and reactive power Q[MVar] and apparent power W[MVA] as (2) by PU when base capacity is W [MVA]. Basic current I A , that is upper current, is represented as (3) with basic capacity that is W , W and basic voltage. Then, (4) can be taken with basic current from (3). KCL is a rule that at any

Design of a Transmission Simulator Based on Hierarchical Model

117

node in an electrical circuit, the sum of currents flowing into that node is equal to the sum of currents flowing out of that node. In Fig. 4, with KCL, I that is feeder is same as sum of three current that is I , I , I . I

I

I

E y

E

I E y

E

E y

(5)

(5) can be transformed as (6) I

E Y

E Y

E Y

(6)

Fig. 4. An example of power circuit

Fig. 5 shows IEEE 4 Node Test Feeder that is consist of both generator G1, G2 and load L1, L2, and it has 4 bus between generators and loads[10]. 2

1 2000 ft.

Infinite Bus

[I 12]

3

4 2500 ft.

[I 34]

Load

Fig. 5. IEEE 4 Node Test Feeder

Fig. 6 describes representative values about line parameters, active power, reactive power. A virtual power system uses representative values in Fig. 5 with difficulty of obtaining real parameters.

Fig. 6. P, Q, i, V, ζ values in IEEE 4 Node Test Feeder

118

3

S.H. Han and Y.K. Kim

Hierarchical Based Transmission Simulator

Fig. 7 shows an architecture of HBTS(Hierarchical Based Transmission Simulator). HBTS has three main components that are BDG(Base Data Generator) that creates base data of transmission substations that similarly can be modeled with Korea's transmission substation's architecture and TDG(Trend Data Generator) that generates period data with trend data of KPX(Korea Power eXchange) and RDG(Result Data Generator) that output the simulation result with an adequate format.

Fig. 7. An Architecture of HBTS

3.1

BDG(Base Data Generator)

BDG is a main component that generates each substation's base data such as voltage, current, power based on transmission structure that is similar to Korea transmission power system. Fig. 8 shows a conceptual architecture of HBTS's transmission substations that has 4 levels. On 1 level, 4 nodes that are referenced from IEEE 4 Node Test Feeder connected each other, and 2-4 levels have hierarchical structure based on KCL. HBTS can be used to represent Korea's transmission power system with configuration of both the count of level and the node count of each level.

Fig. 8. A Conceptual Architecture of HBTS’s transmission substations

Table 2 pictures the relations between HBTS and Korea's power system. 1 level has 2 generators and 2 loads. On Korea's power system, 1 level is starting point with generators and 765kV substations. And we assume that generators in 1 level produce overall power that is equal to overall load, and produced power P does not loss in the

Design of a Transmission Simulator Based on Hierarchical Model

119

power providing process from high level to low levels. Level 2 has 30 loads that is sum of each child's count of 1 level that is each 15. On Korea's power system, Level 2 is consist of 345kV substations. Also, each 3 level and 4 level has 300 loads and 1500 loads. And they have hierarchical structure. On Korea's power system, each 3 level and 4 level is consist of 154kV substations and 22.9kV substations. Table 2. The relations between HBTS and Korea’s power system

Level

Structure

1

4 node feeder

2 3 4

Tree, Hierarchical

node type

count

Korea Power System

generator

2

generator

load

2

765kV

load

30

345kV

load

300

154kV

load

1,500

22.9kV

For generating base data in BDG, first step is generating base data of l level that are power, voltage, current values, and these values are obtained from power formula with convergence values of 4 Node Test Feeder. Second step is generating base data of other levels that can be taken by power current value of 1 level with KCL.

Fig. 9. Flow Diagram for obtaining Slack feeder power and line current

Fig. 9 shows the flow diagram for obtaining slack feeder power and line current. It can be summarized as follows. First, make YBus about each line. Second, get P , Q for making power formula. Third, get real power ∆P and reactive power ∆Qwith Newton-Rhapson method that is non-linear power method. Fourth, if the method isn't

120

S.H. Han and Y.K. Kim

convergent, from second to fourth step is repeated after obtaining improved phase angle and voltage using jacobian matrix until the method is convergent. Fifth, if the method is convergent, calculate line current based on slack feeder. For generating current(i), power(p), voltage(v) in each 1-4 level, the way to obtain values is different between 1 level and 2-4 level. In 1 level, we obtain the values after calculation (7) that has power P as input value. Each voltage value is assigned substation voltage value of each level. But, substation voltage value of each level is not always same, but it has voltage difference in the range of 3%. Each phase angle that is has the value 45 by default. I

A

(7)

In 2-4 level, we obtain power value with (8) that has current I that is taken by KCL with feeder current of upper level. Voltage and Cos are same as the values of 1 level. P

VIcosθ Watt

(8)

With (7) and (8), we can get the base value of each substations in all level. 3.2

TDG(Trend Data Generator)

TDG creates power trend data of each substation transforming the base data of BDG with status of real time power send and receive that provided by KPX(Korea Power eXchange)[8].

Fig. 10. Real time power usage status of KPX

In KPX, it provides the power data graph like as Fig. 10. For using this in HBTS, we need to change a graph form to digital values. Then, we obtained 1 minutes interval values with programming logic. Fig. 11 shows graph of the transformed values.

Fig. 11. Transformed power usage status

Design of a Transmission Simulator Based on Hierarchical Model

121

TDG can generate the trend data of each substation with both the base data from BDG and transformed power usage from KPX data. 3.3

RDG(Result Data Generator)

RDG(Result Data Generator) is the component that outputs the result of each substation's values timely after execution of BDG and TDG. A result format has 4 lists of attribute as Table 3. Each attribute is time T, voltage v, current i, power p. Table 3. Attributes of RDG Result

attribute time voltage current power

4

unit T v i p

description 1 minute time interval v of current node i of current node p of current node

Conclusion and Future Work

In this paper, we described about the design of HBTS. It is modeled as the hierarchical architecture that is similar to Korea's transmission power. Also, we described the result format. With HBTS’s simulation data, we hope to enhance the performance and functionality of main components that are RTDB, HMI, data compression etc. also, we can easily change HBTS for representing Korea's power system because the main architecture of HBTS is similar to Korea's one. In a near future, we will implement HBTS and generate transmission simulation data.

References 1. Lee, K.H., Kil, I.J., Choi, J.Y., Lee, S.K.: Design of Real-Time Power System Simulator for Education using LabVIEW. Journal of the korean Institute of Illuminating and Electrical Installation Engineers 24(6), 177–182 (2010) 2. Baik, S.D., Kim, S.K., Lee, J.H., Lee, S.C.: A study on the development of substation power system simulator for education and training. In: The Korean Institute of Electrical Engineers summer Annual Conference, pp. 67–69 (2004) 3. KEPCO: ELECTRICITY & ENERGY (2010), http://www.kepco.co.kr/museum/e_energy/energy_1_3.html 4. KPX: figure of power system in 2009 (2009), http://www.kpx.or.kr/KOREAN/servlet/action?index=233 5. Song, K.Y.: Power System Engineering. Dongilbook (2009) 6. Song, K.Y.: The newest distribution engineering. Dongilbook (2010) 7. IEEE: IEEE 4 Node Test Feeder. IEEE Power Engineering Society Power System Analysis, Computing and Economics Committee (2006) 8. KPX: Real-time Snapshot of power Supply and Demand (2011), http://www.kpx.or.kr/KOREAN/htdocs/main/sub/ems_info.jsp

122

S.H. Han and Y.K. Kim

9. KERI: Development of Power System Simulator. Final Report of a secondary year in KERI (1996) 10. Bergen, A.R., Vittal, V.: Power System Analysis. Prentice Hall, Englewood Cliffs (1986) 11. Saadat, H.: Power System Analysis. McGraw-Hill, New York (2007) 12. Ducan, J., Sarma, S., Overbye, J.: Power System Analysis And Design. Thomson (2008) 13. Im, J.H., No, C.K., Yeum, J.E., Jun, J.: Load Flow for Smart grid. A graduation thesis of collage graduate in ChungNam national University (2011)

Recommendation System of IPTV TV Program Using Ontology and K-means Clustering Jongwoo Kim1, Eungju Kwon1, Yongsuk Cho2, and Sanggil Kang1 1

Department of Computer Science and Information Engineering, Inha University, 253,Yonghyun-dong, Nam-gu, Incheon, 402-751, Korea {bestkjw,ora1126}@inha.edu, [email protected] 2

Department of Electronic Engineering, Konyang University, Nea-dong, Nonsan-si, Chungcheongnam-do, Korea [email protected]

Abstract. In this paper we introduce a recommendation system for recommending preferred TV genres for each viewer, using ontology technique and K-means clustering algorithm. The algorithm is developed based on the personal VOD viewing history. First, the viewing history is built in an ontology which is able to achieve inference process through a query. In the list of users, each item and class obtain the probability of preference of VODs and then the information is used for building the ontology. From the ontology we select each user’s preferred VODs using K-means algorithm. In the experimental section, we show the feasibility of our algorithm using real TV viewing data. Keywords: clustering, K-means, ontology, recommendation of genres.

1

Introduction

IPTV service is a kind of VOD services by selecting an appropriate TV program from the menu. Once a user turns on an IPTV, he/she has to click the menu at least 5-6 times for searching his/her preferred programs. To help a user to select programs, the information of user’s preferred TV programs is needed to reduce searching time and effort. Therefore, the development of personal VOD recommendation has been one of hot issues presently. To develop recommendation systems, mining techniques and ontologies have been commonly applied in recent years. For example, Benjamin Adrian et al. [2] proposed ConTag document, which extracts topics of documents using WordNet [9] and then recommend the tags related to the concepts of topics. Mining methods have the problem that it is bias on the personality tendencies. To solve the problem, the K-means algorithms have implemented leader clustering and the IAFC (Integrated Adaptive Fuzzy Clustering) artificial neural network algorithms [5]. Leader clustering without the initial number determines the number of clusters based on the similarity of data. In this paper we develop a recommendation system based on the personal VOD viewing history. The viewing history is built in an ontology which is able to achieve inference process through a query. In the list of users, each item and class obtain the T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 123–128, 2011. © Springer-Verlag Berlin Heidelberg 2011

124

J. Kim et al.

probability of preference of VODs and then the information is used for building the ontology. From the ontology we select each user’s preferred VODs using K-means algorithm. The remainder sections of this paper are composed as follow: In Section 2, we introduce the related works of our proposed system. In Section 3 we describe our proposed recommendation algorithm. In Section 4, we show the implementation of our system from running a simple test. Finally, we conclude in Section 5.

2

System Architecture of Recommendation

Fig. 1 shows the overall architecture of our proposed recommendation system in IPTV content provider side. The system consists of four modules such as Data Collection, Knowledge Management, Preference Computing, and Reasoning. Data Collection module collects viewer’s content consumption information from Set-top Box connected to IPTV at viewer side. Based on the information, Knowledge Management module builds ontology-based semantic relations among IPTV contents using description keywords (DKs) extracted by WordNet [9]. Preference Computing module computes the preference of contents from ontology built in Knowledge Management. Reasoning module recommends each user’s preferred TV programs using K-means clustering. iTV Content Provider Viewer Matching Module

Content Matching Module

Recommendation Access Network

Knowledge Management Module

Data Collection Module

Set-top Box User Profile Usage History

User Data

AD Advertisement Content Provider

Fig. 1. Overall architecture of PTA system in iTV

3

Our Recommendation Algorithm

In this section, we show the way to compute the similarity among TV programs by measuring the semantic distance from their program ontology as seen in Eq. (1). sim

i. p C , j. p C

∑ EWS i. p C synset j. p C EWS i. p C

(1)

where i.p(C) and j.p(C) are the context C of the description property p in program i, j. Also, EWS (i.p(c)) is total number of context C union of description key words between

Recommendation System of IP PTV TV Program Using Ontology and K-means Clustering

125

hyperset and hyposet of context c C and EWS Onto . p C synset Onto . p C is the total of similar for con ntext C of program i and j. Eq. (2) is the similarity forr all context C. sim i, j

∑ sim i. p C , j. p C a property number of i

(2)

The case not to be overlook ked from Eq. (2) is that there are more than two prograams have same similarity meassure. In that case, we need to give weight accordingg to relative viewing times of eaach viewer to for each program as seen Eq. (3). Item

number of viewing

(3)

By clustering the variety of viewing tendency, we can recommend each vieweer’s favorite genre and its programs. The center point to coordinate the viewer's prefereence list determined from the previous p equations. If some viewers prefer two kindss of programs, than two clustering is more likely to be generated. To avoid the problem, we use K-means algorithm m to analyze and to generate clustering as an effecttive technique. The number of clustering g based on viewers is different, so it is not easily to preddict. Therefore, the number of clusters c needs to be dynamically adjusted. Also, when the size of two clusters is largee as seen in Fig. 2, the center of two clusters will appeaar as the viewer's preference. Theerefore, the programs in the center are recommended.

Fig. 2. Example of clustering

Fig. 3 shows the clusterring process. The number of clusters start with one at the initial state. As seen in Fig.. 3-(a), there is one cluster. In this case, the diameter off the cluster is greater than the prredetermined maximum diameter of cluster, Rmax. Thus,, we divide the cluster into two o clusters (C1 and C2) as seen in Fig. 3-(b). If C2 is still greater than Rmax, then we divide d the cluster into two clusters as see in Fig. 3-(c). T This process continues until the diameters of all clusters less than Rmax. The value of Rmax is determined by the system m administrator. The smaller number tends to increase the number of clusters. Once the clustering proccess is finished, we can recommend viewer’s preferred TV programs based on the disttance of each program to the centers of clusters. The thhing

126

J. Kim et al.

Fig. 3. K-means clustering process

not to be overlooked is thee case more than one cluster exists as the closet clusters to the viewer, as seen in Fig g. 4. In this case, we need to determine the priority for recommending the program ms. For example, in Fig. 4, C2 has more viewer membbers than C1. As seen in the figure, f we have to recommend two candidate prograams because the target viewer is located in the same distance from the centers of the clusters. The candidate program p including C1 will have lower value than tthat including in C2 because thee number of members included in C1 is smaller than C2. To take into the consideration n, we provide weight to each cluster in order to givve a priority for recommending programs. p

Fig. 4. Tw wo clusters as the closet clusters to the viewer

Recommendation System of IPTV TV Program Using Ontology and K-means Clustering

4

127

Experiments

The proposed method is applied to predict the preference of TV genres for each viewer using 200 viewers’ TV watching history collected from December 1, 2008 to May 31, 2009. The viewers are clustered into 22 groups obtained from our algorithm explained at the previous section. Based on the groups, we tested our recommendation performance using five test viewers, i.e., User1, User2, User3, User4, and User5. For convenience, we divided TV programs into five genres such as Sports, Drama, Entertainment, Education, and Movie. Table 1 is the result of the recommendation ratio and its actual viewing ratio of genres for each test viewer. As seen in the table, User1 has watched 6 times of the genre “Sports” out of 8 times recommendation for the genre. So, it results in 75% recommendation accuracy. For the genre “Drama”, “Entertainment”, “Education”, and “Movie”, the recommendation accuracies are 93%, 90%, 33%, and 100%. The average recommendation accuracy of User1 is 80%. In the same process, the average recommendation accuracies of User2, User3, User4, and User5 are 80%, 73%, 80%, and 94% respectively. Table 1. Recommendation accuracy for test viewers Test viewer

Sports

Drama

Entertainment

Education

Movie

User 1

recommended genres viewed genres

8 6

15 14

8 7

3 1

3 3

User 2

recommended genres viewed genres

0 0

16 18

9 7

0 0

12 8

User 3

recommended genres viewed genres

1 0

13 11

7 5

4 3

1 0

User 4

recommended genres viewed genres

0 0

11 10

3 2

0 0

1 0

recommended genres viewed genres

10 5

11 14

8 5

0 0

7 10

User 5

5

Section

Conclusion

In this paper, we proposed a new IPTV program recommendation system using ontology and K-means algorithm. As seen in the experimental section, we showed the feasibility of recommendation system using real TV viewing history. Despite of the feasibility of our method, there are a couple of further works for the completion of work. As seen in the experimental result, our recommendation is explored using only semantic relations of TV programs and viewers’ TV viewing history. However, if we use viewers’ profile information such as their occupation, age, and gender, then the performance of our recommendation will be improved. Also, we need to develop a prototype in which our algorithm is imbedded, in order to test under real practical situation.

128

J. Kim et al.

References 1. Ansari, A., Skander, E., Rajeev, K.: Internet Revcommendation System. Journal of Marketin Research 37 (August 2000) 2. Chumki, B., Haym, H., William, W., Craig, N.: Recommending Papers by Mining the Web 3. Greg, L., Brent, S., Jeremy, Y.: Amazon.com Recommendations: Published by the IEEE Computer Society Industry Report (January 2003) 4. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. Wileyinterscience, New York (2001) 5. Kim, Y.S., Mitra, S.: Inegrated Adaptive Fuzzy clustering (IAFC) Algorithm. In: Proc. of the Second IEEE International Conference on Fuzzy Systems, San Francisco, vol. 2, pp. 1264–1268 (1993) 6. Kim, J.W., Yoon, T.B., Kim, D.M., Lee, J.H.: A Personalized Music Recommendation System with a Time-weighted Clustering, vol. 19(4), pp. 504–510 (2009) 7. Benjamin., A., Leo., S., Thomas, R.: ConTag: A Semantic Tag Recommendation System. In: I-SEMANTICS 2007 Papers, pp. 297–304 (2007) 8. Martín, L.N., Yolanda, B.F., José, J.P.A., Jorge, G.D., Manuel, R.C., Alberto, G.S., Rebeca, P.D.R., Ana, F.V.: Receiver-side semantic reasoning for digital TV personalization in the absence of return channels. Multimedia Tools and Applications 41(3), 407–436 (2009) 9. WorNet, http://wornetprincetion.edu

A Novel Interactive Virtual Training System Yoon Sang Kim1 and Hak-Man Kim2 1

School of Computer Engineering, Korea Univ. of Technology and Education, Korea [email protected] 2 Dept. of Electrical Engineering, Univ. of Incheon, Korea [email protected]

Abstract. This paper proposes a novel interactive virtual training system that provides PLC trainees with a virtual environment that is identical to actually handling various types of equipment trainees could not access during training programs. The proposed system is applied to an actual technical training program, and the results are analyzed to examine its propriety and applicability. Keywords: virtual training, interactive virtual training, PLC, virtual PLC, virtual conveyor, virtual ladder.

1 Introduction Virtual reality (VR) uses computer to simulate a specific environment and surrounding circumstances to deliver information regarding the environment through five human senses (sight, smell, hearing, taste, and touch), allowing people in a virtual space to engage in experience identical to that of the real world. In addition to simulating the real world, VR allows people to simulate experiences that are not feasible in the actual world. Today, VR technology is used in various fields, including military, medical, construction, design, experience, training, and entertainment [1]. Korea has some of world's highest levels of technology in industrial areas such as electric power, automobile, semiconductors, display, mobile handsets, steel, energy, and shipbuilding. The most important group of technical experts in cutting-edge manufacturing and facility industries are PLC automation professionals. Due to the nature of manufacturing and facility industries, PLC automation training requires very expensive latest equipment. However, since inadequate manipulation or programming errors by trainees can damage the equipment and there are difficulties involved in continuously replacing and providing the latest equipment, most expensive equipment are not used at all or provided only with restricted and limited use during training sessions. Accordingly, most trainees listen to explanations rather than hands-on manipulation, which places limitations compared to actually being able to operate the equipment. In order to overcome the constraints of the training environment due to the nature of the latest automation facilities and address the importance of training, this paper proposes an interactive virtual training system based on 3D stereoscopic imaging, allowing expensive equipment to be replaced with virtual equipment. There has been hardly any research and development on the training system for PLC automation T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 129–138, 2011. © Springer-Verlag Berlin Heidelberg 2011

130

Y.S. Kim and H.-M. Kim

equipment using VR. In order to allow PLC trainees to handle virtual equipment that models actual high-price equipment such as elevators and conveyors, providing the effects of engaging in hands-on training, the virtual training system developed by this study is deployed in an actual training program and the results are analyzed to verify the feasibility of the proposed system.

2 Proposed Interactive Virtual Training System 2.1 System Overview The proposed system consists of the object-based sensor input unit that processes sensor input, the virtual PLC composer unit that receives user's arbitrary wiring input, the virtual ladder tool unit for writing PLC ladder programs, the virtual PLC unit that replaces the actual PLC, and the object-based virtual input/output model unit that provides 3D visualization of the results that correspond to arbitrary input data. The overall block diagram of the proposed system is shown in Figure 1.

Fig. 1. Block diagram of the proposed system

2.2 Virtual PLC Composer This unit receives the initial input from the user (trainee) and supports arbitrary wiring. Whereas most input wiring units accept only pre-assigned (correct) wiring, the virtual PLC composer unit in the proposed system accepts any wiring configuration (even incorrect wiring) from the user. The trainee can study materials related to a specific session and practice wiring using the virtual PLC composer shown in Figure 2. In order to cover various user's arbitrary wiring input, SAI(Scene Access Interface) [2,3] is used for the implementation.

A Novel Interactive Virtual Training System

131

Fig. 2. The virtual PLC composer

- Screen shot of practice wiring using the virtual PLC composer 2.3 Object-Based Virtual Input/Output Model Unit That Provides 3D Visualization

Fig. 3. The object-based virtual input/output model unit

- Screen shot of an object-based 3D input/output model and simulation of a motor sequence control This unit converts the user wiring input configuration from the virtual PLC composer into corresponding 3D visualization and allows the user to verify the result through simulation. Accurate wiring simulates correct action, whereas incorrect

132

Y.S. Kim and H.-M. Kim

wiring configuration produces a simulation result of faulty action (Figure 3). All the virtual input/output models are implemented based on X3D, the Web3D which is international standard graphic format. [4-7] 2.4 Virtual Ladder Tool We need a virtual ladder tool that provides a usage environment similar to the actual environment while satisfying the dedicated ladder tool for virtual PLC. Based on this requirement, we developed a ladder tool for programming ladders that can be loaded into virtual PLC. Ladder diagram (LD) into instruction list (IL) can be transformed based on [8]. Specifications of the functions supported by the ladder tool are based on the international standard IEC61131-3. Figure 4 displays the generic virtual ladder tool implemented by this study.

(a)

(b)

Fig. 4. (a) Screen shot of the generic virtual tool implemented by this study (b) Screen shot of virtual ladder tool simulating GMWIN, the actual GLOFA ladder tool

3 Application in Practice and Result Analysis 3.1 Application in Training Program - Experiment After deploying the proposed system in a technical training program (GLOFA-PLC control) conducted at our university's training center, applicability and feasibility of the system were evaluated by based on the results of the survey conducted among the trainees and instructors of the program. The training session involved controlling a virtual conveyor implemented in a virtual environment to simulate a high-level conveyor, a large-scale and expensive machine that is difficult for trainees to operate for training purposes. The practice session was conducted in a sequence identical to practicing on an actual machine, as shown in Figure 5 and Figure 6. 14 trainees participated in the experiment, including professional training school teachers, industrial high school teachers, and large companies' in-house instructors. After being introduced to the proposed virtual training system, the participants were

A Novel Interactive Virtual Training System

133

Fig. 5. Sequence of the conveyor control practice

Fig. 6. Screen shot of virtual conveyor control training session

trained in the conveyor control practice session. Upon completion of the session, they were asked to respond to a survey consisting of 10 questions, as shown in Figure 7, and display the level of satisfaction according to the Likert scale: highly satisfied (5 points), satisfied (4 points), average (3 points), unsatisfied (2 points), highly unsatisfied (1 point). 3.2 Analysis of the Results from the Training Program Table 1 shows the statistics regarding the survey response from the trainees after the practice session using the proposed system. The overall average was close to "satisfied (4 points)", confirming applicability of the training program based on the proposed system. If we can improve on some aspects, such as user-friendliness and interface, and expand functionality, the system should be fully applicable in virtual PLC training programs.

134

Y.S. Kim and H.-M. Kim

Fig. 7. Questionnaire used for evaluation on the virtual conveyor session using the proposed system

A Novel Interactive Virtual Training System

135

Table 1. Statistics regarding the questionnaire responses

Motivation N

Effective Missing Average

14 0 4.0 Adequacy

N

Effective Missing Average

14 0 4.0

N

Effective Missing Average

Training Effectiveness 1 14 0 4.0

Training Effectiveness 2 14 0 4.0

Reality

Distinction 14 0 4.0

Effectiveness as replacement 14 0 4.0

14 0 4.0

Training Effectiveness 2 14 0 4.0 User-friendliness 14 0 4.0

Applicability 14 0 4.0

Following figures display all the results of the questionnaire survey.

(a)

(b)

Fig. 8. (a) Question 1 - learning motivation (b) Question 2 - training effectiveness I

136

Y.S. Kim and H.-M. Kim

(a)

(b)

Fig. 9. (a) Question 3 - training effectiveness II (b) Question 4 - training effectiveness III

(a)

(b)

Fig. 10. (a) Question 5 - Adequacy of training (b) Question 6 - Reality of training

A Novel Interactive Virtual Training System

(a)

137

(b)

Fig. 11. (a) Question 7 - Distinction from other training (b) Question 8 - User-friendliness

(a)

(b)

Fig. 12. (a) Question 9 - Effectiveness in replacing actual equipment (b) Question 10 - Applicability in technical training

138

Y.S. Kim and H.-M. Kim

4 Conclusions This paper proposed a novel interactive virtual training system. The proposed interactive virtual training system was developed to allow PLC trainees to operate virtual equipment modeled after actual high-price machinery such as elevators and conveyors, providing experience identical to operating actual equipment. Using the virtual equipments, the proposed system could offer a training environment that allows free and unlimited repetitions in virtual space without concerns of safety and equipment damage from malfunctioning and program errors. The proposed system was applied in an actual training environment for students and teachers, and their survey responses were analyzed to confirm that the system is applicable in training programs.

References 1. Burdea, G., et al.: Virtual Reality Technology. IEEE Computer Society Press, Los Alamitos (2003) 2. Apache Ant Project, http://ant.apache.org/ 3. SAI Tutorial, http://www.xj3d.org/tutorials/general_sai.html 4. Brutzman, D., Daly, L.: X3D: Extensible 3D Graphics For Web Authors. Morgan Kaufmann Publishers, San Francisco (2007) 5. Web3D Consortium, http://www.web3d.org/ 6. Xj3D, http://www.xj3d.org/ 7. Xj3D 2.0 Code Library, http://www.xj3d.org/javadoc2/index.html 8. Fen, G., Ning, W.: A Transformation Algorithm of Ladder Diagram into Instruction List Based on AOV Digraph and Binary Tree. Journal of Nanjing University of Aeronautics & Astronautics 38(6), 754–758 (2006)

Recommendation Algorithm of the App Store by Using Semantic Relations between Apps Yujin Lim1, Hak-Man Kim2, Sanggil Kang3, and Tai-hoon Kim4 1 Department of Information Media, University of Suwon, 2-2 San, Wau-ri, Bongdam-eup, Hwaseong-si, Gyeonggi-do 445-743, Korea [email protected] 2 Department of Electrical Engineering, University of Incheon, 12-1 Songdo-dong, Yeonsu-gu, Incheon 406-772, Korea [email protected] 3 Department of Computer Science and Information Engineering, Inha University, 253 Yonghyun-dong, Nam-gu, Incheon 402-751, Korea [email protected] 4 Department of Multimedia Engineering, Hannam University, 133 Ojeong-dong, Daedeok-gu, Daejeon, 306-791, Korea [email protected]

Abstract. In this paper, we propose a recommendation algorithm for recommending mobile application software (app) to mobile user using semantic relations of apps consumed by users. To do that, we define semantic relations between apps consumed by a specific member and his/her social members using Ontology. Based on the relations, we identify the most similar social members from the reasoning process. In the experimental section, we show feasibility of our algorithm using a specific scenario. Keywords: app, recommendation, mobile, semantic relation, social members.

1 Introduction A report commissioned by mobile application store GetJar in 2010 [1] said that the mobile app (application software) market will reach $17.5 billion by 2012, having grown to 50 billion downloads from just 7 billion in 2009. The mobile app market definitely has tremendous room to grow, which takes the mobile app paradigm and slaps it onto a bigger, tablet device. Mobile app recommendation services have sprung out of a growing need to filter, rank and recommend the best apps from the hundreds of thousands now available for download onto mobile phones or tablet PCs. With iTunes now carrying 225,000 apps and Android up to 100,000, it is no wonder users have turned to other resources beyond the search box and category listings found in the official vendor-specific app stores. For the end users, recommendation services like these prove useful, even necessary at times. Personalization is defined as the ability to provide content and services tailored to individuals based on knowledge about their preferences and behavior [2]. The main T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 139–144, 2011. © Springer-Verlag Berlin Heidelberg 2011

140

Y. Lim et al.

goal of personalization is to help users find the information they are interested in, what can significantly enhance their mobile experience. Most of personalization systems try to filter available content by user's preferences and recommend only content found potentially interesting for that particular user. Personalized apps recommendation services analyze app purchases made by users, and then recommends similar apps that users may find useful for their mobile devices. The same approaches are used for recommended music and video purchases. Each service offers its own feature set and ranking algorithm. Some, for example, not only take into account an app's popularity, but also its media coverage when ranking the apps. Others use a combination of signals including apps you own and members' reviews in their algorithms. AppStoreHQ [3] is a site featuring apps for Android, iPhone, and iPad and even Web-based HTML apps. Here, a user can find what apps are hottest on the Web, what are hottest on Twitter and it offers app reviews. Appolicious [4] ranks and recommends iPhone, iPad, Android and Yahoo applications using a number of mechanisms, including reviews, likes, and friend recommendations. Users can follow their friends on the site and train the recommendation engine by sharing what apps they already have installed. Then, when signed in, the site can recommend new apps to try based on your preferences, what apps you own and other signals. Smokin Apps [5] features the top mobile apps for iPhone, Android, Blackberry, Nokia, Palm and Windows Mobile. Its recommendation engine matches a user with apps he would like based on apps similar to those his own. It then combines that information with other members' recommendations using an algorithm that tracks every way user’s rate apps on the site. Apple adds Genius recommendation tab to iPad App Store [6]. A mobile version of Apple's Genius recommendation feature, which suggests applications to users based on their account activity. The most difficult aspect of recommendation service is to understand user's preferences and to use them in an intelligent way for app filtering. In order to solve the problem, we define semantic relations between apps consumed by a specific member and his/her social members using Ontology [7]. Based on the relations, we identify the most similar social members from the reasoning process. The reasoning is explored from measuring the common attributes between apps consumed by the target member and his/her social members. The more attributes shared by them, the more similar is their preference for consuming apps. Once the similar members are identified, then the consumed apps by the members are recommended to the target member. The remainder of this paper is organized as follows. Section 2 introduces related works of our work. Section 3 explains our proposed algorithm. Following this, we demonstrate our algorithm using a scenario in Section 4. Finally, we conclude in Section 5.

2 Related Work According to [8] and [9], personalization techniques are classified in four classes: content based filtering, collaborative filtering, model based techniques, and hybrid

Recommendation Algorithm of the App Store by using Semantic Relations between Apps

141

techniques. Content based filtering uses an individual approach which relies on user's ratings and item descriptions. Items having similar properties as items positively rated by user are being recommended to the user. The most common problem of content based filtering is the new user problem. This problem occurs when a new user is added to the system, hence has an empty profile and cannot receive recommendations. Collaborative filtering recommends a target user the preferred content of the group whose content consumption mind is similar to that of the user. Problems in collaborative filtering occur when new content item is added to the system, because the item cannot take place in personalization without being rated before. It has been attractive for predicting various preference problems such as net-news, e-commerce, digital TV, digital libraries, etc. Model based techniques are usually implemented by using a predetermined model. They represent an improvement in scalability issues, because part of data is pre-processed and stored as model, which is used in the personalization process. Hybrid personalization techniques combine two or more personalization techniques to improve the personalization process. In most cases, content based filtering is combined with collaborative filtering. Traditional personalization techniques can provide very suitable solution for tailoring apps according to user's preferences. On the other hand, traditional personalization has limitations in accuracy of modeling user's behavior. In this paper, we use the semantic relations among apps using Ontology concept. From the following section, we demonstrate our reasoning algorithm in detail.

3 Reasoning Algorithm Using Semantic Relations In this subsection, we describe the reasoning algorithm to provide personalized app to each social member in a collaborative manner. The approach is explored under the assumption that the interests in app of each member tend to agree with those of the social members. We first define the semantic relations between the apps a specific (or target) member and his/her members used and attributes of each app consumed by each member through social network. Fig. 1 is a part of the semantic relationship established from interrelating the specific apps of the target member and the apps of his/her social members. The semantic relations are defined by extracting the common instances for each attribute of the apps using WordNet [22]. In the figure, the target member’s apps and the app of social member are represented on the left side and on the right side respectively. Also, the yellow ellipse, blue ellipse, and gray square denote sub-class, attribute class, and attribute respectively. Due to the diversity of general social concepts, our ontology is built with a multiple hierarchy of classes that represents domain concepts such as Food and Social network. Each class contains sub-classes such as Recipe. Also, the classes and instances of the apps are linked through their semantic attributes according to their semantic similarity. Based on the links, we can infer the semantic similarity between the target’s apps and the apps of social members. If the classes or instances in the target’s apps and the apps of the social members are connected to each other with a link, then we consider the apps to be semantically similar. The more common links

142

Y. Lim et al.

Fig. 1. A part of the semantic relationships established by interrelating the shopping product the stereotyped and the ontology of commercial advertisements

there are between them, the more similar they are in terms of the semantic relation. Consequently, the more relations we can infer between them (and the more relations from that set are established through common attributes), such as Life style of the instances in Fig. 1, the greater their semantic similarity is. Also, the same semantic metric is established through sibling attributes if they have attributes belonging to the same class, such as Recipe of classes. For example, let’s assume four app are consumed, such as Cookcook TV, Touchring Mobile, Weather Desk, and Word Break. The app Cookcook TV has the instances such as Life style, Food category, individual, Recipe, and Korean. The instance Life style is shared with Food&Cafe in the app of member. Also, Recipe is connected to Food&Café and Kimchi, because Recipe is included in the sub-class of class Food, which is connected to the instance Food in Kimchi. The instance Food categories are connected the instance Search and the instance Search is connected the instance Company of Smart 114. Therefore, Cookcook TV and Smart 114 have the equal function relationship. Once the semantic relations among apps consumed by social members as above are defined, then for each target member, we can identify the social members who have similarity preference in consuming apps from the measurement of the number of links in the semantic relations. The more links are shared between the target member and his/her social member, the more similar they are for consuming apps.

4 Experiment In this section, we describe the reasoning process of our recommendation system using a specific scenario. In order to demonstrate our reasoning process, we first collected 24 apps from a social network in which there five members as seen in Table 1.

Recommendation Algorithm of the App Store by using Semantic Relations between Apps

143

Table 1. Attributes of apps consumed by members in the social network Member A

B

C

Apps Cookcook TV, Touchring Mobile, Weather Desk , Word Break Chinese-character Study, Driver´s License Pretest, Sign Language Dictionary, YBM English Dictionary, TOEIC Speaking Smart Dial, Kimchi, Clip English, Kakaotalk, Film Lab, sGeoNotes

D

Food&Café, Konan VoiceStar, Kimchi, Smart114

E

Weather Star, mFAX, One Lock, My Photo Album, iSharing

Attributes life style, food categories, cooking method, Individual, SMS, call, weather, real-time, temperature, song, rainfall, wordbook, video, education, Korean, Japanese, English Chinese character, a qualifying examination, study, vocabulary, writing, driver’s license, pretest, an item pool, sign language, life style, TOEIC, English, Korean phone, study, slow food, cooking method, photo, social network, memo, schedules, English life style, good food restaurant, navigation service, coupon service, food, slow food, side dish, cooking method, vocal mimicry, entertainment, apply, company, productivity, Korean, English weather, a weather forecast, smart-card, security, economy, fax, real-time, GPS tracking, location, call, photo, social network, English

From the table, there are 3 common attributes between the member A and the member B, 2 common attributes with the member C, 7 common attribute with the member D, and 2 common attribute with E. From the result, we consider the member D is the most similar to the member A because the number of common attributes shared between member A and the member D is the largest compared to the other members. Therefore, we recommend the apps consumed by the member D to the member A as shown in Fig. 2. Table 2 shows the list of the identified similar member for consuming apps.

Fig. 2. The recommended apps thru the member A’s mobile

144

Y. Lim et al. Table 2. Attributes of apps consumed by members in the social network Member A B C D E

Identified similar member D C E A C

5 Conclusion In this paper, we proposed the recommendation algorithm using semantic relations between apps consumed by social members. For showing the feasibility of our algorithm, we developed a prototype of a personalized apps recommendation system using OWL by defining ontology-based semantic relations among mobile apps consumed by social members. As a further work, we need to do further experimental analyses using more data. This is necessary because the reasoning performances depend on the richness of the data. If the population in a social network is large, then the identification performance will be improved.

References 1. ReadWriteWeb, http://www.readwriteweb.com/archives/ mobile_app_marketplace_175_billion_by_2012.php 2. Liang, T.-P., Yang, Y.-F., Chen, D.-N., Ku, Y.-C.: A Semantic Expansion Approach to Personalized Knowledge Recommendation. ACM Decision Support Systems 45, 401–412 (2008) 3. AppStore HQ, http://www.appstorehq.com/ 4. Appolicious, http://www.appolicious.com/ 5. Smokin Apps, http://smokinapps.com/ 6. Genius Recommrendations, http://www.appleinsider.com/articles/10/08/ 06/apple_adds_genius_recommendation_tab_to_ipad_app_store.html 7. Maedche, A., Staab, S.: Learning ontologies for the semantic web. In: Semantic Web Worskhop (2001) 8. Blanco-Fernández, Y., Pazos-Arias, J.J., Gil-Solla, A., Ramos-Cabrer, M., López-Nores, M., García-Duque, J., Fernández-Vilas, A., Díaz-Redondo, R.P.: Exploiting Synergies between Semantic Reasoning and Personalization Strategies in Intelligent Recommender Systems: A Case Study. Elsevier The Journal of Systems and Software 81, 2371–2385 (2008) 9. Vuljanic, D., Rovan, L., Baranovic, M.: Semantically Enhanced Web Personalization Approaches and Techniques. In: Proc. of Intl. Conf. on Information Technology Interfaces, pp. 217–222. IEEE Press, Croatia (2010)

A Comparative Study of Bankruptcy Rules for LoadShedding Scheme in Agent-Based Microgrid Operation Hak-Man Kim1 and Tetsuo Kinoshita2 1

Dept. of Electrical Engineering, Univ. of Incheon, Korea [email protected] 2 Graduated School of Information Science, Tohoku Univ., Japan [email protected]

Abstract. A microgrid is a small-scale power system composed of distributed generation systems (DGs), such as solar power, wind power, and fuel cells, distributed storage systems (DSs), and loads. It is expected that many microgrids will be introduced in power grids in the near future as a clean energy grid. In microgrid operation, to meet the rating frequency, 50 or 60 Hz, is an important requirement of microgrid operation. Especially, in the case of supply shortage in the islanded operation mode, load shedding, which is an intentional load reduction, is used in order to maintain the rating frequency. Recently, a loadshedding scheme using the bankruptcy rule has been proposed as a reasonable method. In this paper, five well-known bankruptcy rules are compared and discussed for load shedding of islanded microgrid operation. Keywords: microgrid, microgrid operation, multiagent-based microgrid operation, load-shedding scheme, bankruptcy problem.

1 Introduction A microgrid, which was proposed by Prof. Lasseter in 2001 [1], is a small-scale power system composed of distributed generation systems (DGs), such as solar power, wind power, and fuel cells, distributed storage systems (DSs), and loads as shown in Fig. 1 [2]. The microgrid provides electricity and/or heat to customers such as residential buildings, commercial buildings, public offices and industrial compounds, as shown in Fig. 1, where CHP means combined heat and power and PCC is an abbreviation for the point of common coupling. To meet the rating frequency, 50 or 60 Hz, is an important requirement of microgrid operation. In grid-connected microgrid, a microgrid can solve an imbalance by power trades with the upstream power grid. However, a microgrid should solve an imbalance without trading power with any power grid in islanded operation. Especially, load shedding, which is an intentional load reduction, is used generally in the case of supply shortage. Recently, a load-shedding scheme using the bankruptcy rule has been proposed by Kim et al [3-5] as a reasonable method for load shedding. In those works, well-known bankruptcy rules such as the constrained equal awards (CEA) rule, the constrained T.-h. Kim et al. (Eds.): UCMA 2011, Part II, CCIS 151, pp. 145–152, 2011. © Springer-Verlag Berlin Heidelberg 2011

146

H.-M. Kim and T. Kiinoshita

Fig. 1. Typical configuration of a microgrid [2]

equal losses (CEL) rule, thee Talmud rule, and the random arrival (RA) rule have bbeen applied. In this paper, th he bankruptcy rules including the proportional rule are compared and discussed forr load shedding.

2 Load Shedding in Islanded I Microgrid Operation An important requirement of microgrid operation is to meet the rating frequency,, 50 or 60 Hz, which is closely related to a balance between supply and demand of power M has two operation modes: grid-connected and in microgrid operation. Microgrid islanded. In the grid-conneected mode, an imbalance is solved by trading power w with the upstream power grid. However, H an imbalance should be solved without tradding power with any power grid in the islanded mode. In general, an imbalancee by slight supply shortage can be solved by dischargee of distributed storage systemss (DSs). However, critical imbalance cannot be solvedd by only discharge of DSs. In this case, an intentional reduction of load, i.e. lload shedding, is used in order to solve the critical imbalance. Although load sheddding ortable, it is the only practical option to meet the ratting makes consumers uncomfo frequency as a critical requirement of microgrids. A conventional load-sheddding scheme is to intentionally reduce load amounts by low order of priority of loaads. m of how to deal with loads having same priority. However, there is a problem Recently, an approach using the bankruptcy rule to load shedding has been propoosed m idea of the approach is to consider the load-sheddding by Kim et al [3-5]. The main

A Comparative Study of Bankruptcy Rules for Load-Shedding Scheme

147

problem as a bankruptcy problem dealing with dividing the short estate to claimants. In the approach, well-known bankruptcy rules such as the CEA rule, the CEL rule, the Talmud rule, and the RA rule have been applied. The results showed the feasibility for load shedding using the bankruptcy rules. By the approach, the load-shedding problem was defined as a pair (l, P), where P is available power and l = (l1,······, ln) is the vector of claims of loads, which is described as 0 ≤l1 ≤ ······≤ ln and 0 ≤ P ≤ l1+······+ln.

(1)

The vector of allocated power (l*) of each load is defined by (2), l* = (l1*,······ ,ln*),

(2)

where the available power is allocated by bankruptcy rule. The vector of amount of load-shedding of each load (s*) is calculated by (3) s* = (s1*,······ ,sn*) = l – l*.

(3)

3 Comparison of Bankruptcy Rules for Load-Shedding Scheme 3.1 Bankruptcy Rules In this paper, five bankruptcy rules, i.e. the proportional rule, the Talmud rule, the CEA rule, the CEL rule, and the RA rule as bankruptcy rules are considered. The details of the rules are described in [6]. In (4)-(8), P is available power and l = (l1,······, ln) is the vector of claims of loads. Allocating short power to loads using the proportional rule is defined as Pi (l, P) = l * = λli ,

(4)

where λ is chosen so that Σ λ li = P. Allocating short power to loads using the CEA rule is defined as

CEAi (l, P) = l * = min { l i ,λ } ,

(5)

where λ is chosen so that Σ min{li, λ} = P. Allocating short power to loads using the CEL rule is defined as CEL i (l, P) = l * = max { 0 , l i − λ } ,

(6)

where λ is chosen so that Σ max{0, li - λ} = P. Allocating short power to loads using the Talmud rule is defined as

⎧ min{li /2, λ} Ti (l,P) = l* = ⎨ ⎩ lci - min{li /2, λ}

≥

if Σ (li /2) P , other

(7)

where, and λ is chosen so that Σmin {lci/2, λ} = P and is chosen so that Σ [lci - min {lci/2, λ}] = P, respectively.

148

H.-M. Kim and T. Kinoshita

Allocating short power to loads using the RA rule is defined as

RAi (l, P) = l * =

1 min{ li , max{ P − n!

∑

j∈N ,π ( j )> 8 r1[i][j]= (bi.getRGB(i,j) & 0x00FF0000) >> 16

310

D. Bhattacharyya and T.-h. Kim

int avg = (r1[i][j]+g1[i][j]+b1[i][j])/3 int newRGB = 0xFF000000 + (avg