This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
elements are also indexed as retrievable units. The resulting index contains no overlapping elements. – Article index : We also build an index containing all full-text articles (i.e., all wikipages) as is standard in IR. – Anchor text index : For this index we concatenated all propagated anchor text of an article as a single anchor text representation for that article. For all indexes, stop-words were removed, and terms were stemmed using the Krovetz stemmer. Queries are processed similar to the documents. This year we only used the CO queries for the official runs. 2.2
Category Evidence
Based on previous experiments, we used category distance scores as extra evidence for ranking [11]. We determine two target categories for a query based on the top 20 results. We select the two most frequent categories to which the top 20 results are assigned and compute a category distance score using parsimonious language models of each category. This technique was successfully employed on the INEX 2007 Ad hoc topics by [8] and on the larger INEX 2009 collection [12] with two sets of category labels [11]; one based on the Wikipedia category structure and one based on the WordNet category labels. [11] found that the labels of the original Wikipedia category structure are more effective for ad hoc retrieval. In our experiments, we use the original Wikipedia category labels. The category distance scores are computed as follows. For each target category we estimate the distances to the categories assigned to retrieved document,
142
J. Kamps and M. Koolen
similar to what is done in [17]. The distance between two categories is estimated according to the category titles. In previous experiments we also experimented with a binary distance, and a distance between category contents, but we found the distance estimated using category titles the most efficient and at the same time effective method. To estimate title distance, we need to calculate the probability of a term occurring in a category title. To avoid a division by zero, we smooth the probabilities of a term occurring in a category title with the background collection: n P (t1 , ..., tn |C) = λP (ti |C) + (1 − λ)P (ti |D) i=1
where C is the category title and D is the entire wikipedia document collection, which is used to estimate background probabilities. We estimate P (t|C) with a parsimonious model [2] that uses an iterative EM algorithm as follows: E-step: M-step:
et = tft,C ·
αP (t|C) αP (t|C) + (1 − α)P (t|D)
et P (t|C) = , i.e. normalize the model t et
The initial probability P (t|C) is estimated using maximum likelihood estimation. We use KL-divergence to calculate distances, and calculate a category score that is high when the distance is small as follows: P (t|Ct ) Scat (Cd |Ct ) = −DKL (Cd |Ct ) = − P (t|Ct ) ∗ log t∈D P (t|Cd ) where d is a retrieved document, Ct is a target category and Cd a category assigned to a document. The score for a document in relation to a target category S(d|Ct ) is the highest score, or shortest distance from any of the document’s categories to the target category. So if one of the categories of the document is a target category, the distance and also the category score for that target category is 0, no matter what other categories are assigned to the document. Finally, the score for a document in relation to a query topic S(d|QT ) is the sum of the scores of all target categories: Scat (d|QT ) = argmax S(Cd |Ct ) Ct ∈QT
Cd ∈d
Besides the category score, we also need a query score for each document. This score is calculated using a language model with Jelinek-Mercer smoothing with a linear length prior: P (q1 , ..., qn |d) = P (d)
n
λP (qi |d) + (1 − λ)P (qi |D)
i=1
where P(d) is proportional to the document length, computed as: P (d) =
|d| |D|
Focus and Element Length for Book and Wikipedia Retrieval
143
Finally, we combine our query score and the category score through a linear combination. For our official runs both scores are calculated in the log space, and then a weighted addition is made. S(d|QT ) = μP (q|d) + (1 − μ)Scat (d|QT ) Based on previous experiments [8], we use μ = 0.9. 2.3
Runs
Combining the methods described in the previous section with our baseline runs leads to the following official runs. Article: an article index run with length prior (λ = 0.85 and β = 1). ArticleRF: an article index run with length prior (λ = 0.85 and β = 1) and relevance feedback (top 50 terms from top 10 results). Anchor: anchor text index run without length prior (λ = 0.85 and β = 0). AnchorLen: anchor text index run with length prior (λ = 0.85 and β = 1). Sec: a section index run without length prior (λ = 0.85 and β = 0). SecLen: a section index run with length prior (λ = 0.85 and β = 1). From these initial runs we have constructed our baseline runs: Base: the ArticleRF combined with the category scores based on the 2 most frequent categories of the top 20 results. Base Sec: the Baseline run where an article is replaced by the sections of that article retrieved by the Sec run. If no sections for that article are retrieved, the full article is used. Fusion: a linear combination of the ArticleRF and the AnchorLen runs with weight S(d) = 0.7ArticleRF (d) + 0.3AnchorLen(d). The combined run is used to compute category scores based on the 2 most frequent categories of the top 20 results, which are then combined with the merged article and anchor text scores. Fusion Sec: the Fusion run where an article is replaced by the sections of that article retrieved by the Sec run. If no sections for that article are retrieved, the full article is used. For the Focused Task, systems are restricted to return no more than 1000 characters per topic. We submitted two runs: Base Sec F1000 Topic: The Base Sec run with only the first 1000 characters retrieved for each topic. Base Sec F100 Article: The Base Sec run with only the first 100 characters retrieved per article, cut-off after 1000 characters retrieved for each topic. With the first 1000 characters retrieved, we expect to return only very few documents per topic. With a restriction of at most N characters per document, we can control the minimum number of documents returned, thereby increasing the possible number of relevant documents returned. Both runs have the retrieved
144
J. Kamps and M. Koolen
sections grouped per article, with the sections ordered according to the retrieval score of the Sec run. That is, if sections s1, s2 and s3 of document d1 are retrieved by the Sec run in the order (s2, s3, s1), then after grouping, s2 is still returned first, then s3 and then s1. The first 1000 characters retrieved will come mostly from a single document (the highest ranked document). With a limit of 100 characters per article, the first 1000 characters will come from at least 10 documents. Although precision among the first 10 documents will probably be lower than precision at rank 1, the larger number of retrieved documents might give the user access to more relevant documents. We will look at the set-based precision of the 1000 characters retrieved as well as the article-based precision and the number of retrieved and relevant retrieved articles. For the Relevant in Context Task, systems are restricted to returning no more than 1500 results. We submitted two runs: Base SecLen: the baseline run Base SecLen described above, cut off after the first 1500 results. Fusion Sec: the baseline run Fusion Sec described above, cut off after the first 1500 results. The Base and Fusion runs will allow us to see the impact of using propagated anchor-text for early precision. For the Restricted Relevant in Context Task, systems are restricted to returning no more than 500 characters per result. We submitted two runs: Base F500 Article: the Base run reduced to the first 500 characters per retrieved article, and cut off after the first 1500 results. Base Sec F500 Article: the Base Sec run reduced to the first 500 characters per retrieved article, and cut off after the first 1500 results. Article retrieval is a competitive alternative to element retrieval when it comes to focused retrieval in Wikipedia [3, 6]. The full per-article recall of article retrieval makes up for its lack in focus. However, for the Restricted Relevant in Context Task, the amount of text retrieved per article is limited to 500 characters, which reduces the high impact of full within-document recall and puts more emphasis on achieving high precision. Relevant articles tend to have relevant text near the start of the article [7], which could give fair precision with the first 500 characters of an article. On the other hand, using the more focused evidence of the section index on the same article ranking, we can select the first 500 characters of the most promising elements of the article. With a restricted number of characters per article, and therefore restricted recall, we expect to a see a clearer advantage in using focused retrieval techniques. We discovered an error in the baseline runs, which caused our official runs to have very low scores. In the next sections, we show results for both the officially submitted runs and the corrected runs. 2.4
Thorough Evaluation
We first look at the performance of the baseline runs using the Thorough interpolated precision measure. Results can be found in Table 1. The Fusion run is
Focus and Element Length for Book and Wikipedia Retrieval
145
Table 1. Interpolated precision scores of the baseline runs (runs in italics are official submissions, runs with an asteriks are the corrected versions) Run id Base Fusion Base Sec *Base SecLen *Fusion Sec Base SecLen Fusion Sec
MAiP iP[0.00] iP[0.01] iP[0.05] iP[0.10] 0.2139 0.4398 0.4219 0.3810 0.3577 0.1823 0.4001 0.3894 0.3370 0.3189 0.1555 0.5669 0.5130 0.4039 0.3600 0.1702 0.5507 0.5100 0.4162 0.3784 0.1317 0.5447 0.4632 0.3439 0.2967 0.0723 0.3308 0.2910 0.2184 0.1944 0.0678 0.3027 0.2694 0.2110 0.1906
less effective than the Base run. The anchor text does not help early precision, although we did not range over all possible weighted combinations. Perhaps a lower weight on the anchor text might be beneficial. The length prior on the sections increases recall for the cost of a slight drop in early precision. The focused runs have a lower MAiP but a higher early precision than the article level runs. The article level runs have a much higher recall, and thereby score better on average precision. But the focused runs retrieve less irrelevant text and score better on early precision. 2.5
Focused Task
We have no overlapping elements in our indexes, so no overlap filtering is done. However, systems can return no more than 1000 characters per topic. This means the result list needs to be cut-off. Articles tend to be longer than 1000 characters, so for the article-level runs, some part of an article needs to be selected to be returned. Section elements might also be longer, in which case a part of a section needs to be selected. We choose to select the first 1000 characters returned in retrieval order, based on the idea that precision tends to drop over ranks. For example, if the highest ranked section has 500 characters and the second ranked section has 600 characters, we return the highest ranked section and the first 500 characters of the second ranked section. If the highest ranked section or article is longer than 1000 characters, we select the first 1000 characters. Because relevant text tends to start near the beginning of XML elements [4], we also looked at a method that cuts off results after 100 characters, so that the initial text from multiple high-ranked results is returned. The mean length of Sec results is 936, although most results have far fewer (median is 397) characters. The first 1000 characters often corresponds to more than one section. The SecLen and Base results are longer, with a median length of 2,227 and 6,098 characters respectively (mean lengths are 5,763 and 13,383). For most topics, the first returned result needs to be cut-off after the first 1000 characters. How does precision among the first 1000 characters correspond to older Focused evaluation measures, such as interpolated precision at the first percentage of recall (iP[0.01])? The median number of characters per topic is 129,440 (mean
146
J. Kamps and M. Koolen
Table 2. Results for the Ad Hoc Track Focused Task (runs in italics are official submissions, runs with an asterix are the corrected versions) Run id Base Sec F1000 Topic Base Sec F100 Article Sec F1000 Topic SecLen F1000 Topic *Base Sec F1000 Topic Base SecLen F1000 Topic Base F1000 Topic Sec F100 Article SecLen F100 Article *Base Sec F100 Article Base SecLen F100 Article Base F100 Article
# ret. # rel. ret. Particle Pchar iP[0.00] iP[0.01] 1.25 0.38 0.3301 0.1232 0.1694 0.0386 10.17 3.17 0.3105 0.1162 0.2468 0.0338 3.94 2.13 0.5196 0.2340 0.3372 0.1087 1.56 0.90 0.5667 0.2975 0.3188 0.1261 1.29 0.81 0.6250 0.3490 0.4012 0.1376 1.29 0.81 0.6186 0.3526 0.3903 0.1518 1.10 0.69 0.6250 0.2806 0.2828 0.0737 10.63 4.87 0.4576 0.2127 0.4117 0.0842 10.08 4.85 0.4804 0.1981 0.4403 0.0934 10.06 5.27 0.5229 0.2445 0.4626 0.1140 10.06 5.27 0.5229 0.2677 0.5015 0.1226 10.00 5.23 0.5231 0.1415 0.2623 0.0340
339,252). For most topics, precision of the first 1000 retrieved characters is somewhere between iP[0.00] and iP[0.01]. Table 2 shows the results for the Focused Task, where Particle is the articlelevel precision and Pchar is the character-level precision. Article-level precision is based on binary relevance judgements. If the returned (part of the) article contains relevant text, the article is considered relevant at this level. For the character level precision, the set-based precision of all returned characters is computed as the number of returned highlighted characters divided by the total number of returned characters. As expected, the Pchar for all the runs is somewhere between iP[0.00] and iP[0.01]. The first 1000 retrieved characters (denoted F1000 Topic) gives higher precision than first 100 per article up to 1000 characters (denoted F100 Article). But by restricting each article to 100 characters, many more articles, including relevant articles, are retrieved. Thus, although the set-based precision of the F100 Article runs is lower, they do give direct access to many more relevant documents. The focused runs Base Sec and Base SecLen have a higher set-based character precision than the article level Base run. The length prior on the section index has a positive impact on the precision of the first 1000 characters. The Base Sec and Base SecLen runs have the same number of retrieved articles and retrieved relevant articles, but the Base SecLen run has more relevant text in the first 1000 characters. The query-independent length prior helps locate elements with a larger proportion of relevant text. We note that, although the F100 Article runs have a lower Pchar than the F1000 Topic runs, they have a higher iP[0.00] score. This is because they return smaller and therefore more results. Relevant text tends to be concentrated near the start of XML elements [4], so it is often beneficial to return text from the start of an element. In the case of the F1000 Topic runs, the first retrieved result is usually 1000 characters long, so iP[0.00], which is determined after the first relevant retrieved result, is based on the first 1000 characters of an element, be it article or section. For the F100 Article runs, iP[0.00] will often be based on
Focus and Element Length for Book and Wikipedia Retrieval
147
Table 3. Results for the Ad Hoc Track Relevant in Context Task (runs in italics are official submissions, runs with an asteriks are the corrected versions) Run id Base Sec Len Fusion Sec Base Base Sec *Base Sec Len Fusion *Fusion Sec
MAgP 0.0597 0.0563 0.1613 0.1615 0.1646 0.1344 0.1294
gP[5] 0.1492 0.1207 0.2900 0.3026 0.3149 0.2849 0.2840
gP[10] 0.1330 0.1068 0.2619 0.2657 0.2790 0.2399 0.2427
gP[25] 0.1080 0.1008 0.2123 0.2112 0.2213 0.1945 0.1917
gP[50] 0.1031 0.0963 0.1766 0.1763 0.1817 0.1547 0.1548
Table 4. Results for the Ad Hoc Track Restricted Relevant in Context Task (runs in italics are official submissions, runs with an asteriks are the corrected versions) Run id Base F500 Article Base Sec F500 Article *Base F500 Article *Base Sec F500 Article Base SecLen F500 Article
MAgP 0.0576 0.0566 0.1358 0.1503 0.1545
gP[5] 0.1439 0.1375 0.2516 0.2592 0.2666
gP[10] 0.1191 0.1199 0.2186 0.2288 0.2368
gP[25] 0.1053 0.1040 0.1696 0.1887 0.1868
gP[50] 0.0980 0.0952 0.1473 0.1624 0.1570
a smaller number of characters. If the first 500 characters of the highest ranked section or article are relevant, taking the first 1000 characters from that section or article gives an iP[0.00] of 0.5, while taking the first 100 characters gives an iP[0.00] of 1. 2.6
Relevant in Context Task
For the Relevant in Context Task, we group results per article. Table 3 shows the results for the Relevant in Context Task. We make the following observations: – The difference between the Base and Fusion runs is small. – The length prior on the section index results in higher early and average precision. 2.7
Restricted Relevant in Context Task
The aim of the Restricted Relevant in Context task is to return relevant results grouped per article, with a restriction to return no more than 500 characters per article. Table 4 shows the results for the Best in Context Task. We make the following observations: – Similar to the normal Relevant in Context task, the focused run Base Sec F500 Article has somewhat better precision than the run based on the full articles. – A length prior over the element lengths (Base SecLen F500 Article) leads to a further improvement in precision. Thus, longer elements give higher precision in the first 500 characters.
148
J. Kamps and M. Koolen
In summary, with the restrictions on the amount of text per article and per topic that can be retrieved, focused retrieval techniques clearly outperform standard document retrieval. What is somewhat surprising is that a length prior on the section index is effective even when we use the article ranking of an article level run. The query-independent length prior helps locate elements with a large fraction and amount of relevant text.
3
Book Track
In the INEX 2010 Book Track we participated in the Best Book and Prove It tasks. Continuing our efforts of last year, we aim to find the appropriate level of granularity for focused book search. The BookML markup has XML elements on the page level. In the assessments of last year, relevant passages often cover multiple pages [9]. With larger relevant passages, query terms might be spread over multiple pages, making it hard for a page level retrieval model to assess the relevance of individual pages. Can we better locate relevant passages by considering larger book parts as retrievable units? One simple option is to divide the whole book in sequences of n pages. Another approach would be to use the logical structure of a book to determine the retrievable units. The INEX Book corpus has no explicit XML elements for the various logical units of the books, so as a first approach we divide each book in sequences of pages. Book Index: each whole book is indexed as a retrievable unit. This index contains 50,239 books. 1-Page Index: each individual page is indexed as a retrievable unit. This index contains 16,105,669 1-page units. 5-Page Index: each sequence of 5 pages is indexed as a retrievable unit. That is, pages 1-5, 6-10, etc., are treated as text units. Note that a book with 53 pages is divided into 11 units: 10 units with 5 pages and 1 unit with 3 pages. Most books have more than 300 pages, so the number of units with less than 5 pages is small; less than 2% of units in the index, in fact. The average number of pages per unit is 4.97. This index contains 3,241,347 5-page units. For the 2010 Book Track, there are 83 topics in total. Each of these topics consists of a factual statement derived from a book in the collection, as well as a query with the most important words in the factual statement. Of these 83 topics, 21 have relevance assessments for pages pooled from the official runs. For the Best Book task, the aim is to return the best books in the collection on the general topic of the statement. For the Prove It task, the aim is to return pages that either refute or confirm the factual statement. We submitted six runs in total: two for the Best Book (BB) task and four for the Prove It (PI) task. Book: a standard Book index run. Up to 100 results are returned per topic. Book RF: a Book index run with Relevance Feedback (RF). The initial queries are expanded with 50 terms from the top 10 results.
Focus and Element Length for Book and Wikipedia Retrieval
149
Table 5. Page level statistics for the 2010 Prove It runs Run # pages # books # pages/book 1-page 985 464 2.1 1-page RF 1000 444 2.3 5-page 988 112 8.8 5-page RF 1000 105 9.5 Table 6. Results for the inex 2010 Best Book Task Run map P@5 ndcg@5 P@10 ndcg@10 Book 0.3286 0.4952 0.4436 0.4429 0.4151 Book-RF 0.3087 0.4667 0.3948 0.4286 0.3869
Page: a standard Page index run. Page RF: a Page index run with Relevance Feedback (RF). The initial queries are expanded with 50 terms from the top 10 results. 5-Page: a standard 5-Page index run. Because the returned results need to be individual pages, each 5-page unit is segmented into its individual pages. The highest ranked 5-page unit is split into 5 pages with each page receiving the same score as the 5-page unit. Thus, the first N retrieved units are presented as 5N results. 5-Page RF: a 5-Page index run with Relevance Feedback (RF). The initial queries are expanded with 50 terms from the top 10 results. 3.1
Prove It Run Analysis
With the single page index, we get an average of 2.1 pages per book, while the 5-page index gives us 8.8 pages per book. With an average 4.97 pages per unit, that is 1.77 units per book. The 5-page index gives us less units per book than the single page index, but we get more pages from a single book. The impact of relevance feedback is a slight increase in the number of pages from the same book. Expanding the query with terms from the top ranked book pages probably results in retrieving more pages from those books because similar terms are used throughout the same book. 3.2
Best Book Results
In Table 6 we see the results for the Best Book Task. The standard Book retrieval system performs better than the system with pseudo relevance feedback. PRF expands the topic of the query somewhat, which is unnecessary for the task of finding only the best books on a topic. Books are have graded relevance judgements, with labels excellent, good, marginal, irrelevant and cannot judge. The latter is used for instance when the scanned image of a book is unreadable. For the Best Book task we are mainly interested in books labelled good or excellent. In Figure 1 we see the success rates
150
J. Kamps and M. Koolen 0.9
Book good Book excl Book-RF good Book-RF excl
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
2
3
4
5
6
7
8
9
10
Fig. 1. Success rate over ranks 1 to 10 in the 2010 Best Books task Table 7. Mapping of the Prove It relevance labels to graded relevance values Label Official Extra Irrelevant (0) 0 0 Relevant (1) 1 1 Refute (2) or Confirm (3) 2 10
over the top 10 for retrieving at least one book labelled good or higher, and for retrieving at least one excellent book. We see that the Book run has a success rate for books labelled good or higher of almost 0.4 at rank 1 going up to 0.81 at rank 10. The Book-RF runs has a slightly lower success rate. For the books labelled excellent, the success rates are much lower. The Book run returns an excellent book in the top 10 for 33% of the topics, the Book-RF run for 24% of the topics. 3.3
Prove It Results
For the Prove It task, the goal is to return individual book pages that either confirm or refute a factual statement. The same set of topics is used as for the Best Book task, but here, participants could use the query field, as well as the factual statement. For our runs we only use the query field. The assessed book pages could be assessed as either irrelevant, relevant, refute or confirm. Pages are labelled relevant when they discuss the topic of the statement, but neither refute or confirm that statement. These labels can be mapped to relevance values in multiple ways. We show results for the official mapping, and a mapping in which the refute/confirm pages are weighted extra. These mappings to graded relevance values are shown in Table 7. For the Extra mappings, the refute/confirm pages weight 10 times as much as pages labelled relevant. The nDCG@10 measure uses the graded relevance scores, and we regard the nDCG@10 as the official measure. The results are given in Table 8. The 1-page index gives better results than the 5-page index in terms of early precision. For
Focus and Element Length for Book and Wikipedia Retrieval
151
Table 8. Evaluation results for the INEX 2010 Prove It task, nDCG@10 with official weights 0-1-2 (4th column) and Extra weights 0-1-10 (5th column) Run ID
MAP
P@10
1-page 0.1216 0.3238 1-page RF 0.1521 0.3524 5-page 0.1209 0.2619 5-page RF 0.1163 0.2143
nDCG@10 Official Extra 0.2795 0.2338 0.2946 0.2322 0.2182 0.1714 0.1703 0.1371
the 1-page index, feedback improves precision, while for the 5-page index, feedback hurts performance. The top single page results are probably more focussed on the topic, so the expansion terms are also more related to the topic. The 5page units might have only one or two pages on the topic, with the surrounding pages slowly drifting to other topics, which causes further topic drift through pseudo relevance feedback. If we put extra weight on the confirm/retute pages, the nDCG@10 scores of all four runs drop. However, they drop slightly more for the 1-page RF run than for the other runs. Although feedback improves precision of the 1-page run, it apparently finds more relevant pages, but fewer confirm/refute pages. This could again be caused by topic drift introduced through pseudo relevance feedback. Longer units seem to provide no benefit to finding confirm/refute pages or relevant pages. At least not when treating each page within the 5-page units as individual results. Of course, one could still score a whole 5-page unit if any one of the 5 pages is relevant or confirms/refutes the factual statement. However, the relevance judgements treat each page as a separate document, so scoring larger units based on these page level judgements would give systems returning larger units an unfair advantage, as they can return more pages within the same number of ranks as systems returning only individual pages. We have also looked at near-misses, that is, retrieved pages that are not relevant themselves, but are next to a relevant pages in the book. Because pages typically 0.3 0.25 0.2 0.15 0.1 1-page 5-page 1-page RF 5-page RF
0.05 0
0
2
4
6
8
10
Fig. 2. Impact of near misses on nDCG@10 performance in the 2010 Prove It task
152
J. Kamps and M. Koolen
do not coincide with the logical of a book—section and paragraphs can start or end anywhere on the page and be split over multiple pages—a topic might be discussed over multiple pages. Although the information requested by a user might be on one page, the page preceding or succeeding it might still be closely related, and contain clues for the reader that the actual relevant information is nearby.
4
Conclusion
In this paper we discussed our participation in the INEX 2010 Ad Hoc and Book Tracks. For the Ad Hoc Track we found that, with the restrictions on the amount of text per article and per topic that can be retrieved, focused retrieval techniques clearly outperform standard document retrieval. What is somewhat surprising is that a length prior on the section index is effective even when we use the article ranking of an article level run. The query-independent length prior helps locate elements with a large fraction and amount of relevant text. For the Book Track, we found that pseudo relevance feedback is effective in combination with an index of individual pages as units. With larger units, feedback hurts precision. Given that the topics consist of specific factual statements, feedback only works when the top ranked results are highly focused on the topic of these statements. Although larger units might make it easier to find relevant material—early precision of the book-level runs is higher than that of the page-level runs—they also provide more off-topic text with which feedback terms introduce topic drift. Acknowledgments. Jaap Kamps was supported by the Netherlands Organization for Scientific Research (NWO, grants # 612.066.513, 639.072.601, and 640.001.501). Marijn Koolen was supported by NWO under grants # 639.072.601 and 640.001.501.
Bibliography [1] Fachry, K.N., Kamps, J., Koolen, M., Zhang, J.: Using and detecting links in wikipedia. In: Fuhr, N., Kamps, J., Lalmas, M., Trotman, A. (eds.) INEX 2007. LNCS, vol. 4862, pp. 388–403. Springer, Heidelberg (2008) [2] Hiemstra, D., Robertson, S., Zaragoza, H.: Parsimonious language models for information retrieval. In: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 178–185. ACM Press, New York (2004) [3] Kamps, J., Koolen, M.: The impact of document level ranking on focused retrieval. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2008. LNCS, vol. 5631, pp. 140– 151. Springer, Heidelberg (2009) [4] Kamps, J., Koolen, M.: On the relation between relevant passages and XML document structure. In: Trotman, A., Geva, S., Kamps, J. (eds.) SIGIR 2007 Workshop on Focused Retrieval, pp. 28–32. University of Otago, Dunedin (2007)
Focus and Element Length for Book and Wikipedia Retrieval
153
[5] Kamps, J., Koolen, M., Sigurbj¨ ornsson, B.: Filtering and clustering XML retrieval results. In: Fuhr, N., Lalmas, M., Trotman, A. (eds.) INEX 2006. LNCS, vol. 4518, pp. 121–136. Springer, Heidelberg (2007) [6] Kamps, J., Koolen, M., Lalmas, M.: Locating relevant text within XML documents. In: Proceedings of the 31th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 847–849. ACM Press, New York (2008) [7] Kamps, J., Geva, S., Trotman, A., Woodley, A., Koolen, M.: Overview of the INEX 2008 ad hoc track. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2008. LNCS, vol. 5631, pp. 1–28. Springer, Heidelberg (2009) [8] Kaptein, R., Koolen, M., Kamps, J.: Using Wikipedia categories for ad hoc search. In: Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, New York (2009) [9] Kazai, G., Milic-Frayling, N., Costello, J.: Towards methods for the collective gathering and quality control of relevance assessments. In: SIGIR 2009: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 452–459. ACM, New York (2009), doi: http://doi.acm.org/10.1145/1571941.1572019 ISBN 978-1-60558-483-6 [10] Koolen, M., Kamps, J.: The importance of anchor-text for ad hoc search revisited. In: Chen, H.-H., Efthimiadis, E.N., Savoy, J., Crestani, F., Marchand-Maillet, S. (eds.) Proceedings of the 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 122–129. ACM Press, New York (2010) [11] Koolen, M., Kaptein, R., Kamps, J.: Focused search in books and wikipedia: Categories, links and relevance feedback. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2009. LNCS, vol. 6203, pp. 273–291. Springer, Heidelberg (2010) [12] Schenkel, R., Suchanek, F., Kasneci, G.: Yawn: A semantically annotated wikipedia xml corpus (2007), http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.140.5501 [13] Sigurbj¨ ornsson, B., Kamps, J.: The effect of structured queries and selective indexing on XML retrieval. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 104–118. Springer, Heidelberg (2006) [14] Sigurbj¨ ornsson, B., Kamps, J., de Rijke, M.: An Element-Based Approach to XML Retrieval. In: INEX 2003 Workshop Proceedings, pp. 19–26 (2004) [15] Sigurbj¨ ornsson, B., Kamps, J., de Rijke, M.: Mixture models, overlap, and structural hints in XML element retrieval. In: Fuhr, N., Lalmas, M., Malik, S., Szl´avik, Z. (eds.) INEX 2004. LNCS, vol. 3493, pp. 196–210. Springer, Heidelberg (2005) [16] Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: a language-model based search engine for complex queries. In: Proceedings of the International Conference on Intelligent Analysis (2005) [17] Vercoustre, A.-M., Pehcevski, J., Thom, J.A.: Using Wikipedia categories and links in entity ranking. In: Focused Access to XML Documents, pp. 321–335 (2007)
Combining Page Scores for XML Book Retrieval Ray R. Larson School of Information University of California, Berkeley Berkeley, California, USA, 94720-4600 [email protected]
Abstract. In the 2010 INEX Evaluation UC Berkeley participated only in the Book track, and specifically the “Best Books to Reference” task that seeks to produce a list of the “best books” for a topic. This year we wanted to compare our best performing method from last year with approaches that combine the scores obtained by ranking at the page level to arrive at the ranking for a book. We tested a number of combinations for this approach, and were able to obtain the top score for the “Best Books” task.
1
Introduction
In our 2009 work on the INEX Book Track, we tried a variety of different approaches for our Book Track runs, including the TREC2 logistic regression probabilistic model as well as various fusion approaches including combining the Okapi BM-25 algorithm with other approaches. But in most cases our previous approaches used only full-book level indexes. We observed that most of the fusion approaches that we had tried were not as effective as the TREC2 Logistic Regression with Blind Feedback, so we took that as our baseline for this year, and none of our approaches were as effective as those reported by some other groups. After testing a number of approaches using book-level indexes and different fusion weights, we found that this was still the case, and that these attempts often led to poor matches being ranked highly. We decided instead to try some radical simplification of the ranking process for books. This was driven by observations from earlier INEX evaluations and from some of our digital library work that often the books with highly ranked pages turned out to be more better choices for the user than books with high overall ranking scores. Since we had generated page-level indexes for all of the books (see below), we decided to try two simple approaches. A probabilistic approach based on our logistic regression algorithm (but without blind feedback), and a simple coordination-level match for pages. In this paper we will first discuss the algorithms and operators used in our official INEX 2010 Book Track runs. Then we will look at how these algorithms and operators were used in combination with page-level indexes for our submissions, and finally we will discuss possible directions for future research. S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 154–163, 2011. c Springer-Verlag Berlin Heidelberg 2011
Combining Page Scores for XML Book Retrieval
2
155
The Retrieval Algorithms and Fusion Operators
This section largely duplicates parts of earlier INEX papers in describing the probabilistic retrieval algorithms used for the Book track in INEX this year. Although the algorithms are the same as those used in previous years for INEX and in other evaluations (such as CLEF and NTCIR), including a blind relevance feedback method used in combination with the TREC2 algorithm, we are repeating the formal description here instead of refering to those earlier papers alone. In addition we will discuss the simple methods used to combine the results of searches of book page elements in the collections. All runs used our Cheshire II XML/SGML search engine [9,8,7] which also supports a number of other algorithms for distributed search and operators for merging result lists from ranked or Boolean sub-queries. 2.1
TREC2 Logistic Regression Algorithm
Once again the primary algorithm used for our INEX baseline runs is based on the Logistic Regression (LR) algorithm originally developed at Berkeley by Cooper, et al. [5]. The version that we used for Adhoc tasks was the Cheshire II implementation of the “TREC2” [4,3] that has provided good retrieval performance earlier evaluations[9,10]. As originally formulated, the LR model of probabilistic IR attempts to estimate the probability of relevance for each document based on a set of statistics about a document collection and a set of queries in combination with a set of weighting coefficients for those statistics. The statistics to be used and the values of the coefficients are obtained from regression analysis of a sample of a collection (or similar test collection) for some set of queries where relevance and non-relevance has been determined. More formally, given a particular query and a particular document in a collection P (R | Q, D) is calculated and the documents or components are presented to the user ranked in order of decreasing values of that probability. To avoid invalid probability values, the usual calculation of P (R | Q, D) uses the “log odds” of relevance given a set of S statistics derived from the query and database, such that: log O(R|C, Q) = log
p(R|C, Q) p(R|C, Q) = log 1 − p(R|C, Q) p(R|C, Q)
= c0 + c1 ∗ + c2 ∗ − c3 ∗
|Qc |
qtfi |Qc | + 1 i=1 ql + 35 1
1 |Qc | + 1 1 |Qc | + 1
|Qc |
log
tfi cl + 80
log
ctfi Nt
i=1 |Qc |
i=1
+ c4 ∗ |Qc | where C denotes a document component and Q a query, R is a relevance variable, and
156
R.R. Larson
p(R|C, Q) is the probability that document component C is relevant to query Q, p(R|C, Q) the probability that document component C is not relevant to query Q, (which is 1.0 - p(R|C, Q)) |Qc | is the number of matching terms between a document component and a query, qtfi is the within-query frequency of the ith matching term, tfi is the within-document frequency of the ith matching term, ctfi is the occurrence frequency in a collection of the ith matching term, ql is query length (i.e., number of terms in a query like |Q| for non-feedback situations), cl is component length (i.e., number of terms in a component), and Nt is collection length (i.e., number of terms in a test collection). ck are the k coefficients obtained though the regression analysis. Assuming that stopwords are removed during index creation, then ql, cl, and Nt are the query length, document length, and collection length, respectively. If the query terms are re-weighted (in feedback, for example), then qtfi is no longer the original term frequency, but the new weight, and ql is the sum of the new weight values for the query terms. Note that, unlike the document and collection lengths, query length is the relative frequency without first taking the log over the matching terms. The coefficients were determined by fitting the logistic regression model specified in log O(R|C, Q) to TREC training data using a statistical software package. The coefficients, ck , used for our official runs are the same as those described by Chen[1]. These were: c0 = −3.51, c1 = 37.4, c2 = 0.330, c3 = 0.1937 and c4 = 0.0929. Further details on the TREC2 version of the Logistic Regression algorithm may be found in Cooper et al. [4]. 2.2
Blind Relevance Feedback
It is well known that blind (also called pseudo) relevance feedback can substantially improve retrieval effectiveness in tasks such as TREC and CLEF. (See for example the papers of the groups who participated in the Ad Hoc tasks in TREC-7 (Voorhees and Harman 1998)[11] and TREC-8 (Voorhees and Harman 1999)[12].) Blind relevance feedback is typically performed in two stages. First, an initial search using the original queries is performed, after which a number of terms are selected from the top-ranked documents (which are presumed to be relevant). The selected terms are weighted and then merged with the initial query to formulate a new query. Finally the reweighted and expanded query is run against the same collection to produce a final ranked list of documents. It was a simple extension to adapt these document-level algorithms to document components for INEX. The TREC2 algorithm has been been combined with a blind feedback method developed by Aitao Chen for cross-language retrieval in CLEF. Chen[2] presents
Combining Page Scores for XML Book Retrieval
157
a technique for incorporating blind relevance feedback into the logistic regressionbased document ranking framework. Several factors are important in using blind relevance feedback. These are: determining the number of top ranked documents that will be presumed relevant and from which new terms will be extracted, how to rank the selected terms and determining the number of terms that should be selected, how to assign weights to the selected terms. Many techniques have been used for deciding the number of terms to be selected, the number of topranked documents from which to extract terms, and ranking the terms. Harman [6] provides a survey of relevance feedback techniques that have been used. Obviously there are important choices to be made regarding the number of top-ranked documents to consider, and the number of terms to extract from those documents. We used the same default as last year, i.e., top 10 terms from 10 topranked documents. The terms were chosen by extracting the document vectors for each of the 10 and computing the Robertson and Sparck Jones term relevance weight for each document. This weight is based on a contingency table where the counts of 4 different conditions for combinations of (assumed) relevance and whether or not the term is, or is not in a document. Table 1 shows this contingency table. Table 1. Contingency table for term relevance weighting Relevant Not Relevant In doc Rt Nt − Rt Nt Not in doc R − Rt N − Nt − R + Rt N − Nt R N −R N
The relevance weight is calculated using the assumption that the first 10 documents are relevant and all others are not. For each term in these documents the following weight is calculated: wt = log
Rt R−Rt Nt −Rt N −Nt −R+Rt
(1)
The 10 terms (including those that appeared in the original query) with the highest wt are selected and added to the original query terms. For the terms not in the original query, the new “term frequency” (qtfi in main LR equation above) is set to 0.5. Terms that were in the original query, but are not in the top 10 terms are left with their original qtfi . For terms in the top 10 and in the original query the new qtfi is set to 1.5 times the original qtfi for the query. The new query is then processed using the same TREC2 LR algorithm as shown above and the ranked results returned as the response for that topic. 2.3
Coordination Level Matching
Coordination level matching is the first simple step towards ranking results beyond simple Boolean matches. Basic coordination level matching (CML) is
158
R.R. Larson
simply the number of terms in common between the query and the document component or |Qc | as defined above. In the implementation that we use in the Cheshire II system, the coordination level match (CLM) also takes into account term frequency, thus it is simply: |Qc |
CLMc =
tfi
(2)
i=1
Where the variables are the defined the same as defined above. Obviously, with this simple form, it is possible for terms that have very high frequency to dominate. To combat this an additional filter removes all results that match on fewer than 1/4 of the search terms. 2.4
Result Combination Operators
As we have also reported previously, the Cheshire II system used in this evaluation provides a number of operators to combine the intermediate results of a search from different components or indexes. With these operators we have available an entire spectrum of combination methods ranging from strict Boolean operations to fuzzy Boolean and normalized score combinations for probabilistic and Boolean results. These operators are the means available for performing fusion operations between the results for different retrieval algorithms and the search results from different components of a document. However, our approach for the page-level searches done for this evaluation was to simply sum the page-level results for each book. Thus, for the CLM runs, if a particular query retrieved 10 pages from a given book, the final ranking score would be the sum of the CLM values for each page. Although the runs using the TREC2 logistic regression algorithm return estimates of the probability of relevance for each page, we decided to treat these also as simple scores and sum each matching page estimate for each book.
3
Database and Indexing Issues
The Book Track data used this year was the same as last year. In indexing we attempted to use multiple elements or components that were identified in the Books markup including the Tables of Contents and Indexes as well as the full text of the book, since the goal of the “Best Books” task was to retrieve entire books and not elements, the entire book was retrieved regardless of the matching elements. Table 2 lists the Book-level (/article) indexes created for the INEX Books database and the document elements from which the contents of those indexes were extracted. Cheshire system permits parts of the document subtree to be treated as separate documents with their own separate indexes. Tables 3 & 4 describe the XML components created for the INEX Book track and the component-level indexes that were created for them.
Combining Page Scores for XML Book Retrieval
159
Table 2. Book-Level Indexes for the INEX Book Track 2009-10 Name topic toc index
Description Full content Tables of Contents Back of Book Indexes
Contents Vector? //document Yes //section@label=”SEC TOC” No //section@label=”SEC INDEX” No
Table 3. Components for INEX Book Track 2009-10 Name Description Contents COMPONENT PAGE Pages //page COMPONENT SECTION Sections //section
Table 3 shows the components and the paths used to define them. The first, refered to as COMPONENT PAGE, is a component that consists of each identified page of the book, while COMPONENT SECTION identifies each section of the books, permitting each individual section or page of a book to be retrieved separately. Because most of the areas defined in the markup as “section”s are actually paragraphs, we treat these as if they were paragraphs for the most part. Table 4. Component Indexes for INEX Book Track 2009-10 Component or Index Name Description Contents Vector? COMPONENT SECTION para words Section Words * (all) Yes COMPONENT PAGES page words Page Words * (all) Yes
Table 4 describes the XML component indexes created for the components described in Table 3. These indexes make the individual sections (such as COMPONENT SECTION) of the INEX documents retrievable by their titles, or by any terms occurring in the section. These are also proximity indexes, so phrase searching is supported within the indexes. We also have indexes created using the MARC data (book-level metadata) made available, but these were not used this year. 3.1
Indexing the Books XML Database
Because the structure of the Books database was derived from the OCR of the original paper books, it is primarily focused on the page organization and layout and not on the more common structuring elements such as “chapters” or “sections”. Because this emphasis on page layout goes all the way down to the individual word and its position on the page, there is a very large amount
160
R.R. Larson
of markup for page with content. For this year’s original version of the Books database, there are actually NO text nodes in the entire XML tree, the words actually present on a page are represented as attributes of an empty word tag in the XML. The entire document in XML form is typically multiple megabytes in size. A separate version of the Books database was made available that converted these empty tags back into text nodes for each line in the scanned text. This provided a significant reduction in the size of database, and made indexing much simpler. The primary index created for the full books was the “topic” index containing the entire book content. We also created page-level “documents” as we did last year. As noted above the Cheshire system permits parts of the document subtree to be treated as separate documents with their own separate indexes. Thus, paragraph-level components were extracted from the page-sized documents. Because unique object (page) level identifiers are included in each object, and these identifiers are simple extensions of the document (book) level identifier, we were able to use the pagelevel identifier to determine where in a given book-level document a particular page or paragraph occurs, and generate an appropriate XPath for it. Indexes were created to allow searching of full page contents, and component indexes for the full content of each of individual paragraphs on a page. Because of the physical layout based structure used by the Books collection, paragraphs split across pages are marked up (and therefore indexed) as two paragraphs. Indexes were also created to permit searching by object id, allowing search for specific individual pages, or ranges of pages. The system problems encountered last year have been (temporarily) corrected for this years submissions. Those problems were caused by the numbers of unique terms exceeding the capacity of the integers used to store them in the indexes. For this year, at least, moving to unsigned integers has provided a temporary fix for the problem but we will need to rethink how statistical summary information is handled in the future – perhaps moving to long integers, or even floating point numbers and evaluating the tradeoffs between precision in the statistics and index size (since moving to Longs could double index size).
4
INEX 2010 Book Track Runs
We submitted nine runs for the Book Search task of the Books track, As Table 5 shows, a small number of variations of algorithms and search elements were tried this year. The small number was largely due to some issues in indexing (due to a bug in page indexes that took a lot of time to locate and fix). With more that 16 million pages, response time was very good for the basic search operations, but slowed dramatically whenever data from records was needed. In Table 5 the first column is the run name (all of our official submissions had names beginning with “BOOKS10” which has been removed from the name), the second column is a short description of the run. We used only the main “fact” element of the topics in all of our runs. The third column shows which algorithms
Combining Page Scores for XML Book Retrieval
161
Table 5. Berkeley Submissions for the INEX Book Track 2009 Name T2FB BASE BST
Description Uses book-level topic index and blind feedback CLM PAGE SUM Uses page components and page words index CLM PAGE SUM 300 Uses page components and page words index T2 PAGE SUM 300 Uses page components and page words index
Algorithm Combined? TREC2 NA +BF CLM Sum CLM
Sum
TREC2
Sum
Table 6. Evaluation results for the INEX 2010 Best Books task Run ID p14-BOOKS2010 CLM PAGE SUM.trec p14-BOOKS2010 CLM PAGE SUM 300.trec p14-BOOKS2010 T2FB BASE BST.trec p14-BOOKS2010 T2 PAGE SUM 300.trec p6-inex10.book.fb.10.50.trec p6-inex10.book.trec p98-baseline 1.trec p98-baseline 1 wikifact.trec p98-baseline 2.trec p98-baseline 2 wikifact.trec p98-fact query 10wikibests.trec p98-fact query entropy.trec p98-fact query tfidfwiki.trec p98-fact query tfwiki.trec p98-fact stanford deps.trec
MAP 0.1507 0.1640 0.3981 0.5050 0.3087 0.3286 0.4374 0.4565 0.4806 0.5044 0.4328 0.4250 0.3442 0.4706 0.4573
P@10 NDCG@10 0.2714 0.2017 0.2810 0.2156 0.5048 0.5456 0.6667 0.6579 0.4286 0.3869 0.4429 0.4151 0.5810 0.5764 0.5905 0.5960 0.6143 0.6302 0.6381 0.6500 0.5714 0.5638 0.5476 0.5442 0.4667 0.4677 0.5571 0.5919 0.5857 0.5976
where used for the run, TREC2 is the TREC2 Logistic regression algorithm described above, “BF” means that blind relevance feedback was used in the run, and CLM means that the CLM algorithm described above (2) was used. Table 6 shows the official results for the Best Books task in the book retrieval tasks. The first four lines of Table 6 show the results for our official submitted runs described in Table 5 (as Mean Average Precision, Precision at 10, and Normalized Discounted Cumulative Gain at 10, the official measures used for the task). As these results show, the simplest methods are definitely not the best methods in the best book retrieval task. Our runs using simple coordination level matching both showed significantly worse results (note that the full result data for each participant was not available, and so we could not calculate true statistical significance. They certainly appear to be considerably worse than any of the other submitted runs). On the other hand, our baseline run (T2FB BASE BST), appears in the middle range of the score distribution, and our test run using the TREC 2 algorithm at the page level and simple summation of scores obtained the best results of any official run.
162
5
R.R. Larson
Conclusions and Future Directions
The results of the Books track were only recently made available to participants, but a few observations can be made about the runs. Our belief that the simple CLM on pages might outperform the TREC2 approach on pages (as suggested in our notebook paper) was wildly wrong, given how the CLM approaches were outperformed by every other approach tried this year. Perhaps this can be seen as an object lesson of the unreliability of “eyeballing” evaluation results in the absence of actual result data. We have known for some time that the base TREC 2 algorithm usually provides fairly high precision (and usually good recall too when combined with the blind feedback approach described above). This appears to be the main factor for the success in the Best Books task, high precision search at the page level, with book ranking scores based on a summation of page scores for the books with highly ranked pages. Of course, it is quite reasonable that the TREC2 algorithm outperforms the CLM approach. The CLM ranking value, as noted above is purely concerned with term occurrences in the text, while the TREC2 logistic regression algorithm attempts to estimate the probability that a document component is relevant for a given query. While a CLM value can be inflated by multiple occurrences of any given term (even very common terms), the TREC2 approach averages the contributions of the various terms, and therefore tends to favor pages with more query terms with higher IDF values. As noted by one reviewer, there is a question of why page-level search seems to capture more appropriate information than document (or book-level) search. I would suggest that this may be more an artifact of the types of questions/topics than necessarily a characteristic of book searching. Most of the topics are actually framed as “factoid” types of questions, most of which can be answered in one or a few pages, and which are unlikely to have full book-length treatments (although some of the particular people, places, or events making up parts of the questions might well have full books, which contain the pages that could answer the questions). Now that the relevance data for this task is available, we plan to test other approaches to merging page-level results, as well as other basic ranking algorithms for the page search component, including, naturally, the use of blind feedback at the page level. The reason that this latter was not included in the official runs was that we underestimated the time it would take to build the vector files needed for each page, and ran out of time to make the official deadline.
References 1. Chen, A.: Multilingual information retrieval using english and chinese queries. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF 2001. LNCS, vol. 2406, pp. 44–58. Springer, Heidelberg (2002) 2. Chen, A.: Cross-Language Retrieval Experiments at CLEF 2002. In: Peters, C., Braschler, M., Gonzalo, J. (eds.) CLEF 2002. LNCS, vol. 2785, pp. 28–48. Springer, Heidelberg (2003)
Combining Page Scores for XML Book Retrieval
163
3. Chen, A., Gey, F.C.: Multilingual information retrieval using machine translation, relevance feedback and decompounding. Information Retrieval 7, 149–182 (2004) 4. Cooper, W.S., Chen, A., Gey, F.C.: Full Text Retrieval based on Probabilistic Equations with Coefficients fitted by Logistic Regression. In: Text Retrieval Conference (TREC-2), pp. 57–66 (1994) 5. Cooper, W.S., Gey, F.C., Dabney, D.P.: Probabilistic retrieval based on staged logistic regression. In: 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Copenhagen, Denmark, June 21-24, pp. 198–210. ACM, New York (1992) 6. Harman, D.: Relevance feedback and other query modification techniques. In: Frakes, W., Baeza-Yates, R. (eds.) Information Retrieval: Data Structures & Algorithms, pp. 241–263. Prentice-Hall, Englewood Cliffs (1992) 7. Larson, R.R.: A logistic regression approach to distributed IR. In: SIGIR 2002: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland, August 11-15, pp. 399–400. ACM, New York (2002) 8. Larson, R.R.: A fusion approach to XML structured document retrieval. Information Retrieval 8, 601–629 (2005) 9. Larson, R.R.: Probabilistic retrieval, component fusion and blind feedback for XML retrieval. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 225–239. Springer, Heidelberg (2006) 10. Larson, R.R.: Ranking and fusion approaches for XML book retrieval. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2009. LNCS, vol. 6203, pp. 179–189. Springer, Heidelberg (2010) 11. Voorhees, E., Harman, D. (eds.): The Seventh Text Retrieval Conference (TREC7). NIST (1998) 12. Voorhees, E., Harman, D. (eds.): The Eighth Text Retrieval Conference (TREC-8). NIST (1999)
OUC’s Participation in the 2010 INEX Book Track Michael Preminger and Ragnar Nordlie Oslo University College
Abstract. In this article we describe the Oslo University College’s participation in the INEX 2010 Book track. The OUC has submitted retrieval results for the ”prove it” task with traditional relevance detection combined with some rudimental detection of confirmation. We call for a broader discussion of a more meaning-oriented (semantics-aware) approach to retrieval in digitized books, with the ”prove it” task (classifiable as a simple semantics- aware retrieval activity) providing the INEX milieu with a suitable context to start this discussion.
1
Introduction
In recent years large organizations like national libraries, as well as multinational organizations like Microsoft and Google have been investing labor, time and money in digitizing books. Beyond the preservation aspects of such digitization endeavors, they call on finding ways to exploit the newly available materials, and an important aspect of exploitation is book and passage retrieval. The INEX Book Track[1], which has been running since 2007, is an effort aiming to develop methods for retrieval in digitized books. One important aspect here is to test the limits of traditional methods of retrieval, designed for retrieval within ”documents” (such as news-wire), when applied to digitized books. One wishes to compare these methods to book-specific retrieval methods. One important mission of such retrieval is supporting the generation of new knowledge based on existing knowledge. The generation of new knowledge is closely related to access to – as well as faith in – existing knowledge. One important component of the latter is claims about facts. This year’s ”prove it” task, may be seen as challenging the most fundamental aspect of generating new knowledge, namely the establishment (or refutal) of factual claims encountered during research. On the surface, this may be seen as simple retrieval, but proving a fact is more than finding relevant documents. This type of retrieval requires from a passage to ”make a statement about” rather than ”be relevant to” a claim, which traditional retrieval is about. The questions we pose here are: – what is the difference between simply being relevant to a claim and expressing support for a claim – how do we modify traditional retrieval to reveal support or refutal of a claim? S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 164–170, 2011. c Springer-Verlag Berlin Heidelberg 2011
OUC’s Participation in the 2010 INEX Book Track
165
We see proving and denial of a statement as different tasks, both classifiable as semantics-aware retrieval, suspecting that the latter is a more complicated task. This paper attempts at applying some rudimentary techniques of detecting the confirmation (proving) of a statement. The rest of the paper discusses these tasks in the context of meaning-oriented retrieval in books.
2
Indexing and Retrieval Strategies
The point of departure of the strategies discussed here is that confirming or refuting a statement is a simple action of speech that does not require from the book (the context of the retrieved page) to be ABOUT the topic covering the fact. This means that we do not need the index to be context-faithful (pages need not be indexed in a relevant book context). It is more the formulation of the statement in the book or page that matters. This is why we need to look for words (or sequences of words) or sentences that indicate the stating of a fact. A simple strategy is looking for the occurrence of words like ”is”, ”are”, ”have”, ”has” a.s.o, that, in combination with nouns from the query (or fact formulation), indicate a possible act of confirming the fact in question. Further focus may be achieved by detecting sentences that include (WH-) question indicators or a question-mark and pruning these from the index, so that pages that only match the query through such sentences are omitted or weighed down during retrieval. Against this background we were trying to construct runs that emphasized pages that are confirmative in style. We attempted to divide the pages in the collection into categories of how confirmative they are, and indexed them individually (each page comprising a document). Occurrences of the words is, are, was, were, have, has were counted in each page, and a ratio between this sum and the total number of words in the page was calculated. Based on a sample of the pages, three levels were defined, so that pages belonging to each of the levels were assigned a tag, accordingly. It may be argued that a richer set of confirmation indicators could be applied. Our claim is that the selected words should function as style indicators, not as content indicators (the content catered for by the topic, i.e. the factual claim under scrutiny), and were therefore sufficient. A larger collection could incur noise. These tags then facilitated weighting pages differently (based on their proportion of confirmatory words) when retrieving candidates of confirming pages. Retrieval was performed using the Indri language model in the default mode, weighting the confirmatory pages differently, as indicated above. As the primary aim here is to try and compare traditional retrieval with ”prove it”, there was no particular reason to divert from the default. The 2010 tasks have been featuring a relatively large number of topics (83). As a new method was employed for collecting relevance assessments, only 21 of these queries were ready for evaluation at deadline time, and it is these queries that are used for retrieval.
166
3
M. Preminger and R. Nordlie
Runs and Results
Based on the index that we were constructing, we weighted relevant pages on two levels: pages that featured 1 percent or more confirmatory words, and pages that featured 3 percents or more confirmatory words (the latter including the former) were weighted double, quintuple (5x) and decuple (10x) the baseline. Our baseline was normal, non-weighted retrieval, as we would do for finding relevant pages. We were using the indri combine operation with no changes to the default setting (regarding smoothing, a.s.o). The analysis was carried out twice: once against the entire official qrel file (meaning that all assessed book pages judged either confirming/refuting – or merely relevant to – the statement-query are taken to be relevant (Figure 1). The second analysis was done against a filtered version of the official qrel file, featuring only the pages assessed as confirming/refuting (Figure 2). The purpose was to see if the rate of confirmatory words can be used as a ”prove it” indicator, given that the relevance assessments properly reveal pages that confirm the factual statement. Weighting retrieved pages that feature 1 percent or more confirmatory words does not seem to outperform the baseline at any weighting level, at any region of the precision recall curve (Subfigures 1(a) and 2(a)). The reason for that may be that quite many pages belong to this category. The weighting thus seems to hit somewhat randomly. An occurrence rate of confirmatory words of one percent seems not to discriminate ”proving” pages. The results are more interesting when restricting the weighting to pages that feature 3 percents or more confirmatory words. Here the results are different at the low recall and the high recall regions. Directing our attention to the low recall region first, we see in Figure 1(b) that both doubling the weight and, to a lesser extent quintupling it, slightly outperform the the baseline. The effect is a bit clearer when evaluating by pages assessed as confirming (Figure 2(b)). Here also decupling the weight given to pages with 3 percent or more of confirmatory words slightly outperforms the baseline. No treatment seems to outperform the baseline in the higher recall regions. Subject, of course, to a more thorough scrutiny of the result, this could indicate that collecting many books that prove a statement is not likely to be better supported by this approach than by traditional retrieval, whereas only finding very few such books (early hits) might benefit from it. The reason for that may be approached by looking at single relevant pages retrieved at the low and high recall regions. This kind of treatment was beyond the scope of the present paper. The value of pursuing it may be limited in light of the overall results. Looking at the ”prove it” task in terms of traditional retrieval, the temporary conclusion would be that the treatment experimented with here may be in the right direction, and further pursuit of it has some potential of good retrieval, particularly if it is the low recall region that is important (early hits). If the purpose is collecting as many books as possible as evidence for a claim then the approach does not seem as promising.
OUC’s Participation in the 2010 INEX Book Track
167
(a) Weighting relevant pages with 1 percent or more confirmatory words
(b) Weighting relevant pages with 3 percent or more confirmatory words Fig. 1. Precision-recall curves for detecting relevant pages. Baseline marked by solid lines in both subfigures.
168
M. Preminger and R. Nordlie
(a) Weighting relevant pages with 1 percent or more confirmatory words
(b) Weighting relevant pages with 3 percent or more confirmatory words Fig. 2. Precision-recall curves for detecting confirming (proving) pages. Baseline marked by solid lines in both subfigures.
OUC’s Participation in the 2010 INEX Book Track
4
169
Discussion
Utilizing digital books poses new challenges on information retrieval. The mere size of the book text poses both storage, performance and content related challenges as compared to texts of more moderate size. But the challenges are even greater if books are to be exploited not only for finding facts, but also to support exploitation of knowledge, identifying and analyzing ideas, a.s.o. For example, we suspect that confirming and refuting a factual statement, the Book Track 2010 ”prove it” task, both belong to a class of activities that extend the current scope of information retrieval. The notion of relevance is a well known challenge in IR [2]. We suspect that the ”prove it” notion is by no means simpler. Confirming a fact may have many facets, based on how complicated the fact is. A fact like: The ten tribes forming the northern kingdom of Israel (aka the ten lost tribes) disappeared after being driven to exile by the Assyrians, several hundreds years before Christ (topic 2010003) may be confirmed on several levels. Should all minor details be in place for the fact to be confirmed? What if the book states that it was the Babylonians, rather than the Assyrians who sent the tribes into exile, the rest of the details being in agreement with the statement: is the fact then confirmed? Moreover, detecting the refutal of a statement is arguably a totally different activity than detecting its confirmation. This poses challenges not only to mere retrieval, but also to its evaluation, at all levels. Even though such activities may be developed and refined using techniques from e.g. Question Answering[3], we suspect that employing semantics-aware retrieval [4,5], which is closely connected to the development of the Semantic Web [6] would be a more viable (and powerful) path to follow. Within the INEX Book track, the ”prove it” task can thus serve as a splendid start of a broader discussion around detecting meaning rather than only matching strings. Many projects under way are already using ontologies to aid in tagging texts of certain kinds (e.g. philosophical essays)[7] to indicate certain meaning, with the aim of supporting the analysis of these texts. Is this a viable task for the INEX Book track? Is it a viable path for information retrieval?
5
Conclusion
This article is an attempt to start a discussion about semantics-aware retrieval in the context of the INEX book track. Proving of factual statements is discussed in light of some rudimental retrieval experiments incorporating the detection of confirmation (proving) of statement. We also discuss the task of proving statement, raising the question whether it is classifiable as a semantics-aware retrieval task.
170
M. Preminger and R. Nordlie
References 1. Kazai, G., Doucet, A., Koolen, M., Landoni, M.: Overview of the INEX 2009 book track. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2009. LNCS, vol. 6203, pp. 145–159. Springer, Heidelberg (2010) 2. Mizzaro, S.: Relevance: The whole history. Journal of the American Society of Information Science 48(9), 810–832 (1997) 3. Voorhees, E.M.: The trec question answering track. Natural Language Engineering 7, 361–378 (2001) 4. Finin, T., Mayfield, J., A.J.R.S.C, Fink, C.: Information retrieval and the semantic web. In: Proc. 38th Int. Conf. on System Sciences, Digital Documents Track (The Semantic Web: The Goal of Web Intelligence), 5. Mayfield, J., Finin, T.: Information retrieval on the semantic web: Integrating inference and retrieval. In: SIGIR Workshop on the Semantic Web, Toronto, 6. Berners-Lee, T., Hendler, J., Lassila, O.: The semantic web. In: Scientific American (2001) 7. Z¨ ollner-Weber, A.: Ontologies and logic reasoning as tools in humanities? DHQ: Digital Humanities Quarterly 3(4) (2009)
Overview of the INEX 2010 Data Centric Track Andrew Trotman1 and Qiuyue Wang2 1
Department of Computer Science University of Otago Dunedin New Zealand 2 School of Information Renmin University of China Beijing China
Abstract. The INEX 2010 Data Centric Track is discussed. A dump of IMDb was used as the document collection, 28 topics were submitted, 36 runs were submitted by 8 institutes, and 26 topics were assessed. Most runs (all except 2) did not use the structure present in the topics; and consequently no improvement is yet seen by search engines that do so.
1
Introduction
2010 sees the introduction of the Data Centric Track at INEX. The results of INEX up-to and including 2009 showed that whole document retrieval was effective. This result was, perhaps, a consequence of the IEEE and Wikipedia collections used in the past. It is reasonable to assume that the Wikipedia will include a whole document result to almost any ad hoc query. In the Data Centric Track we ask: Can whole document retrieval approaches outperform focused retrieval on highly structured document collection? To answer this question a new highly structured collection was developed and made available for download from the INEX website. That collection was a snapshot of the IMDb taken early in 2010. Highly structured queries were solicited from participants. Together with the assessments, these form the new INEX Data Centric Collection. Most of the runs submitted to the track did not use the structure present in the topics. This is not surprising in a new track because participants are inclined to use their existing systems on a new collection before making modifications to it. Consequently the track has not yet seen improvements in precision from structure. It is hoped that in future years participating groups will prefer to conduct experiments using the structure present in the topics. The track has generated a topic set that can be used for training.
2
The Task
In its first year, the track focused on ad hoc retrieval from XML data. An XML document is typically modeled as a rooted, node-labeled tree. An answer to a keyword S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 171–181, 2011. © Springer-Verlag Berlin Heidelberg 2011
172
A. Trotman and Q. Wang
query was defined as a set of closely related nodes that are collectively relevant to the query. So each result could be specified as a collection of nodes from one or more XML documents that are related and collectively cover the relevant information1. The task was to return a ranked list of results estimated relevant to the user's information need. The content of the collections of nodes was not permitted to overlap. This is similar to the focused task in the ad hoc track, but using a data-centric XML collection and allowing the construction of a result (i.e. a collection of nodes) from different parts of a single document or even multiple documents.
3
INEX Data Centric Track Collection
3.1
Document Collection
The track used the IMDb data collection newly built from www.IMDb.com. It was converted from the plain text files (April 10, 2010) published on the IMDb web site. The plain text files were first loaded into a relational database by the Java Movie Database system 2 . Then the relational data are published as XML documents according to the DTDs. There are two kinds of objects in the IMDb data collection, movies and persons involved in movies, e.g. actors/actresses, directors, producers and so on. Each object is richly structured. For example, each movie has title, rating, directors, actors, plot, keywords, genres, release dates, trivia, etc.; and each person has name, birth date, biography, filmography, etc. Please refer to Appendix A and B for the movie DTD and person DTD respectively. Information about one movie or person is published in one XML file, thus each generated XML file represents a single object, i.e. a movie or person. In total, 4,418,102 XML files were generated, including 1,594,513 movies, 1,872,492 actors3, 129,137 directors who did not act in any movies, 178,117 producers who did not direct or act in any movies, and 643,843 other people involved in movies who did not produce or direct nor act in any movies. 3.2
Topics
Each participating group was asked to create a set of candidate topics, representative of a range of real user needs. Both Content Only (CO) and Content And Structure (CAS) variants of the information need were requested. In total 30 topics were submitted by 4 institutes (IRIT / SIG, Renmin University of China, Universidade Federal do Amazonas, and Universitat Pompeu Fabra). From these a total of 28 topics were selected based on uniqueness, preciseness, and being correctly formed. An example topic (2010001) is given in Fig. 1:
1
2 3
However, as it is unclear how to evaluate this the standard INEX metrics were eventually used. http://www.jmdb.de/ 21 of the actor files were empty and removed.
Overview of the INEX 2010 Data Centric Track
173
Yimou Zhang 2010 2009 //movie[about(.//director, "Yimou Zhang") and (about(.//releasedate, 2010) or about(.//releasedate, 2009))] I want to know the latest movies directed by Yimou Zhang. I am interested in all movies directed by Yimou Zhang, and I want to learn the latest movies he directed. Fig. 1. INEX 2010 Data Centric Track Topic 2010001
4
Submission Format
The required submission format was a variant of the familiar TREC format used by INEX, the so called TREC++ format. The following information was collected about each run: • • •
The participant ID of the submitting institute, Whether the query was constructed automatically or manually from the topic, Topic fields used (from: Title, CASTitle, Description, and Narrative),
A run was permitted to contain a maximum of 1000 results for each topic. A result consisted of one or more nodes from a single or multiple XML documents. A node is uniquely identified by its element path in the XML document tree. The standard TREC format is extended with one additional field for specifying each result node: Q0 Here: • • • • •
•
the first column is the topic number. the second column is the query number within that topic (unused and should always be Q0). the third column is the file name (without .xml) from which a result node is retrieved. the fourth column is the rank of the result. Note that a result may consist of one or more related nodes, so there can be multiple rows with the same rank if these nodes belong to the same result. the fifth column shows the score that generated the ranking. This score must be in descending (non-increasing) order and is important to include so that assessment tools can handle tied scores (for a given run) in a uniform fashion (the evaluation routines rank documents from these scores, not from ranks). If you want the precise ranking that you submit to be evaluated, the scores should reflect that ranking. the sixth column is called the "run tag" and should be a unique identifier from within a participating group. It should also include a brief detail of the method used. The run tags contained 12 or fewer letters and numbers, with no punctuation.
174
A. Trotman and Q. Wang
•
the seventh column gives the element path of the result node. Element paths are given in XPath syntax. To be more precise, only fully specified paths are allowed, as described by the following grammar:
Path ::= ElementNode ::= AttributeNode Index ::=
'/' ElementNode Path | '/' ElementNode | '/' AttributeNode ElementName Index ::= '@' AttributeName '[' integer ']'
For Example the path /article[1]/bdy[1]/sec[1]/p[1] identifies the element which can be found if we start at the document root, select the first article element, then within that, select the first body element, within which we select the first section element, and finally within that element we select the first p element. An example submission is: 1 Q0 9996 1 0.9999 I09UniXRun1 /article[1]/bdy[1]/sec[1] 1 Q0 9996 1 0.9999 I09UniXRun1 /article[1]/bdy[1]/sec[2]/p[1] 1 Q0 9888 1 0.9999 I09UniXRun1 /article[1]/bdy[1]/sec[3] 1 Q0 9997 2 0.9998 I09UniXRun1 /article[1]/bdy[1]/sec[2] 1 Q0 9989 3 0.9997 I09UniXRun1 /article[1]/bdy[1]/sec[3]/p[1] Here there are three results. The first result contains the first section and first paragraph of the second section from 9996.xml, and the third section from 9888.xml. The second result only consists of the second section in 9997.xml, and the third result consists of the first paragraph of the third section from 9989.xml.
5
Submitted Runs
Participants were permitted to submit up to 10 runs. Each run was permitted to contain a maximum of 1000 results per topic, ordered by decreasing value of relevance. Runs were permitted to use any fields of the topics, but only runs using either the , or , or a combination of them were regarded as truly automatic. The results of one run was contained in one submission file and so up to 10 files per group could be submitted. In total 36 runs were submitted by 8 institutes. Those institutes were: Benemérita Universidad Autónoma de Puebla, Indian Statistical Institute, Kasetsart University, Peking University, Renmin University of China, Universidade Federal do Amazonas, Universitat Pompeu Fabra, and the University of Otago. Only 29 runs were assessed since other runs were submitted after the deadline. Of note is that more runs were submitted than topics, and more institutes submitted runs than that submitted topics. This suggests an increase in interest in the track throughout the year.
6
Assessment and Evaluation
Shlomo Geva ported the tool to work with the IMDb collection and in doing so identified some problems with the document collection. The collection was,
Overview of the INEX 2010 Data Centric Track
175
consequently, cleaned for use with the assessment tool. The new collection will most likely be used in 2011, if the track continues. Assessment was done by those groups that submitted runs. In total 26 of the 28 topics were assessed. Topics 2010003 and 20100013 were not assessed, all others were. The evaluation results presented herein were computed using just the 26 assessed topics with the other 2 topics dropped from the runs. Jaap Kamps used the (unmodified) INEX and TREC evaluation tools on the runs. The TREC MAP metric was used to measure the performance of the runs at whole document retrieval. The INEX thorough retrieval MAiP metric and the INEX Relevant-in-Context MAgP T2I(300) metrics were used to measure Focused Retrieval4. Although the run submission permitted the use of aggregated retrieval, it has not yet become clear how to measure aggregation and so the track organisers chose to fall-back to more traditional measures for 2010. Descriptions of the INEX and TREC measures are not given herein as they are well known and described elsewhere (see the ad hoc track overview paper pre-proceedings).
Fig. 2. Best runs measured with MAP
7
Results
The performance of the runs using the whole document based MAP metric are presented in Fig. 2. The best run, SIGMACLOUD01 was submitted by Peking University and performed substantially better than the next best run at all recall 4
See the ad hoc track overview paper (in this volume) for details on the metrics.
176
A. Trotman and Q. Wang
points. We note that this run used the description and narrative of the topic whereas the other runs did not (formally it is not an INEX automatic run and must be considered a manual run). The runs from Benemérita Universidad Autónoma de Puebla used the castitle and all other runs used the title. That is, despite being data centric, most runs did not use structure in ranking.
Fig. 3. Best runs measured with MAgP T2I(300)
Fig. 4. Best runs measured with MAiP
Overview of the INEX 2010 Data Centric Track
177
From visual inspection, there is little difference between the next three runs. The Otago run (that placed 3rd amongst the automatic runs) is a whole document run generated from the title of the topic by using the BM25 ranking function trained on the INEX 2009 document collection – it is equivalent to the ad hoc reference run. It can be considered a baseline for performance. When measured using the MAgP T2I(300) metric (see Fig. 3 the Otago referencelike run performs best, however there is a cluster of 3 runs performing at about (from visual inspection) the same level. When measured using MAiP (Fig. 4) the referencelike run shows high early precision but quickly decreases. Of course, whole document retrieval is not a good strategy for thorough retrieval because precisely 1 element is returned per document. Those runs that exhibited overlap were not evaluated using the MAgP metric.
8
Conclusions
The track has successfully produced a highly structured document collection including structured documents (IMDb), structured queries, and assessments. The participants of the track submitted runs and those runs were evaluated. Because most runs did not use structured queries no claim can be made about the advantage of doing so. This is expected to change in future years. The track was overly ambitions in allowing result aggregation. No method of measuring the performance of aggregated retrieval was developed for the track in 2010 and is left for future years. Acknowledgements. Thanks are given to the participants who submitted the topics, the run, and performed the assessment process. Special thanks go to Shlomo Geva for porting the assessment tools, and to Jaap Kamps for performing the evaluation. Finally, some of the contents of this paper was taken from the INEX web site which was authored by many people – we thank each of those for their contribution to the text in this paper. Qiuyue Wang is supported by the 863 High Tech. Project of China under Grant No. 2009AA01Z149.
178
A. Trotman and Q. Wang
Appendix A: Movie DTD
Overview of the INEX 2010 Data Centric Track
179
180
A. Trotman and Q. Wang
Appendix B: Person DTD
Overview of the INEX 2010 Data Centric Track
181
DCU and ISI@INEX 2010: Adhoc and Data-Centric Tracks Debasis Ganguly1 , Johannes Leveling1 , Gareth J.F. Jones1 Sauparna Palchowdhury2, Sukomal Pal2 , and Mandar Mitra2 1
CNGL, School of Computing, Dublin City University, Dublin, Ireland 2 CVPR Unit, Indian Statistical Institute, Kolkata, India {dganguly,jleveling,gjones}@computing.dcu.ie, [email protected], {sukomal r,mandar}@isical.ac.in
Abstract. We describe the participation of Dublin City University (DCU) and Indian Statistical Institute (ISI) in INEX 2010 for the Adhoc and Data Centric tracks. The main contributions of this paper are: i) a simplified version of Hierarchical Language Model (HLM), which involves scoring XML elements with a combined probability of generating the given query from itself and the top level articl node, is shown to outperform the baselines of LM and VSM scoring of XML elements; ii) the Expectation Maximization (EM) feedback in LM is shown to be the most effective on the domain specific collection of IMDB; iii) automated removal of sentences indicating aspects of irrelevance from the narratives of INEX ad hoc topics is shown to improve retrieval effectiveness.
1
Introduction
Traditional Information Retrieval systems return whole documents in response to queries, but the challenge in XML retrieval is to return the most relevant parts of XML documents which meet the given information need. Since INEX 2007 [1] arbitrary passages are also permitted as retrievable units, besides XML elements. A retrieved passage can be a sequence of textual content either from within an element or spanning a range of elements. INEX 2010 saw the introduction of the restricted versions of the “Focused” task which has been designed particularly for displaying results on a mobile device with limited screen size. The Adhoc track tasks comprises of the following tasks: a) the “Restricted Focused” task which asks systems to return a ranked list of elements or passages to the user restricted to at most 1000 characters per topic; b) the (un)restricted “Relevant in Context” task which asks systems to return relevant elements or passages grouped by article, a limit of at most 500 characters being imposed on the restricted version; and c) the “Efficiency” task which expects systems to return thorough article level runs. In addition, we participated in the new “Data Centric track” which is similar to ad hoc retrieval of elements or passages on a domain specific collection of IMDB movie pages. In INEX 2010 we submitted 9 ad hoc focused runs, (3 for each Focused task) and 3 Thorough runs for the Ad Hoc track. In addition we submitted 10 runs for the Data Centric task. S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 182–193, 2011. c Springer-Verlag Berlin Heidelberg 2011
DCU and ISI@INEX 2010: Adhoc and Data-Centric Tracks
183
The remainder of this paper is organized as follows: Section 2 elaborates on the approaches to indexing and retrieval of whole documents followed by Section 3, which describes the strategy for measuring the similarities of the individual XML elements to the query. In Section 4 we propose a simplified version of HLM for XML retrieval, which involves scoring an XML element with a linear combination of the probability of generating the given query from itself and its parent article. Using INEX 2009 topic set for training and 2010 topic set for testing, we show that it outperforms the standard LM and VSM method of scoring of XML elements. Section 5 explores the effectiveness of Blind Relevance Feedback (BRF) on the domain specific collection of movie database. Section 6 describes the work of automatically identifying and removing negation from the verbose ‘narrative’ sections of the INEX queries, and observing how retrieval effectiveness improves. The entire topic (TDN ) is treated as the query in this experiment. Section 7 concludes the paper with directions for future work.
2
Document Retrieval
2.1
Preprocessing
Similar to INEX 2009, for extracting useful parts of documents, we shortlisted about thirty tags that contain useful information:
, , , etc. [2]. Documents were parsed using the libxml2 parser, and only the textual portions included within the selected tags were used for indexing. Similarly, for the topics, we considered only the title and description fields for indexing, and discarded the inex-topic, castitle and narrative tags. No structural information from either the queries or the documents was used. The extracted portions of the documents were indexed using single terms and a controlled vocabulary (or pre-defined set) of statistical phrases employing the SMART1 system. Words in the standard stop-word list included within SMART were removed from both documents and queries. The default stemmer implementation of SMART which is a variation of the Lovin’s stemmer [3] was used. Frequently occurring word bi-grams (loosely referred to as phrases) were also used as indexing units. We used the N-gram Statistics Package 2 (NSP) on the English Wikipedia text corpus and selected the 100,000 most frequent word bi-grams as the list of candidate phrases. 2.2
Language Model (LM) Term Weighting
Our retrieval method is based on the Language Modeling approach proposed by Hiemstra [4]. In this Subsection we summarize the Language Modeling method to IR used by us for document retrieval. In LM based IR, a document d is ranked by a linear combination of estimated probabilities P (q|d) of generating a query q from the document d and P (ti ) of generating the term from the collection. The 1 2
ftp://ftp.cs.cornell.edu/pub/smart/ http://www.d.umn.edu/∼tpederse/nsp.html
184
D. Ganguly et al.
document is modelled to choose q = {t1 , t2 . . . tn } as a sequence of independent words as proposed by Hiemstra [4]. P (q|d) = P (d) log P (d)
n
λi P (ti |d) i=1 n
log P (q|d) = log P (d) +
log(1 +
i=1
+ (1 − λi )P (ti )
(1)
λi P (ti |d) ) 1 − λi P (ti )
(2)
P (d) is the prior probability of relevance of a document d and it is typically assumed that longer documents have higher probability of relevance. The term weighting equation can be derived from Equation 1 by dividing it with (1 − λi )P (ti ) and taking logarithm on both sides to convert the product to summation. This trasnformation also ensures that the computed similarities between documents and a given query are always positive. We index each query vector λk q as qk = tf (tk ) and each document vector d as dk = log(1 + PP(t(tkk|d) ) 1−λk ), so that the dot product d · q gives the likelihood of generating q from d and hence can be used as the similarity score to rank the documents.
3
XML Element Retrieval
For the element-level retrieval, we adopted a 2-pass strategy. In the first pass, we retrieved 1500 documents (since the thorough runs for INEX are required to report at most 1500 documents per topic) for each query using the LM retrieval method as described in the previous section 2.2. In the second pass, these documents were parsed using the libxml2 parser, and leaf nodes having textual content were identified. The total set of such leaf-level textual elements obtained from the 1500 top-ranked documents were then indexed and compared to the query as before to obtain the final list of 1500 retrieved elements. The preprocessing steps are similar to those as described in Section 2.1. The following section provides the details of our proposed method of scoring XML elements. 3.1
Simplified Hierarchical Language Model
Motivation. The objective of the focused task is to retrieve short relevant chunks of information. Scoring an XML element by its similarity with the query may retrieve short XML elements such as a small paragraph or sub-sections with a dense distribution of query terms in the top ranks. But there is an implicit risk associated with the retrieval of short high scoring elements namely that the text described in the short element might be a digression from the main topic of the parent article. Thus it is unlikely that the retrieved text from the XML element would serve any useful purpose to the searcher because it would not have the necessary context information to do so. As an example consider the artificial paragraph of text as shown in Fig. 1. Now let us imagine that a searcher’s query is “top-10 IMDB movies actors”. Let us also imagine that this paper has been converted into an XML formatted document and put into
DCU and ISI@INEX 2010: Adhoc and Data-Centric Tracks
We crawled the
IMDB
movie
185
collection and categorized the
crawled IMDB data into categories such as movies , actors etc. Movie reviews and ratings were also stored. Fig. 1. An artificial paragraph of text to illustrate the implicit risk of out-of-context retrieval with response to a query “top-10 IMDB movies actors”
the INEX document collection. The contents of the above paragraph thus could be retrieved at a top rank because of its high similarity due to the presence of three matching query terms as highlighted with boxes. But the searcher was certainly not looking for a technical paragraph on indexing the IMDB collection. This shows that the retrieval model for XML elements needs to be extended to take into consideration the distribution of query terms in the parent articles in addition to the element itself. If the query terms are too sparse in the parent article, then it is more likely that the XML element itself is an out-of-context paragraph e.g. the highlighted words in Fig. 1 are very sparsely distributed throughout the rest of this documentm, which suggests that the shown artificial paragraph is not a good candidate for retrieval. It is also desirable to retrieve an element with a few query terms if the other missing terms are abundant in the article. This is particularly helpful for assigning high scores to elements (sections or paragraphs) which densely discusses a single sub-topic (the sub-topic typically being one specific facet of the user information need) whereas the rest of the article throws light on the other general facets hence providing the necessary context for the specific subtopic. The Scoring Function. An extension of LM for Field Search named Field Language Model (FLM) was proposed by Hiemstra [4] which involves scoring a document according to the probability of generation of the query terms either from the document itself as a whole, or from a particular field of it (e.g title) or from the collection. We propose to assign a score to the constituent element itself from the parent article evidence thus differing from FLM in the directionality of assignment of the scores. We use Equation 3 to score an XML element e. P (q|e) = log P (d)
n
μi P (ti |e) + λi P (ti |d) + (1 − λi − μi )P (ti )
(3)
i=1
The parameter λi denotes the probability of choosing ti from the parent article of the element e, whereas μi denotes the probability of choosing ti from the element text. The residual event involves choosing ti from the collection. Thus even if a query term ti is not present in the element a non zero probability of generation is contributed to the product. Two levels of smoothing are employed in this case. Ogilvie and Callan developed the general HLM which involves two way propagation of the LM element scores from the root to the individual leaf nodes and vice-versa [6]. Our model is much simpler in the sense that we use only
186
D. Ganguly et al.
the current node and the top level article node for the score computation. We call this method Simplified Hierarchical Language Model (SHLM) because we restrict our smoothing choice to the root article element only in addition to the collection. The SHLM equation can be further simplified by using λi = λ∧μi = μ ∀i = 1 . . . n. It can be seen that Equation 3 addresses the motivational requirements as described in the previous section in the following ways: a) An element e1 which has a query term t only in itself but not anywhere else in the top level article, scores lower than an element e2 which has the term present both in itself and somewhere else in the article. Thus the model favours elements with some pre-defined contextual information about the query terms over individual snippets of information which do not have any associated context. b) An element with a few of the given query terms might is retrieved at a high rank if the missing terms are abundant in the article. The experimental evaluations of SHLM for adhoc XML retrieval are provided in Section 4.2.
4 4.1
Adhoc Track Experiments and Results Thorough Task
We trained our system using the INEX 2009 topic set. All the initial article level runs were performed using LM retrieved as described in Equation 1. We assign λi = λ ∀i = 1 . . . n and also assigned uniform prior probabilities to the documents. After a series of experiments by varying λ we found that best retrieval results are obtained with λ = 0.4 and henceforth we this setting for all the article level LM runs reported in the paper. We officially submitted three article level runs containing 15, 150 and 1500 documents as thorough runs. On getting unexpectedly low retrieval precision, we conducted a posthoc analysis after the INEX results were officially out. We found that there was a bug in our retrieval control-flow where we used inverted lists constituted from the raw document vectors instead of the LM weighted ones. The results for the thorough runs, along with post-submission corrected versions are shown in Table 1.
Table 1. Official evaluation of the thorough runs Run Id
# docs retrieved
Submitted MAP
ISI2010 thorough.1500 ISI2010 thorough.150 ISI2010 thorough.15
1500 150 15
MAiP
0.0431 0.0846 0.0309 0.0826 0.0110 0.0714
Corrected MAP
MAiP
0.1539 0.1750 0.1185 0.1421 0.0647 0.0930
DCU and ISI@INEX 2010: Adhoc and Data-Centric Tracks
4.2
187
Focused Task
We submitted 3 element level runs for the restricted focused task. The first two subsections report the training of SHLM on the INEX 2009 topics and the last section reports the official submissions under the restricted focused task. Tuning SHLM. To test the effectiveness of SHLM (as outlined in Section 3.1) for element retrieval, we run SHLM with different combinations of λ and μ (simplifying Equation 3 by employing λi = λ ∧ μi = μ ∀i = 1 . . . n) on the INEX 2009 training topics. A revisit of Equation 3 suggests that a higher value of μ as compared to λ attaches too much importance on the presence of the query terms. While this might be good for queries with highly correlated terms, typically user queries are faceted, each term representing one such facet. It is highly unlikely that a single section or paragraph would cover all the facets. The more likely situation is that a small paragraph would cover one facet of the user’s information need whereas the other facets are covered somewhere else in the document. Our hypothesis is that a value of μ lower than λ ensures that we lay emphasis on not retrieving out-of-context small elements and we do expect better retrieval performance for the setting μ < λ. Another critical issue to explore in the model of Equation 3 is the issue of assigning prior probabilities to the elements. Singhal [7] analyzes the likelihood of relevance against the length of TREC documents and reports that longer documents have a higher probability of relevance. While this scheme of assigning document prior probabilities proportional to their lengths suits the traditional adhoc retrieval of documents (the retrieval units being whole documents) from the news genre, for a more flexible retrieval scenario such as the Restricted Focused INEX task where retrieval units can be arbitrary passages and shorter passages are favoured over longer ones, it might be worth trying to assign prior probabilities to elements inversely proportional to their lengths. SHLM Results. As baseline we use standard LM scoring of the elements which is a special case of SHLM obtained by setting λ = 0. To verify our hypothesis that λ should be higher than μ, we ran two versions of SHLM one with λ < μ and the other μ > λ. Table 2 reports the measured retrieval effectiveness of the different cases and also shows the effect on precision for the three different modes of element priors - i) uniform, ii) proportional and iii) inversely proportional for the case μ < λ. Table 2 provides empirical evidence to the hypothesis that elements when scored with contextual information from their parent article yield better retrieval results. The first row of the table reports the case where elements are LM weighted without any parent article information. It can be seen that the first row yields the least iP [0.01] value. The table also justifies the hypothesis of assigning μ < λ since iP [0.01] of the third and fifth rows are higher than that of the second row. Official Results. The “Restricted Focused” task required systems to return a ranked list of elements or passages restricted to at most 1000 characters per topic.
188
D. Ganguly et al.
Table 2. SHLM for element retrieval for INEX 2009 topics λ
μ
0.0 0.15 0.25 0.25 0.25
0.15 0.25 0.15 0.15 0.15
Element Prior
Retrieval Effectiveness
probability
iP[0.01] iP[0.05] iP[0.10] MAiP
Uniform 0.2639 0.1863 0.1335 0.0448 Uniform 0.4082 0.2648 0.1894 0.0566 Uniform 0.5256 0.3595 0.2700 0.0991 Shorter favored 0.3459 0.1682 0.0901 0.0314 Longer favored 0.4424 0.3582 0.2787 0.1064
The evaluation metric used was P@500 characters. Since this metric favours retrieval runs with a high precision at low recall levels (recall is expected to be low when only 1000 characters are retrieved), we use the settings as reported in the third row of Table 2 i.e. with the settings (λ, μ) = (0.25, 0.15) with uniform element prior probability. We perform SHLM element retrieval on i) our best performing LM retrieved article level run (λ = 0.4) and ii) reference BM25 run provided by the INEX organizers. To show that SHLM performs better than the pivoted length normalized scoring which was our element level retrieval strategy for INEX 2009 [2], we also submitted a run scoring the elements by Equation 4. normalization = 1 +
slope #unique terms · (1 − slope) pivot
(4)
Table 3 shows that the corrected SHLM based element retrieval on the reference run yields the highest character-precision among our runs. We also see that the best character precision we achieved ranks third considering the list of official submissions (after LIP6-OWPCparentFo and DURF10SIXF). SHLM clearly outperforms pivoted normalized element retrieval on the same document level run showing that given the same document level run, it is more effective than VSM based element retrieval. Table 3 also shows that SHLM outputs a better element level run for a better input article run as evident from Table 3. Official evaluation of the Focused runs Run Id ISI2010 rfcs ref ISI2010 rfcs flm
ISI2010 rfcs vsm
Methodology SHLM element retrieval on article level reference run SHLM element retrieval (μ < λ and uniform prior of the elements) on article level LM run Pivoted normalized element retrieval on article level LM run
LIP6-OWPCparentFo (Best run)
P@500 chars submitted corrected 0.2451
0.3755
0.2151
0.2841
0.1289
0.2343
0.4125
DCU and ISI@INEX 2010: Adhoc and Data-Centric Tracks
189
the first and second rows (MAP of the reference run is higher than our LM based article run).
5
Data Centric Track Experiments and Results
We indexed the IMDB collection using SMART in a similar way as outlined in Section 2, the only difference being we did not use pre-extracted list of commonly occuring bigrams for constructing the phrase index. Our Data Centric official submissions also suffered from the same bug as had been the case with our Adhoc submissions. In this section we report a set of refined results after carrying out experiments with a corrected version. Our approach in this track comprises of exploring the standard IR techniques on a domain specific collection like the movie database. Our initial experiments show that we get the optimal baseline MAP by using λ = 0.6. While performing feedback experiments we found that a selection of query expansion terms by the LM score [8] results in worse retrieval performance. Fig. 2a shows the effect of query expansion with different settings for R (the number of pseudo-relevant documents used) and t (the number of terms used for query expansion) on MAP. We implemented the EM feedback in SMART as proposed by Hiemstra [4] where each λi , associated with the query term ti , is modified aiming to maximize the expectation of the LM term generation probability from the top ranked pseudorelevant documents as shown in Equation 5. λp+1 = i
R
1 λpi P (ti |Dj ) R j=1 λ(p) P (ti |Dj ) + (1 − λ(p) )P (ti ) i
(5)
i
We use only one iteration of the feedback step i.e. we calculate λ1i s from the initial λ0i = λ for each i. We also tried out applying EM to an expanded query with additional terms but we found out that it did not improve the MAP. The results as shown in Figure 2b reveal that EM performs better than the LM term based expansion as shown in Figure 2a. While doing a per-topic analysis of the document retrieval for the IMDB collection, we made the following interesting observation. Query 2010015 reads “May the force be with you” of which all are stopwords except the word force. As a result the obtained MAP for this query is 0. Interestingly, when the IMDB search is fed with the same query, it also returns documents with the term force in it but not with all the other words present which suggests that IMDB retrieval is also non-phrase based. A characteristic of the IMDB collection is that the documents are grouped into categories such as movies, actors etc. To find if relevance is biased towrads one of the categories, we computed the percentage of relevant documents in each of the categories from the article level manual assessments. We also computed the percentage of retrieved documents in each of category. Fig. 3 shows that relevance is heavily biased to the movie documents suggesting that the searchers mostly seek movie pages in response to a submitted query. For the retrieved set we find that movie pages are retrieved highest in number followed by actor pages and so on. The relative ranks of the number of hits for the relevant and
D. Ganguly et al.
0.34 0.32 0.30 0.28 0.26 0.24 0.22 0.20 0.18 0.16
R=5 R=10 R=15 R=20 No fdbk. MAP
MAP
190
5
10
15
20
25
0.34 0.32 0.30 0.28 0.26 0.24 0.22 0.20 0.18 0.16
30
EM fdbk. No fdbk. 5
10
15
t
20
25
30
R
(a) LM score based query expansion
(b) EM
Fig. 2. Feedback effects on the IMDB collection
er
s
s
er
uc
rs
to
th
O
Ac
od
Pr
s
or
ct
ire
D
ie
ov
M
90 retrieved categories are almost Relevant 80 Retrieved the same with an exception for the 70 categories of producers and direc60 tors where producer pages have 50 the least number of hits in the rel40 evant set whereas this category is 30 not with least number of hits for 20 10 the retrieved set. Although rela0 tive ranks are similar, there is a noticeable difference in the percentages between the two sets, especially in the categories movies Fig. 3. Percentage of relevant and retrieved and actors which suggests that documents in the different categories adjusting the prior relevance P (d) (Equation 1) not according to the length of a document but according to its category could be a way to improve on the retrieval effectiveness. This would help to reduce the percentage gaps in the relevant and retrieved sets.
6
Using Negative Information Present in Verbose Queries
Motivation. It is easier for a user to formulate a complex information need in natural language rather than using terse keyword queries. The INEX topics have a detailed narrative (N ) section that reinforces the information expressed in the title (T ) and description (D ) fields. Notably, in the narrative section, the topic creator specifies what he is not looking for. One such INEX query is shown in Table 4. The emphasized sentence of N is the negative information. To support our claim that such negative information can introduce a query drift towards non-relevant documents, we report a comparison of retrieval results, using queries processed using manual and automatic methods. Results show that the modified queries, with negation removed, yield higher retrieval effectiveness.
DCU and ISI@INEX 2010: Adhoc and Data-Centric Tracks
191
Table 4. INEX 2009 Topic No. 80 with negative information international game show formats I want to know about all the game show formats that have adaptations in different countries. Any content describing game show formats with international adaptations are relevant. National game shows and articles about the players and producers are not interesting. Table 5. Classifier performance Test set 2008 2009 2010
Training set 2007 2008 2009
# of training sentences Accuracy 589 90.4% 679 89.1% 516 93.8%
Approach. Our unmodified set of queries is Q. From Q we create two new sets. The first one, P , consists of only those queries in Q which have negation, and, with these negative sentences or phrases removed. (P , stands for ‘positive’ in the context of this experiment). The second set, PM (‘positive’, ‘machine’processed set), is similar, but negation was now automatically identified using a Maximum Entropy Classifier (Stanford Classifier) 3 , and removed. In Table 6 the cardinalities of P and PM differ because the classifier does not identify all the negative phrases and sentences with full accuracy. Some queries in Q, which have negation, and can be found in P , may not make it to PM . We did retrieval runs using Q, P and PM and noted the change in MAP (Refer to [9] for more deatils). The classifier performed well with accuracies around 90% (Table 5). Retrieval Results. We used the SMART retrieval system to index the document collection using pivoted document length normalized VSM (Equation 4), and the initial retrieval run was followed by a BRF run employing Rocchio feedback. For feedback we used the most frequent 20 terms and 5 phrases occurring in the top 20 pseudo-relevant documents setting (α, β, γ) = (4, 4, 2). Table 6 that the positive sets show an improvement in all cases. Of particular interest is the PM results; the P results are included only to refer to the maximum relative gain in performance that is achievable. Although, as expected, the relative gains for the PM set is lower as compared to the P set, the differences between the two relative gains are not too big, which shows that the automated process does well. The Wilcoxon test verifies that the differences in the relative gains of Q and P are statistically significant, corroborating the fact that removal of negation 3
http://nlp.stanford.edu/software/classifier.shtml
192
D. Ganguly et al.
Table 6. Comparison of performance of the manually processed (P ) and automatically processed (PM ) positive query sets Topic Set
Manually processed |P| MAPQ
Automatically processed
MAPP change |PM | MAPQ
INEX 2008 44 0.2706 0.2818 4.1% INEX 2009 36 0.2424 0.2561 5.7% INEX 2010 26 0.3025 0.3204 6.0%
MAPPM change
31 0.2638 0.2748 4.2% 30 0.2573 0.2581 0.3% 20 0.2922 0.2983 2.1%
Table 7. MAP values for retrieval using increasing query size INEX year 2008 2009 2010
T 0.2756 0.2613 0.2408
TD 0.2815 0.2612 0.2406
Δ (%) 2.14 -0.03 -0.08
TDN 0.2998 0.2547 0.2641
Δ (%) 8.78 -2.52 9.67
trend (Δ, Δ ≥ 5%) −↑ −− −↑
improves performance. Also, a test of significance between P and PM shows that their difference is statistically insignificant showing that the automated process is as good as the manual one. One must note that our baseline comprises retrieval runs over the set Q where the queries are of maximum length. The queries in P and PM , are shorter. It is expected that the retrieval effectiveness will improve with an increase in query size for the bag-of-words approach. We needed this to be empirically verified to rule out the possibility of an improvement in the retrieval effectiveness due to query length shortening. Three retrieval runs were done using T, TD and TDN. The results shown in Table 7 shows that there is a positive correlation between the MAP and query length. This eliminates the possibility of an improvement in MAP due to a negative correlation between query length and MAP for INEX topics. We also observe that the results for the 2009 set degrades across T, TD and TDN.
7
Conclusions and Future Work
Through our participation in the adhoc track of INEX 2010, we wanted to explore an extension of LM for IR which we call SHLM as a new element retrieval strategy. Trial experiments on INEX 2009 topics show that it outperforms the baseline LM element retrieval of the elements. Official restricted focused runs show that SHLM element retrieval outperforms the pivoted normalized VSM element retrieval. Also our corrected official focused runs ranks third among the submitted runs. The concept of SHLM can be extended to arbitrary passages by defining a series of fixed length window-subwindow structures. For the data-centric track, we have shown that LM retrieval works well on a domain specific collection such as the movie database. We also show that query expansion using terms selected by LM scores do not improve retrieval
DCU and ISI@INEX 2010: Adhoc and Data-Centric Tracks
193
effectiveness whereas the EM feedback does well on this collection. We report a bias of relevance towards the movie and actor categories which suggests a possible future work of assigning higher prior probabilities of relevance for documents in these categories to help improve MAP. Using the ad hoc track topics, we show that it is possible to automate the process of removing negative sentences and phrases and this improves retrieval effectiveness. Future work may involve detection of sub-sentence level negation patterns and handling complex negation phrases so as to prevent loss of keywords. Acknowledgments. This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (CNGL) project.
References 1. Kamps, J., Geva, S., Trotman, A., Woodley, A., Koolen, M.: Overview of the INEX 2008 ad hoc track. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2008. LNCS, vol. 5631, pp. 1–28. Springer, Heidelberg (2009) 2. Pal, S., Mitra, M., Ganguly, D.: Parameter tuning in pivoted normalization for XML retrieval: ISI@INEX09 adhoc focused task. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2009. LNCS, vol. 6203, pp. 112–121. Springer, Heidelberg (2010) 3. Lovins, J.B.: Development of a stemming algorithm. Mechanical Translation and Computational Linguistics 11, 22–31 (1968) 4. Hiemstra, D.: Using language models for information retrieval. PhD thesis, University of Twente (2001) 5. Sigurbjrnsson, B., Kamps, J., de Rijke, M.: An element-based approach to xml retrieval. In: INEX 2003 Workshop Proceedings (2004) 6. Ogilvie, P., Callan, J.: Hierarchical language models for XML component retrieval. In: Fuhr, N., Lalmas, M., Malik, S., Szl´ avik, Z. (eds.) INEX 2004. LNCS, vol. 3493, pp. 224–237. Springer, Heidelberg (2005) 7. Singhal, A.: Term Weighting Revisited. PhD thesis, Cornell University (1996) 8. Ponte, J.M.: A language modeling approach to information retrieval. PhD thesis, University of Massachusetts (1998) 9. Palchowdhury, S., Pal, S., Mitra, M.: Using negative information in search. In: Proc. 2011 Second International Conference on Emerging Applications of Information Technology (EAIT 2011), pp. 53–56 (2011)
Automatically Generating Structured Queries in XML Keyword Search Felipe da C. Hummel1 , Altigran S. da Silva1 , Mirella M. Moro2 , and Alberto H.F. Laender2 1
Departamento de Ciˆencia da Computac¸a˜ o Universidade Federal do Amazonas Manaus, Brazil {fch,alti}@dcc.ufam.edu.br 2 Departamento de Ciˆencia da Computac¸a˜ o Universidade Federal de Minas Gerais Belo Horizonte, Brazil {mirella,laender}@dcc.ufmg.br
Abstract. In this paper, we present a novel method for automatically deriving structured XML queries from keyword-based queries and show how it was applied to the experimental tasks proposed for the INEX 2010 data-centric track. In our method, called StruX, users specify a schema-independent unstructured keyword-based query and it automatically generates a top-k ranking of schemaaware queries based on a target XML database. Then, one of the top ranked structured queries can be selected, automatically or by a user, to be executed by an XML query engine. The generated structured queries are XPath expressions consisting of an entity path (e.g., dblp/article) and predicates (e.g., /dblp/article[author="john" and title="xml"]). We use the concept of entity, commonly adopted in the XML keyword search literature, to define suitable root nodes for the query results. Also, StruX uses IR techniques to determine in which elements a term is more likely to occur. Keywords: XML, Keyword Search, XPath, Entities.
1 Introduction Specifying queries through keywords is currently very common. Specially in the context of search engines on the World Wide Web, users with different levels of computer skills are familiar with keyword-based search. As a consequence, such a concept has been exploited outside the scope of the Web. For example, in relational databases, several methods [2, 10, 1, 8, 3, 21] have been proposed. Considering the vast number of applications that use such databases, it is clear why there is so much interest in adopting such an approach to develop more intuitive interfaces for querying in this environment. Likewise, recently, there has been an increasing interest in the field of keyword-based search over XML documents, given the growth and consolidation of such a standard. Many techniques employ the concept of Lowest Common Ancestor (LCA) [7] with variations to specific requirements, including Meaningful LCA (MLCA) [17], Smallest S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 194–205, 2011. c Springer-Verlag Berlin Heidelberg 2011
Automatically Generating Structured Queries in XML Keyword Search
195
LCA (SLCA) [24], Valuable LCA (VLCA) [14], XSeek [18] and Multiway-SLCA [23]. Another structure-related concept, Minimum Connecting Trees [9], has led to a different approach to the problem. All of these methods aim at finding, inside the XML document, subtrees in which each query term occurs at least once in one of its leafs, and then return the root node of the subtrees as a query result. Specifically, LCA-based methods make restrictions on the choice of the root node. Notice that, for one-term queries, such methods tend to return a single-element subtree, a query result not desired in general. Furthermore, after an initial indexing phase, they disregard the source XML document, as any query will be answered considering its own indexes only. This behavior may not be suitable in a dynamic environment in which data is frequently updated or when XML data is stored in a DBMS. Considering such an environment, it becomes relevant to develop XML keyword search methods that can easily cope with data stored and managed by a DBMS. For instance, as current XML-aware DBMS can perform XQuery and XPath queries in an efficient way, one could use that to abstract the frequent updates and storage of XML data. This paper presents a novel method for keyword search over XML data based on a fresh perspective. Specifically, our method, called StruX1 , combines the intuitiveness of keyword-based queries with the expressiveness of XML query languages (such as XPath). Given a user keyword query, StruXis able to construct a set of XPath expressions that maps all possible interpretations of the user intention to a corresponding structured query. Furthermore, it assigns each XPath expression a score that measures its likelihood of representing the most appropriate interpretation of the user query. Then, a system can submit one or more of such queries to a DBMS and retrieve the results. Like in [21], the process of automatically transforming an unstructured keyword-based query into a ranked set of XPath queries is called query structuring. This paper is organized as follows. Section 2 summarizes related work. Section 3 presents an overview of background concepts and introduces StruX. Section 4 describes how StruX was applied to the experimental tasks proposed in INEX 2010 data-centric track. Section 5 discusses the experimental results, and Section 6 concludes the paper.
2 Related Work The basic principle behind StruX is similar to LABRADOR [21], which efficiently publishes relational databases on the Web by using a simple text box query interface. Other similar systems for searching over relational data include SUITS [6] and SPARKS [19]. Our proposal is different from those for two reasons: it works for a different type of data (XML instead of relational) and does not need any interaction with the user after she has specified a set of keywords. Thus, this section focuses on XML keyword search. Using keywords to query XML databases has been extensively studied. Current work tackles keyword-based query processing by modeling XML documents as labeled trees, considers that the desired answer is a meaningful part of the XML document and retrieves nodes of the XML graph that contain the query terms. In [11] and [24] the authors suggest that trees of smaller size bear higher importance. Following such an idea, 1
The name StruX is derived from the latin verb form struxi, which means to structure or build things.
196
F. da C. Hummel et al.
Algorithm 1. StruX Processing 1: procedure StruX (Input: unstructured query U , schema tree T ) 2: segs ← GenerateSegments(U ) 3: combs ← GenerateCombinations(segs) 4: for each combination c in combs do 5: cands ← GenerateCandidates(c) 6: for each candidate d in cands do 7: rank ← CalculateScores(d, S.root) 8: localRank.add(rank) 9: globalRank.add(localRank) 10: for i from 1 to k do 11: topK ranks ← GenerateXP ath(globalRank[i]) 12: StructuredQuery ← topK xpaths[0]
sorted add sorted add
the top query
XSEarch [5] adopts an intuitive concept of meaningfully related sets of nodes based on the relationship of the nodes with its ancestors on the graph. XSEarch also extends the pure keyword-based search approach by allowing the user to specify labels (element tags) and keyword-labels in the query. In a similar direction, XRANK [7] employs an adaptation of Google’s PageRank [22] to XML documents for computing a ranking score for the answer trees. A distinct approach is XSeek [18], in which query processing relies on the notion of entities inferred from DTDs. StruX shares a similar view with XSeek in this regard, but the latter does not generate queries. In [12], the authors propose the concept of mapping probability, which captures the likelihood of mapping keywords from a query to a related XML element. This mapping probability serves as weight to combine language models learned from each element. Such method does not generate structured queries, as StruXdoes. Instead, it uses the structural knowledge to refine a retrieval model. There are also other methods that go beyond simple keyword-based search. NaLIX [16] is a natural language system for querying XML in which the queries are adjusted by interacting with the user until they can be parsed by the system. Then, it generates an XQuery expression by using Schema-Free XQuery [17], which is an XQuery extension that provides support for unstructured or partially unstructured queries. Another approach, called EASE, considers keyword search over unstructured, semi-structured and structured data [15]. Specifically, EASE builds a schema graph with data graph indexes. It also provides a novel ranking algorithm for improving the search results. Finally, LCARANK [4] combines both SLCA and XRank for keyword-based search over XML streams. Although these approaches work on XML keyword-based queries, their goals are slightly different from our work, since they consider broader perspectives (i.e., natural language, graph-oriented data and XML streams).
3 StruX This section presents StruX, our method for generating structured XML queries from an unstructured keyword-based query. First, it gives an overview of StruX and then it details each of its steps.
Automatically Generating Structured Queries in XML Keyword Search .
197
! "# $% & , "# $ # & !( " $ * & , ( " $ # & ! ' "# $% & , ( " $ # & !( " $ * & , "# $ # & !( " $ * & ! "# $% & ! "# $% & , "# $ # & )$*+#($ "# ,% * # ! "# $% & )$*+#($ "# ,% !( " $ * & , ( " $ # & )$*+#($( " ,% * # !( " $ * & )$*+#($( " , * -
Fig. 1. Example of StruX steps
3.1 Overview Algorithm 1 describes StruX general steps. Next, we describe each step using an example shown in Fig. 1. Given an unstructured keyword-based query as input (Fig. 1a), StruX first splits the input sequence of keywords into segments (Fig. 1b) and then generates combinations of these segments (Fig. 1c) to obtain possible semantic interpretations of the input unstructured query. This process assumes that each keyword-based query is an unstructured query U composed of a sequence of n terms, i.e., U = {t1 , t2 , ..., tn }. This assumption is based on the intuition that the user provides keywords in a certain order. For example, a keyword query “John Clark Greg Austin” is probably intended to represent the interest in two persons named “John Clark” and “Greg Austin”, respectively. But we cannot say the same for the query “John Austin Greg Clark”. Although both queries have the same terms, the order in which they are specified may be used to describe a different intention. Also, this intuition helps StruX dealing with possible ambiguous keywords. The next step labels segment combinations with element names from the target XML database, forming sets of element-segment pairs (Fig. 1d), or candidate predicates. Once these candidate predicates have been formed, StruX finds adequate entities for each candidate (Fig. 1e). In fact, StruX relies on the concept of entity [18, 20] in order to intuitively represent real-world objects. For this task, it uses a few simple heuristics. For instance, consider an element y that has multiple sub-elements x, then y is considered an entity. This can be observed by looking at the document schema (or by traversing the actual document) and verifying that x can occur multiple times within an instance of element y. For example, considering the DTD specification of author as “”, book is a possible entity, while curriculum, according to this heuristic, is not. In addition, we extend the concept of entity by adding another constraint that avoids very specific queries: an element x must have at least one direct descendant to be
198
F. da C. Hummel et al.
considered an entity. For example, if book is defined as an element with two subelements like “”, then it is an entity. StruX identifies the elements in which the keywords are more likely to occur. It then computes scores for candidate structured queries. Such scores are needed to determine which XML query represents more accurately the user’s intention. Finally, one or more top ranked structured queries are evaluated, returning the results to the users. The final result of StruX is a query expressed in XPath2 , which specifies patterns of selection predicates on multiple elements that have some specified tree structure relationship. Hence, XML queries are usually formed by (tree) path expressions. Those expressions define a series of XML elements (labels) in which every two elements are related to each other (for example, through a parent-child relationship). Although other methods consider recursive schemas, we do not, since this kind of schema is not commonly found on the Web [13]. Also, notice that, in this paper, elements are always identified by their complete path to the document root, not only by their tag label. 3.2 Input Transformation The first step executed by StruX (Algorithm 1, line 2) splits the input sequence of keywords into segments that represent possible interpretations of the query. A segment is a subsequence Sij of terms from an unstructured query U, where Sij contains the keywords from i to j. For example, considering the query U in Fig. 1a, the generated segments are shown in Fig. 1b. Notice that, following the intuition discussed in Section 3.1, we assume that users tend to specify related keywords in sequence. This intuition is captured by the segments. Therefore, sets of tokens that are not in sequence such as “John” and “XML” are not considered. For each segment Sij , StruX retrieves all elements in which all segment keywords appear at least once within a single leaf node. Segments that retrieve no elements are called not viable and are discarded from the structuring process. For example, the segment S23 =“Smith XML” would be considered not viable if the database includes no leaf having both “Smith” and “XML”. In order to evaluate the likelihood of a segment Sij occurring within an element n, StruX uses a function called Segment-Element Affinity (SEA), which is defined by Equation 1: SEA(n, Sij ) =
j
TF-IEF(n, tk ),
(1)
k=i
where, TF-IEF(n, tk ) measures the relevance of a keyword tk for an element type n in the XML database. Such a function is similar to TF-IAF [21], which defines the relevance of a keyword with respect to the values of an attribute in a relational table. Nonetheless, StruX adapts the concept of “relational attributes” to “XML element type”. This new measure is defined by Equation 2: TF-IEF(n, tk ) = T F (n, tk ) × IEF (tk ), where each frequency is calculated by Equations 3 and 4, respectively. 2
http://www.w3.org/TR/xpath.html
(2)
Automatically Generating Structured Queries in XML Keyword Search
log(1 + fnk ) T F (n, tk ) = (3) log(1 + m)
e IEF (tk ) = log 1 + ek
199
(4)
where fnk is the number of occurrences of keyword tk as element type n, m is the total number of distinct terms that occur in n, e gives the total number of distinct element types in the XML document, and ek is the total number of element types in which the term k occurs. Equation 1 evaluates every segment no matter its number of keywords. Note that a segment with two (or more) keywords is intuitively more selective than a segment with a single keyword. For example, S13 (from Fig. 1b) is more selective than segments S11 , S22 and S33 . Hence, we consider such heuristic and propose an advanced version for function SEA in Equation 5, called Weighted SEA (WSEA), in which the number of keywords is used to favor more selective segments. W SEA(n, Sij ) = (1 + j − i) ×
j
TF-IEF(n, tk )
(5)
k=i
For representing all possible semantic interpretations of an unstructured query, StruX defines all possible combinations for a set of segments (Algorithm 1, line 3). Moreover, given a combination Ci , a keyword can belong to only one segment. For example, Fig. 1c illustrates some of the combinations for the segments in Fig. 1b. For each segment combination Ci , StruX generates all possible sets of elementsegment pairs (Algorithm 1, line 5). For example, using combination C5 = {“John Smith”, “XML”}, StruX obtains the sets of element-segment pairs illustrated in Fig. 1d, in which each set of pairs Di is called a candidate predicate, or simply candidate. Note that procs/title in D4 is different from article/title in D5 as StruX identifies elements by their complete path to the root. At the end of the input transformation procedure, the set of candidates is able to represent every possible interpretation of the user’s unstructured query. It is now necessary to determine which interpretation is more suitable to represent the original user’s intention. 3.3 Candidate Predicate Selection Once the candidates have been defined, StruX needs to find adequate entities for each candidate (Algorithm 1, lines 7 and 8). This is accomplished by using the recursive function presented in Algorithm 2. This function, called CalculateScores, performs a postorder traversal of the schema tree (which is given as input to Algorithm 1). During the traversal, the scores are propagated in a bottom-up fashion. The propagation constant α (Algorithm 2, line 8) determines the percentage of a child’s score that is assimilated by its parent score (bottom-up propagation). As a result, Algorithm 2 produces a rank that is then added to a local rank of entities for each candidate. All local ranks are merged into a sorted global rank (Algorithm 1, line 9). Each entry in the rank is a structured query, containing a score, an entity element (structural constraint) and a candidate Di (value-based predicates). The score of a structured query tries to measure how well it represents the user’s original intention while
200
F. da C. Hummel et al.
Algorithm 2. CalculateScores Function 1: function C ALCULATE S CORES(d,node) Input d: candidate, node: node from the XML database 2: for each child h in node.children do 3: CalculateScores(d, h) 4: for each element-segment e in c do 5: if e.element = node.element then 6: node.score ← node.score + e.score 7: for each child h in node.children do 8: node.score ← node.score + (α ∗ h.score) 9: if node.score > 0 AND node.isEntity() then 10: Rank.add(root)
writing her query. By doing so, is its possible to determine which interpretations of the keyword-based query are more suitable according to the underlying database. Next, a structured query can be trivially translated to XPath. Specifically, for each top-k structured query, StruX generates an XPath query statement based on the corresponding entity and the candidate predicate, as illustrated in Fig. 1e. One important final note is that we chose to transform the keyword-based queries to XPath query statements for the language simplicity. However, StruX may also be extended in order to consider other XML query languages, such as XQuery. 3.4 Keywords Matching Element Names So far, we have only discussed how our method addresses matches between keywords and the content of XML elements. Indeed, in StruX we regard such a match as the main evidence to be considered when evaluating the relevance of a structured query. However, to handle cases in which keywords match element labels, we use a very simple strategy: we boost the likelihood of all structured queries in which this is observed by adding a constant α to its score value. 3.5 Indexing In order to build a structured query from user-provided keywords, StruX relies on an index of terms. This index is defined based on the target database. Specifically, each term is associated with an inverted list containing the occurrences of this term in the elements of the database. Such an association allows the query structuring process to evaluate where a term is more likely to occur within some element. Each term occurrence in an element contains a list of leaves (each one is assigned with a leaf id) in which the term occurs. Hence, our method can determine if two or more keywords are likely to occur in a same leaf node.
4 Experimental Setup Following the INEX 2010 experimental protocol, we employed StruX to process the tasks on the data-centric track that considered the IMDB datasets. The execution was organized in runs, each one consisting in processing all topics under a certain setup.
Automatically Generating Structured Queries in XML Keyword Search
201
Table 1. Description of the runs used in the experiments Run Structured Queries used Target Datasets Name 1 top-5 "movies" ufam2010Run1 2 top-10 "movies" ufam2010Run2 3 top-5 "movies", "actors" ufam2010Run3 4 top-5 all except "other" ufam2010Run4 5 top-5 all ufam2010Run5 6 top-10 all ufam2010Run6
Specifically, given a topic T , we first generated an unstructured query U T for this topic. Next, U T was given as input to StruX, producing a list of structured queries S1T , S2T , . . . , SnT as a result, being each SiT associated with a likelihood score, calculated as described in Section 3.3. Then, we executed the top-K structured queries over the IMDB datasets. We used runs with different types of target documents to assess how the redundancy and ambiguity between entities in the datasets affect StruX (e.g., “Steven Spielberg” may appear in many parts of a movie document and also on person documents as director, producer or actor). The complete description of the runs is presented in Table 1. In the following, we discuss details regarding the generation and the processing of the runs. Dealing with Entities. As we have already explained, StruX aims at generating structured queries that return single entities found in a collection of entities. In the IMDB datasets, every document root, such as and , is intuitively an entity. However, StruX inherently considers a root element as not suitable to be an entity. To overcome this, we extended StruX to consider two virtual root nodes: (i) that has all elements as its descendants; and (ii) with all elements as descendants. With such an extension, and elements can now be considered entities. Generating Queries from Topics. For each given topic from the data-centric track, we generated a keyword-based query to be used as input for StruX. In order to do so, we took the element of the topic and applied a few transformations. This step is fully automated and aims mostly at dealing with topics specified using natural language expressions, such as “romance movies by Richard Gere or George Clooney”, “Movies Klaus Kinski actor movies good rating”, “Dogme movies”. Specifically, we applied the following transformations: i) ii) iii) iv)
simple stemming of plural terms, e.g.: movies → movie, actors → actor; removal of stop-words, e.g: by, and, to, of; disregard of terms indicating advanced search operators, such as “or”; removal of terms preceded by “−”, indicating exclusion of terms from the answers.
Fig. 2 illustrates a complete example of the whole process, including: the field of a topic, the corresponding keyword-based query and a path expression generated from it. This particular path expression corresponds to the top-1 structured query
202
F. da C. Hummel et al.
Topic: KB Query: Path Expression: Result:
true story drugs +addiction -dealer true story drug addiction /movie[ overview/plot, "true story drug addiction"] Happy Valley (2008) ... ... ... The real-life true story, Happy Valley ... that have been dramatically affected by prescription drug abuse leading to street drug abuse and addiction ... INEX Format: 2010012 Q0 1162293 1 2.6145622730255127 ufam2010Run1 /movie[1] Fig. 2. Example of the steps involved in processing an INEX data-centric topic with StruX
generated by StruX. The result obtained from applying this path expression over the IMDB dataset is also presented in the figure in two formats: as an XML sub-tree, and using the INEX output format. Next, we detail how this result was obtained. Processing Structured Queries. The final result for a given run is obtained by processing the top-k structured queries against the target IMDB datasets. This could be performed by using some native XML DBMS such as eXists-db3 . However, for our experiments, we developed a simple XPath matcher, which is used to match a document against a structured query. By doing so, we could directly output the results in the INEX result submission format, instead of having to transform the output provided by the DBMS. Fig. 2 illustrates the result for one of the topics using both formats. Regarding the scores of the results, as the final answers for the structured queries are produced by an external system (in our case a simple XPath matcher), there are no relevance scores directly associated to them. Thus, we simply assigned to the result the same score StruX has generated for the structured query from which it was obtained. Nonetheless, a single ranking of results is generated for all top-k structured queries. In this ranking, results from the top-1 query occupy the topmost positions, followed by the results from the top-2 query and so on.
5 Experimental Results In this section, we present the results obtained in the INEX 2010 experiments. Regarding the focused retrieval evaluation, ufam2010Run2 was the best run among all, with MAiP value of 0.1965. As detailed in Section 4, this run returned structured queries targeting only movies documents. 3
http://exist.sourceforge.net/
Automatically Generating Structured Queries in XML Keyword Search
203
(a)
(b) Fig. 3. Focused Retrieval – MAiP Metric (a) and Document Retrieval – MAP Metric (b) Results
Notice that ufam2010Run2 uses top-10 structured queries, while ufam2010Run1 (fourth best run) uses only top-5 structured queries. This illustrates that considering more structured queries does not affect the quality of the results. On the contrary, it can even improve it. This happens because many of the generated top-10 structured queries are often not feasible, i.e., they do not retrieve any results but, on the other hand, they do not harm the final results. Regarding the document retrieval metric, our best run was, again, ufam2010Run2, who achieved the seventh best MAP value among all runs. It was followed by runs ufam2010Run1 and ufam2010Run6, with no significant difference between them. Runs ufam2010Run3, ufam2010Run4 and ufam2010Run5 achieved the same MAP values, meaning that adding other target document types beyond movie and actor did not affect
204
F. da C. Hummel et al.
the overall precision. Also, resembling the focused retrieval metric, the two runs with top-10 structured queries performed better than their counterparts with top-5 queries.
6 Conclusions We presented a novel method called StruX for keyword-based search over XML data. In summary, StruX combines the intuitiveness of keyword-based queries with the expressiveness of XML query languages. Given a user keyword-based query, StruX is able to construct a set of XPath expressions that maps all possible interpretations of the user intention to a corresponding structured query. We used StruX to perform the tasks proposed in the INEX 2010 data-centric track for IMDB datasets. The results demonstrated that query structuring is feasible and that our method is quite effective. As future work, we plan to optimize even further our method. Specifically, we need to improve StruX for handling very large datasets. We also want to study other heuristics for improving the set of the structured queries generated. This should be accomplished by ranking the XML fragments to ensure that results closer to the original user’s intention are presented first. Finally, we want to perform experiments with different SegmentElement Affinity (SEA) functions using other Information Retrieval techniques. Acknowledgements. This work was partially supported by projects InWeb, Amanaj´e and MINGroup (CNPq grants no. 573871/2008-6, 47.9541/2008-6 and 575553/ 2008-1), and by the authors’ scholarships and individual grants from CNPq, CAPES, FAPEAM and FAPEMIG.
References 1. Aditya, B., Bhalotia, G., Chakrabarti, S., Hulgeri, A., Nakhe, C., Parag, P., Sudarshan, S.: BANKS: Browsing and Keyword Searching in Relational Databases. In: Proceedings of the 28th International Conference on Very Large Data Bases, pp. 1083–1086 (2002) 2. Agrawal, S., Chaudhuri, S., Das, G.: DBXplorer: A System for Keyword-Based Search over Relational Databases. In: Proceedings of the 18th International Conference on Data Engineering, pp. 5–16 (2002) 3. Balmin, A., Hristidis, V., Papakonstantinou, Y.: ObjectRank: Authority-Based Keyword Search in Databases. In: Proceedings of the Thirtieth International Conference on Very Large Data Bases, pp. 564–575 (2004) 4. Barros, E.G., Moro, M.M., Laender, A.H.F.: An Evaluation Study of Search Algorithms for XML Streams. Journal of Information and Data Management 1(3), 487–502 (2010) 5. Cohen, S., Mamou, J., Kanza, Y., Sagiv, Y.: XSEarch: A Semantic Search Engine for XML. In: Proceedings of the 29th International Conference on Very Large Data Bases, pp. 45–56 (2003) 6. Demidova, E., Zhou, X., Zenz, G., Nejdl, W.: SUITS: Faceted User Interface for Constructing Structured Queries from Keywords. In: Proceedings of the International Conference on Database Systems for Advanced Applications, pp. 772–775 (2009) 7. Guo, L., Shao, F., Botev, C., Shanmugasundaram, J.: XRANK: Ranked Keyword Search over XML Documents. In: Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, pp. 16–27 (2003)
Automatically Generating Structured Queries in XML Keyword Search
205
8. Hristidis, V., Gravano, L., Papakonstantinou, Y.: Efficient IR-style Keyword Search over Relational Databases. In: Proceedings of the 29th International Conference on Very Large Data Bases, pp. 850–861 (2003) 9. Hristidis, V., Koudas, N., Papakonstantinou, Y., Srivastava, D.: Keyword Proximity Search in XML Trees. IEEE Transactions on Knowledge and Data Engineering 18(4), 525–539 (2006) 10. Hristidis, V., Papakonstantinou, Y.: DISCOVER: Keyword Search in Relational Databases. In: Proceedings of 28th International Conference on Very Large Data Bases, pp. 670–681 (2002) 11. Hristidis, V., Papakonstantinou, Y., Balmin, A.: Keyword Proximity Search on XML Graphs. In: Proceedings of the 19th International Conference on Data Engineering, pp. 367–378 (2003) 12. Kim, J., Xue, X., Croft, W.: A probabilistic retrieval model for semistructured data. Advances in Information Retrieval, pp. 228–239 (2009) 13. Laender, A.H.F., Moro, M.M., Nascimento, C., Martins, P.: An X-ray on Web-Available XML Schemas. SIGMOD Record 38(1), 37–42 (2009) 14. Li, G., Feng, J., Wang, J., Zhou, L.: Effective Keyword Search for Valuable LCAs over XML Documents. In: Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management, pp. 31–40 (2007) 15. Li, G., Feng, J., Wang, J., Zhou, L.: An Effective and Versatile Keyword Search Engine on Heterogenous Data Sources. Proceedings of the VLDB Endowment 1(2), 1452–1455 (2008) 16. Li, Y., Yang, H., Jagadish, H.V.: NaLIX: an Interactive Natural Language Interface for Querying XML. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 900–902 (2005) 17. Li, Y., Yu, C., Jagadish, H.V.: Schema-Free XQuery. In: Proceedings of the Thirtieth International Conference on Very Large Data Bases, pp. 72–83 (2004) 18. Liu, Z., Chen, Y.: Identifying Meaningful Return Information for XML Keyword Search. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 329–340 (2007) 19. Luo, Y., Wang, W., Lin, X.: SPARK: A Keyword Search Engine on Relational Databases. In: Proceedings of the 24th International Conference on Data Engineering, pp. 1552–1555 (2008) 20. Mesquita, F., Barbosa, D., Cortez, E., da Silva, A.S.: FleDEx: Flexible Data Exchange. In: Proceedings of the 9th ACM International Workshop on Web Information and Data Management, pp. 25–32 (2007) 21. Mesquita, F., da Silva, A.S., de Moura, E.S., Calado, P., Laender, A.H.F.: LABRADOR: Efficiently publishing relational databases on the web by using keyword-based query interfaces. Information Process Management 43(4), 983–1004 (2007) 22. Page, L., Brin, S., Motwani, R., Winograd, T.: The PageRank Citation Ranking: Bringing Order to the Web. Technical report, Stanford Digital Library Technologies Project (1998) 23. Sun, C., Chan, C.Y., Goenka, A.K.: Multiway SLCA-based keyword search in XML data. In: Proceedings of the 16th International Conference on World Wide Web, pp. 1043–1052 (2007) 24. Xu, Y., Papakonstantinou, Y.: Efficient Keyword Search for Smallest LCAs in XML Databases. In: Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, pp. 527–538 (2005)
UPF at INEX 2010: Towards Query-Type Based Focused Retrieval Georgina Ram´ırez Universitat Pompeu Fabra, Barcelona, Spain [email protected]
Abstract. This paper describes our participation at INEX 2010. We participated in two different tracks: ad-hoc and data-centric. We first propose a classification of INEX topics and analyze several characteristics of the relevance assessments from INEX 2009 for each of the topic classes. The goal of our study is to investigate whether there are differences in relevance judgements between topic classes in order to use this information at retrieval time. We also present the experiments we performed on the INEX 2010 data. In the ad-hoc track we study the performance effects of changing the article order (fetching phase) while in the data-centric track we experiment with the use of different indices and retrievable element types. Our main finding is that indexing uniquely movie documents leads to much better performance than indexing the complete collection. Keywords: XML, focused retrieval, INEX, query-based retrieval, query classification.
1
Introduction
Retrieval tasks such as XML retrieval, where focused access to relevant information is provided, allow users to perform very focused searches and pose restrictions on the type of information being requested (e.g., I want references or experimental results). This is one of the reasons why several query languages and interfaces have been designed—to allow users to explicitly express more complex needs. However, these tools are not always available and users often specify in their keyword queries not only what they are looking for but also the type and the specificity of the information they are searching for. Thus, the number and variety of topic types that XML retrieval introduces differ from those of traditional document retrieval, where the task is to return whole documents. In this paper we propose a classification of XML retrieval topics based on three different dimensions: 1) the type of information sought (general or restricted), 2) the specificity of the topic (generic or specific), and 3) the complexity of the topic (simple or compound). Once a classification is defined, it can be used in multiple ways. Our goal is to use it to perform specific retrieval strategies for each of the topic classes. A classification can also help to provide a more balanced benchmark topic-set, avoiding to reward retrieval systems that perform well solely on the most popular topic type. S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 206–218, 2011. c Springer-Verlag Berlin Heidelberg 2011
UPF at INEX 2010: Towards Query-Type Based Focused Retrieval
207
We analyze several characteristics of the relevant judgements from INEX 2009 for each of the topic classes defined in our classification. The aim of the study is to investigate whether the differences between topic classes could be used to decide on specific retrieval strategies for each of the topic types, a first step towards query-type based focused retrieval. We also describe the experiments performed on the INEX 2010 data for two different tracks: ad-hoc and data-centric. Our experiments for the ad-hoc track study the performance effects of changing the article order (fetching phase) while in the data-centric track we experiment with the use of different indices and retrievable element types. The rest of the paper is organized as follows: In Section 2 we present our INEX topic classification and explain the dimensions used. The analysis performed on INEX 2009 data is described in Section 3. Section 4 presents the results of our experiments for both tracks: ad-hoc and data-centric. Finally, we discuss the main contributions and future work in Section 5.
2
INEX Topic Classification
In this section we propose a classification of INEX topics. We extend our previous work on classification of INEX information needs [2] with a new dimension: the type of information sought. 2.1
Dimensions
Our classification uses the following three dimensions: 1) the type of information sought (general or restricted), 2) the specificity of the topic (generic or specific), and 3) the complexity of the topic (simple or compound). Topics that are restricted regarding the type of information sought can be further divided according to the type of restriction (topical or structural). Type of Information Sought. In an XML retrieval scenario, where focused access to relevant information is provided, users can pose restrictions on the type of information being searched for (e.g., I want images or results). We classify topics into General and Restricted requests. General requests are those that ask for any type of information about a topic, without restrictions. Note that the topic might be very specific but the type of information the user wants to see is generic (any type of information about it). Restricted requests are those in which some type of constraint on the type of information being sought is specified. This constraint can be topical (e.g., I want exercises or experiment results) or structural (e.g., I want references or images). In other words, the restrictions can specify which part of the content has to be returned, e.g., “I like to know the speed capacity of vehicles” (not any other information on vehicles) or the type of object that it is returned, e.g., “I like too see images of sunflowers” (not any other information/object about sunflowers). Note that General requests are those that are typically used in web search and document retrieval, where the task is to return whole documents or web pages.
208
G. Ram´ırez
Complexity. In the complexity dimension, two categories are used: Simple and Compound. Simple requests are those that ask for information about just one topic or aspect of a topic (i.e., mono-faceted requests). While Compound requests are those that ask for information about several topics or aspects of the same topic (i.e., multifaceted requests) or want information about the relationship between two topics (e.g. technique A in the field of B or information about A for B). Specificity. In the specificity dimension, we classify requests into Specific and Generic, depending on the topical broadness of the information being searched for. In other words, General requests are those that ask for information about a broad topic, while Specific requests are those that ask for information about a narrow topic. Note that here we talk about the information being searched for and not about the type of information being searched for. While the specificity and the complexity of the request are dimensions that have already been used to classify standard IR requests [5], the type of information sought is specific to focused retrieval. Notice also the difference between restricting a topic (e.g., classical movies) and restricting the type of information sought (e.g., pictures of movies). The first one would be specific while the second one would be structurally restricted. We hypothesize that the characteristics of the relevant information between the classes of each dimension differ. If these differences exist, XML retrieval systems should use this information in order to optimize their retrieval performance by using specific retrieval strategies for each of the topic classes. 2.2
Data Classification
The INEX ad-hoc topics are created by the participants following precise instructions. Candidate topics contain a short CO (keyword) query, an optional structured CAS query, a phrase title, a one line description of the search request, and a narrative with details of the topic of request and the task context in which the information need arose. An example of an INEX topic with all its fields can be seen in Table 1. We used the description field of the topics to classify the INEX 2009 topics into different classes. If needed, we used the narrative field from the topic to clarify. All 68 topics were classified by two different volunteers. Table 2 shows the resulting topic classes, the number of topics belonging to each class and gives an example for each of them. We also investigate how intuitive the dimensions and categories used in our classification are. We do so by analyzing the level of agreement between volunteers. Table 3 shows the agreement on each of the dimensions between the two volunteers. We can see that the agreements on the first two dimensions are rather high, suggesting that these dimensions are quite intuitive and objective. However, the specificity dimension has a very low agreement percentage and it is probably too subjective to be used in a real setting.
UPF at INEX 2010: Towards Query-Type Based Focused Retrieval
209
Table 1. Example of INEX topic Topic ID Title CAS query Phrase title Description Narrative
3
2009011 olive oil health benefit //food[about(., olive oil) and about(., health benefit)] “olive oil” “health benefit” Find information about what sort of health benefit olive oil has I’m a health/beauty buff. I recently learned that olive oil is “good for you”. What are the specific health/beauty benefits for consuming olive oil? Any article that mentions health benefits of olive oil is fine, EXCEPT those in which the claim is based on either unpopular/obscure or unscientific methods. So for example, if XXX diet recommends consuming olive oil then it’s irrelevant. Note that since Mediterranean diet is not a weight-loss fat diet, but the traditional Mediterranean ways of eating, an article describing the health benefits of olive oil in this setting is relevant. Anything outside of health benefit is irrelevant, how olive oil is produced, the different grades of olive oil etc.
Relevance Assessments Analysis
In this section we investigate whether the characteristics of the relevance judgements differ between the different topic classes described above. Having in mind that for some classes the number of topics is generally too low to draw statistically significant conclusions, we analyze INEX 2009 relevance judgements and look at the relevance characteristics of each of the topic classes. We analyze the following characteristics: 1) the number of relevant documents, 2) the density of the relevant documents, and 3) the number and size of the relevant fragments. Table 2. Number of INEX 2009 topics belonging to each of the topic classes and example of topic description for each of them Dimension Type of information sought
Class General
Num. Example 64 Information about Nobel prize.
Restricted (structurally)
2
Explain “mean average precision” and “reciprocal rank” with images or plots. Provide references in proceedings and journals.
Restricted (topically) Complexity Simple
2
I want to know vehicles and its speed capacity
46
Information about classical movies
Compound
22
Generic
13
Find information about applications of bayesian networks in the field of bioinformatics I want to find some information about IBM computer
Specific
55
Specificity
Find information on causes of vitiligo and treatment for it
210
G. Ram´ırez Table 3. Agreement between the two volunteers that classified the topics Type of information sought 94%
Complexity 85 %
Specificity 46%
Number of Relevant Documents. By number of relevant documents we refer to the number of unique documents in the collection that contain relevant information given a topic, even if the fraction of relevant information is small. Figure 1 (upper part) shows the average number of documents containing relevant information for each of the topic classes. We can clearly see that restricted requests tend to find much less relevant documents than the general ones. On average, there are 18 relevant documents for the restricted topics and 75 for the general ones (13 and 56.5 when looking at the median). Although the difference is not that big for the other dimensions, we see that compound requests tend to find less relevant documents than simple ones (51 vs. 81 on average and 26 vs. 58 when looking at the median). Specific topics are also satisfied with a smaller number of documents than generic ones (65 vs. 100 on average and 48 vs. 65 when looking at the median). These tendencies are not surprising, it seems reasonable that the more complex, restricted, and specific a topic is, the more difficult is to find information that satisfies it. Density. We also analyze how densely relevant are the documents that contain relevant information. According to recent work [6], focused search works better
Fig. 1. Average number of documents containing relevant information (top) and average density of those documents (bottom)
UPF at INEX 2010: Towards Query-Type Based Focused Retrieval
211
on sparsely dense documents. We define density of a document as the percentage of relevant text contained in that document (i.e., ratio of relevant text to all text). Text size is given by the number of characters. Figure 1 (bottom part) shows the average density of the documents containing relevant information for each of the topic classes. We can see that documents that contain relevant information for the restricted topics tend to be sparsely dense. On average, 18% of the text in a document is relevant for the restricted topics while that is 44% for the general ones (19% and 38% when looking at the median). Focused retrieval seems to be more desirable for restricted topics. Regarding the other dimensions, we see that there is not much difference in terms of density between the compound and simple topics. In both cases, documents are quite dense on average. The difference is bigger in the specificity dimension. While generic topics tend to be answered with highly dense documents (on average 48% and median 59%), specific topics tend to be answered with less dense documents (41% on average and median 34%). Number and Size of Relevant Fragments. To see how the relevant information is distributed within an article, we look at the number and size of relevant fragments, the fragments that contain the relevant information. Figure 2 shows this information for each of the topic classes. While there are not big differences in the number of relevant fragments between the restricted topics (average 2, median 2) and the general topics (average 1.7, median 1.5), the fragments for the restricted topics tend to be much smaller (see Figure 2 bottom part). On average, relevant fragments for the restricted topics are 540 characters long
Fig. 2. Number and average size of relevant fragments
212
G. Ram´ırez
(median 512) while the average length for the general topics is 2668 (median 1644). This is not the case for the other two dimensions where the number and average size of the relevant fragments are very similar between classes. We can see that, in general, a very small number of very long fragments are assessed as relevant, not the best scenario for focused retrieval. We also look at two characteristics of the topic itself: 1) the number of query terms and 2) the type of CAS query. If there are differences between topic classes, these characteristics can help to automatically classify topics. Number of Query Terms. By number of query terms we refer to the number of terms in the title field of the topic after removing stop-words. Figure 3 (upper part) shows the average number of query terms for each of the topic classes. Restricted topics tend to be long, they have an average of 6.5 terms per topic (median 6) while the average number of query terms for the general ones is 3.7 (median 4). The difference can be explained from the fact that restricted requests specify not only what the users are searching for but also the type of information they would like to see. We can see a similar pattern for the specificity dimension. While generic topics are expressed, on average, with a very small number of query terms (2.5, median 2), specific topics tend to be longer (average 4, median 4). We can also see that there are not big differences regarding the number of query terms between simple and compound topics (complexity dimension).
Fig. 3. Average number of query terms and percentages of CAS query types
UPF at INEX 2010: Towards Query-Type Based Focused Retrieval
213
Table 4. CAS query patterns Pattern //article[about(.,X)]
Meaning Return articles about topic X
//*[about(.,X)]
Return any type of element about topic X
//A[about(.,X)]
Return element types A about topic X (where A can be any element type except article)
//article[about(.//A,X)]
Return articles that contain an element type A about topic X
//article[about(.,X)]//A[about(.,Y)]
Return element types A about topic Y that are contained in articles about topic X.
Type of CAS Query. We also analyze which types of CAS query are associated with each of the topic classes. We used five different CAS patterns to classify all topics (see Table 4). Figure 3 (bottom part) shows the percentage of CAS queries of each pattern for each of the topic classes. We can see some differences. While restricted topics tend to be specified with longer and more specific CAS queries, the majority of the general ones use the most generic form of CAS query1 . More surprising is that a big portion of the simple and generic topics use the more specific CAS queries. In general terms however, it is difficult to associate any of the CAS query pattern to a specific topic class.
4
Experiments
This section describes the setup and discusses the results of the experiments carried out for the ad-hoc and the data-centric tracks of INEX 2010. For all our experiments we have used the Indri Search Engine [1]. Indri uses a retrieval model based on a combination of language modeling and inference network retrieval frameworks. We have used linear smoothing and varying lambda value. Topics and documents have been pre-processed using the porter stemmer [3] and the smart stop-word list [4]. 4.1
Ad-Hoc Track Experiments
For the ad-hoc track experiments we have used the Indri search engine [1] with linear smoothing and lambda 0.45. The lambda value has been set to 0.45 after training on the INEX Wikipedia 2009 collection. The only indexed fields are articles, sections, and paragraphs, meaning that only these element types can be explicitly retrieved. We study the importance of the fetching phase, i.e., the performance effects of changing the article order. 1
Note that topics that were submitted without a CAS query were assigned the most generic one: //article[about(.,X)].
214
G. Ram´ırez
Relevant in Context. The aim of the Relevant in Context Task is to first identify relevant articles (the fetching phase), and then to identify the relevant results within the fetched articles (the browsing phase). As mentioned above, we experiment with the performance effects of the fetching phase. For that, we use the same baseline run and reorder its articles in three different ways. Our baseline is a paragraph run (retrieving only paragraphs) grouped by article. The final article order is given by 1) our own article run order (retrieving only articles), 2) the reference run order, 3) the baseline run order (i.e., the article where the most relevant paragraph appears, followed by the article were the second most relevant paragraph appears, etc.). Our original submissions were not valid due to a bug in our code and could not be evaluated. The results of the un-official runs (after fixing) are shown in Table 5. We can see that changing the article order affects considerably the performance of the run. The reference run article order outperforms the other two. Paragraphs are not good indicators of article relevance. Ranking articles by their most relevant paragraph performs the worst. Table 5. Un-official results for the relevant in context runs. The number in parentheses indicates the estimated run position in the official ranking. run name UPFpLM45coRC1 UPFpLM45coRC2 UPFpLM45coRC3
MAgP (est. position) 0.1109 (34) 0.1571 (9) 0.0763 (39)
Restricted Relevant in Context. The Restricted Relevant in Context Task is a variant of the Relevant in Context task, where only 500 characters per article are allowed to be retrieved. Overlapping results are not permitted. For this task, we have followed a similar approach to the one of our Relevant in Context runs. This time however, our baseline is a section run (retrieving only sections) and a second post-processing step is made in order to return only 500 characters per article. That is, per article we return the most relevant sections until the 500th character is reached. As in the previous task, the final article order is given by 1) our own article run order (retrieving only articles), 2) the reference run order, 3) the baseline run order (i.e., the article where the most relevant section appears, followed by the article were the second most relevant section appears, etc.). As in the previous task, our original submissions were not valid due to a bug in our code and could not be evaluated. The results of the un-official runs (after fixing) are shown in Table 6. We see similar performances as in the previous task; the reference run article order outperforms the other two. In absolute numbers paragraph runs seem to outperform section runs. However, the lower performance of the section runs could be due to the task restrictions. When looking at the relative position of the runs regarding the other groups, the three runs performed relatively well (top 15-20).
UPF at INEX 2010: Towards Query-Type Based Focused Retrieval
215
Table 6. Unofficial results for the restricted relevant in context runs. The number in parentheses indicates the estimated run position in the official ranking. run name UPFpLM45coRRC1 UPFpLM45coRRC2 UPFpLM45coRRC3
MAgP (est. position) 0.0894 (15) 0.1210 (13) 0.0633 (17)
Table 7. Official results for the restricted focused runs. The number in parentheses indicates the run position in the official ranking. run name UPFpLM45coRF1 UPFpLM45coRF2 UPFpLM45coRF3
char prec (position) 0.2984 (19) 0.3066 (15) 0.1156 (30)
Table 8. Official runs for the data-centric track run name UPFL15Tall UPFL45Tall UPFL15Tmovie UPFL45Tmovie UPFL15TMI UPFL15TMImov
index all all all all movies movies
retrievable elements no restriction no restriction movie movie no restriction movie
lambda 0.15 0.45 0.15 0.45 0.15 0.15
Table 9. Official results for the data-centric track. The number in parentheses indicates the run position in the official ranking. run name UPFL15Tall UPFL45Tall UPFL15Tmovie UPFL45Tmovie UPFL15TMI UPFL15TMImov
MAgP 0.2459 (2) 0.2434 (3)
MAiP 0.1486 (7) 0.1338 (11) 0.0770 (20) 0.0410 (24) 0.1809 (2) 0.1762 (3)
Document Retrieval 0.2961 (6) 0.2822 (8) 0.1983 (16) 0.1578 (20) 0.3687 (3) 0.3542 (4)
Restricted Focused. The Restricted Focused task aims at giving a quick overview of the relevant information in the whole of Wikipedia. Results are restricted to max. 1,000 characters per topic. For this task, we return a single paragraph per article (the most relevant) until we reach the 1,000 characters per topic. The assumption is that users prefer to see an overview based on the largest number of articles rather than seeing several relevant paragraphs of the same article. Our three official runs are again based on different article order as in the previous tasks; The final article order is given by 1) our own article run order (retrieving only articles), 2) the reference run order, 3) the baseline run order (i.e., the article where the most relevant paragraph appears, followed by the article were the second most relevant paragraph appears, etc.).
216
G. Ram´ırez
MAgP
MAiP
MAP Fig. 4. Official evaluation graphs for the data-centric track
UPF at INEX 2010: Towards Query-Type Based Focused Retrieval
217
The results of these runs are shown in Table 7. As in the other tasks, we can see that the article order is an important factor on the overall result of the run. There is a big difference in terms of performance from our article order and the reference run order and our paragraph run order. Paragraphs are not good estimators of the total relevance of an article. In other words, a relevant paragraph does not imply that the article is relevant to the same degree. 4.2
Data-Centric Track Experiments
For our data-centric track experiments we used the Indri search engine [1] with linear smoothing and two different lambdas, 0.45 and 0.15. Since this is a new collection and we did not have training data to optimize lambda, we experimented with two different values that have been successfully used in other collections. We also experimented with the use of two different indices (indexing all the collection vs. indexing only movies) and by restricting the type of elements to be retrieved (no restriction vs. movie elements)2 . Table 8 shows the parameters used for each of our official runs and Table 9 the official results. Our best performing runs are the ones that use the movie index, indicating that for this specific topic set the use of other types of documents introduces noise. We also see that lambda 0.15 always performs better than lambda 0.45, indicating that it is better to give less emphasis to the collection statistics. Figure 4 show the official graphs. In general terms we can see that using the movie index (our best runs) leads to high precision at early recall levels while, not surprisingly, it does not manage to do so at middle and/or high recall levels (MAiP and MAP graphs). This is because a large part of the collection is not indexed, which makes it difficult, if not impossible, to have a high overall recall.
5
Discussion and Conclusions
This paper described our participation at INEX 2010. We presented a classification of INEX topics and an analysis of the characteristics of the relevance assessments for each of the topic classes. The goal of our study was to investigate whether there are differences in relevance judgements between topic classes in order to use them for retrieval. We have seen, for instance, that restricted topics have a small set of relevant documents which are sparsely dense and relevant information is contained in small fragments of documents. Although some of the analyzed relevance characteristics differ between classes, it is not clear whether this information could be used for retrieval. More data needs to be analyzed in order to see whether these differences are statistically significant. The classification presented is based on three different dimensions (type of information sought, complexity, and specificity), generic enough to be used in 2
Note that movie elements can have very different forms: from a complete movie document to a movie element within a list of movies played by an actor.
218
G. Ram´ırez
other focused retrieval scenarios. Our goal is to use it to perform different retrieval strategies for each of the topic classes. A classification can also help to provide a more balanced benchmark topic-set, avoiding to reward retrieval systems that perform well solely on the most popular topic type. We note that not all the dimensions are objective enough to be easily used. The specificity of a topic is a subjective matter and it might not be easy to apply in real settings. As future work we plan to investigate whether it is beneficial to use different retrieval strategies for the different topic types. We reported on our experiments for INEX 2010, in the ad-hoc and data-centric tracks. In the ad-hoc track we studied the performance effects of changing the article order (fetching phase) while in the data-centric track, we experimented with the use of different indices and retrievable element types. Our results on the ad-hoc track confirm that article order is a very important factor on the overall performance of the systems. In all of our experiments, the article order of the reference run outperforms the other runs. In the data-centric track, our main finding is that indexing only the movie documents leads to much better performance than indexing the complete collection. Acknowledgments. This work has been supported by the Spanish Ministry of Science and Education under the HIPERGRAPH project and the Juan de la Cierva Program.
References 1. Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: a language model based search engine for complex queries. In: Proceedings of the International Conference on Intelligent Analysis (2005) 2. Ram´ırez Camps, G.: Structural Features in XML Retrieval. PhD thesis, University of Amsterdam (2007) 3. Porter, M.F.: An algorithm for suffix stripping. Program 14(3), 130–137 (1980) 4. Salton, G.: The SMART Retrieval System Experiments in Automatic Document Processing. Prentice-Hall, Inc., Upper Saddle River (1971) 5. Ingwersen, P., J¨ arvelin, K.: The Turn: Integration of Information Seeking and Retrieval in Context. The Information Retrieval Series, vol. 18. Springer, Heidelberg (2005) 6. Arvola, P., Kekalainen, J., Junkkari, M.: Focused Access to Sparsely and Densely Relevant Documents. In: Proceedings of SIGIR 2010 (2010)
BUAP: A First Approach to the Data-Centric Track of INEX 2010 Darnes Vilari˜ no, David Pinto, Carlos Balderas, Mireya Tovar, and Saul Le´ on Facultad de Ciencias de la Computaci´ on Benem´erita Universidad Aut´ onoma de Puebla, M´exico {darnes,dpinto,mtovar}@cs.buap.mx, charlie [email protected], [email protected]
Abstract. In this paper we present the results of the evaluation of an information retrieval system constructed in the Faculty of Computer Science, BUAP. This system was used in the Data-Centric track of the Initiative for the Evaluation of XML retrieval (INEX 2010). This track is focused on the extensive use of a very rich structure of the documents beyond the content. We have considered topics (queries) in two variants: Content Only (CO) and Content And Structure (CAS) of the information need. The obtained results are shown and compared with those presented by other teams in the competition.
1
Introduction
Current approaches proposed for keyword search on XML data can be categorized into two broad classes: one for document-centric XML, where the structure is simple and long text fields predominate; the other for Data-Centric XML, where the structure is very rich and carries important information about objects and their relationships [1]. In previous years, INEX focused on comparing different retrieval approaches for document-centric XML, while most research work on Data-Centric XML retrieval cannot make use of such a standard evaluation methodology. The new Data-Centric track proposed at INEX 2010 aims to provide a common forum for researchers or users to compare different retrieval techniques on Data-Centric XML, thus promoting the research work in this field [2]. Compared to traditional information retrieval, where whole documents are usually indexed and retrieved as single complete units, information retrieval from XML documents creates additional retrieval challenges. Until recently, the need for accessing the XML content has been addressed diferently by the database (DB) and the information retrieval (IR) research communities. The DB community has focussed on developing query languages and eficient evaluation algorithms used primarily for Data-Centric XML documents. On the other hand, the IR community has focussed on document-centric XML documents
This work has been partially supported by the CONACYT project #106625, VIEP # VIAD-ING11-I, as well as by the PROMEP/103.5/09/4213 grant.
S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 219–226, 2011. c Springer-Verlag Berlin Heidelberg 2011
220
D. Vilari˜ no et al.
by developing and evaluating techniques for ranked element retrieval. Recent research trends show that each community is willing to adopt the well-established techniques developed by the other to efectively retrieve XML content [3]. The Data-Centric track uses the IMDB data collection newly built from the following website: http://www.imdb.com. It consists of information about more than 1,590,000 movies and people involved in movies, e.g. actors/actresses, directors, producers and so on. Each document is richly structured. For example, each movie has title, rating, directors, actors, plot, keywords, genres, release dates, trivia, etc.; and each person has name, birth date, biography, filmography, and so on. The Data-Centric track aims to investigate techniques for finding information by using queries considering content and structure. Participating groups have contributed to topic development and evaluation, which will then allow them to compare the effectiveness of their XML retrieval techniques for the DataCentric task. This will lead to the development of a test collection that will allow participating groups to undertake future comparative experiments.
2
Description of the System
In this section we describe how we have indexed the corpus provided by the task organizers. Moreover, we present the algorithms developed for tackling the problem of searching information based on structure and content. The original XML file has been previously processed in order to eliminate stopwords and punctuation symbols. Moreover, we have transformed the original hierarchical structure given by XML tags to a similar representation which may be easily analyzed by our parser. For the presented approach we have used an inverted index tree in order to store the XML templates of the corpus. The posting list contains the reference of the document (document ID) and the frequency of the indexed term in the given context (according to the XML tag). With respect to the dictionary of the inverted index, we have considered to include both, the term and the XML tag (the last one in the hierarchy). In Figure 1, we show an example of the inverted index (pay special attention to the dictionary). The aim was to be able to find the correct position of each term in the XML hierarchy and, therefore, to be able to retrieve those parts of the XML file containing the correct answer of a given query. In this way, the inverted index allows to store the same term which occurs in different contexts. We assumed that the last XML tag would be enough for identifying the full path in which the term occurs, however, it would be better to use all the hierarchy in the dictionary. Further experiments would verify this issue. In the following subsection we present the algorithms developed for indexing and searching information based on content and structure. 2.1
Data Processing
Before describing the indexing techniques used, we first describe the way we have processed the data provided for the competition. We have cleaned the XML files
BUAP: A First Approach to the Data-Centric Track of INEX 2010 title hannibal 40
: 981731:1 994171:1
character lecter 440 : 959641:1 959947:1
221
78811:1 [...] 1161611:1 [...] 1161611:1 969446:1 [...]
name hopkins 3068
: 1154469:1 1154769:2 1154810:1 [...] 1161611:1 [...]
name anthony 31873
: 943773:1 [...] 944137:2 1161611:1 1224420:3 [...]
director scott 4771
: 1157203:1 1157761:1 1157773:1 [...] 1161611:1 [...]
director ridley 62
: 1289515:1 1011543:1 1011932:1 [...] 1161611:1 [...]
writer harris 2114
: 1120749:1 1121040:1 1121294:1 [...] 1161611:1 [...]
writer thomas 7333
: 115985:1 115986:1 [...] 1161611:1 1161616:2 [...]
:
:
Fig. 1. Example of the type of inverted index used in the experiments
in order to obtain an easy way of identifying the XML tag for each data. For this purpose, as we previously mentioned, we have traduced the original hierarchical structure given by XML tags to a similar representation which may be easily analyzed by our parser. Thereafter, we have created five different inverted indexes, for the each one of the following categories: actors, directors, movies, producers and others. The inverted index was created as mentioned in the previous section. Once the dataset was indexed we may be able to respond to a given query. In this case, we have also processed the query by identifying the corresponding logical operators (AND, OR). Let us consider the query presented in Figure 2, which is then traduced to the sentence shown in Figure 3. The first column is the topic or query ID; the second column is the number associated to the ct no tag; the third column indicates the number of different categories that will be processed, in this case, we are considering only one category: movies. In the competition we submitted two runs. The first one uses the complete data of each record (word n-gram), whereas the second approach considered to split the data, that corresponds to each content, into unigrams, with the goal of being more specific in the search process. However, as will be seen in the experimental results section, both approaches perform similar. An example topic showing the second approach is given in Figure 4, whereas its traduced version is given in Figure 5. In order to obtain the list of candidate documents for each topic, we have calculated the similarity score between the topic and each corpus document as shown in Eq. (1) [4], which was implemented as presented in Algorithm 1.
222
D. Vilari˜ no et al.
Yimou Zhang 2010 2009 //movie[about(.//director, "Yimou Zhang") and (about(.//releasedate, 2010) or about(.//releasedate, 2009))] I want to know the latest movies directed by Yimou Zhang. I am interested in all movies directed by Yimou Zhang, and I want to learn the latest movies he directed. Fig. 2. An example of a given query (topic) 2010001 3 1 //movie//director yimou zhang and //movie//releasedate 2010 or //movie//releasedate 2009 Fig. 3. Representation of the topic tom hanks steven spielberg //movie[about(., tom hanks steven spielberg)] movies where both tom hanks and steven spielberg worked together The user wants all movies where Tom Hanks and Steven Spielberg have worked together (as actors, producers, directors or writers). A relevant movie is a movie where both have worked together. Fig. 4. An example of another topic 2010025 19 1 //movie tom and //movie hanks and //movie steven and //movie spielberg Fig. 5. The representation of a topic by splitting the data
SIM(q, d) =
ck ∈B cl ∈B
CR(ck , cl )
t∈V
weight(q, t, ck )
weight(d, t, cl )
c∈B,t∈V
weight(d, t, c)2
(1) where the CR function is calculated as shown in Eq. (2), V is the vocabulary of non-structural terms; B is the set of all XML contexts; and weight(q, t, c) and weight(d, t, c) are the weights of term t in XML context c in query q and document d, respectively (as shown in Eq. (3)).
BUAP: A First Approach to the Data-Centric Track of INEX 2010
CR(cq , cd ) =
1+|cq | 1+|cd |
0
if cq matches cd otherwise
223
(2)
where cq and cd are the number of nodes in the query path and document path. weight(d, t, c) = idft ∗ wft,d
(3)
where idft is the inverse document frequency of term t, and wft,d is the frequency ot term t in document d. Algorithm 1. Scoring of documents given a topic q 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
2.2
Input: q, B, V , N : Number of documents, normalizer Output: score for n = 1 to N do score[n] = 0 foreach cq , t ∈ q do wq = weight(q, t, cq ) foreach c ∈ B do if CR(cq , c) > 0 then postings = GetPostings(c, t) foreach posting ∈ postings do x = CR(cq , c) ∗ wq ∗ PostingWeight(posting) score[docID(posting)]+ = x end end end end end for n = 1 to N do score[n] = score[n]/normalizer[n] end return score
Experimental Results
We have evaluated 25 topics with the corpus provided by the competition organizers. This dataset is made up of 1,594,513 movies, 1,872,492 actors, 129,137 directors, 178,117 producers and, finally, 643,843 files categorized as others. As it was mentioned before, we submitted two runs which we have named: “FCC-BUAP-R1” and “FCC-BUAP-R2”. The former uses the complete data of each record (n-gram), whereas the latter split the words contained in the query by unigrams. The obtained results, when evaluating the task as focused retrieval (MAgP measure) are presented in Table 1 and in Figure 6. As may be seen, we have obtained a low performance, which we consider is derived of the fact of using only one tag for identifying each indexed term. We
224
D. Vilari˜ no et al. Table 1. Evaluation measured as focused retrieval (MAgP)
# 1 2 3 4 5 6 7 8 9 10 11 12 13 14
MAgP 0.24910409 0.24585548 0.24337897 0.18113477 0.15617634 0.06517544 0.0587039 0.04490731 0.04426635 0.04091211 0.04037697 0.03788804 0.03407941 0.02931703
Institute Run University of Otago OTAGO-2010-DC-BM25 Universitat Pompeu Fabra UPFL15TMI Universitat Pompeu Fabra UPFL15TMImov Kasetsart University NULL University of Otago OTAGO-2010-DC-DIVERGENCE INDIAN STATISTICAL INSTITUTE ISI fdbk em 10 INDIAN STATISTICAL INSTITUTE ISI nofdbk article .4.-1 INDIAN STATISTICAL INSTITUTE ISI nofdbk article .4.0 INDIAN STATISTICAL INSTITUTE ISI fdbk 10 B. Univ. Autonoma de Puebla FCC-BUAP-R1 B. Univ. Autonoma de Puebla FCC-BUAP-R2 INDIAN STATISTICAL INSTITUTE ISI nofdbk article .6.0 INDIAN STATISTICAL INSTITUTE ISI nofdbk article .4.1 INDIAN STATISTICAL INSTITUTE ISI nofdbk article .2.0
Fig. 6. Evaluation measured as focused retrieval (MAgP)
BUAP: A First Approach to the Data-Centric Track of INEX 2010
225
assumed that the last XML tag in each context would be enough for identifying the complete path in which the term occurs, however, it would be better to use the complete hierarchy in the dictionary. Once the gold standard is being released, we are considering to carry out more experiments in order to verify this issue. The evaluation of the different runs at the competition, measured as document retrieval, may be found in Table 2. Table 2. Evaluation measured as document retrieval (whole document retrieval) # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
3
MAP 0.5046 0.5046 0.3687 0.3542 0.3397 0.2961 0.2829 0.2822 0.2537 0.2512 0.2263 0.2263 0.2263 0.2103 0.2044 0.1983 0.1807 0.18 0.1783 0.1578 0.1126 0.0888 0.0674 0.0672 0.0602 0.0581 0.0544 0.0507 0.0424
Institute Run SEECS, Peking University NULL SEECS, Peking University NULL Universitat Pompeu Fabra UPFL15TMI Universitat Pompeu Fabra UPFL15TMImov University of Otago OTAGO-2010-DC-BM25 Universitat Pompeu Fabra UPFL15Tall Universidade Federal do Amazonas ufam2010Run2 Universitat Pompeu Fabra UPFL45Tall Universidade Federal do Amazonas ufam2010Run1 Universidade Federal do Amazonas ufam2010Run5 Universidade Federal do Amazonas ufam2010Run3 Universidade Federal do Amazonas ufam2010Run4 Universidade Federal do Amazonas ufam2010Run5 University of Otago OTAGO-2010-DC-DIVERGENCE Kasetsart University NULL Universitat Pompeu Fabra UPFL15Tmovie INDIAN STATISTICAL INSTITUTE ISI elts.0 INDIAN STATISTICAL INSTITUTE ISI elts.1 INDIAN STATISTICAL INSTITUTE ISI elts.-1 Universitat Pompeu Fabra UPFL45Tmovie INDIAN STATISTICAL INSTITUTE ISI fdbk em 10 INDIAN STATISTICAL INSTITUTE ISI nofdbk article .4.-1 INDIAN STATISTICAL INSTITUTE ISI nofdbk article .4.0 INDIAN STATISTICAL INSTITUTE ISI fdbk 10 B. Univ. Autonoma de Puebla FCC-BUAP-R2 INDIAN STATISTICAL INSTITUTE ISI nofdbk article .6.0 B. Univ. Autonoma de Puebla FCC-BUAP-R1 INDIAN STATISTICAL INSTITUTE ISI nofdbk article .4.1 INDIAN STATISTICAL INSTITUTE ISI nofdbk article .2.0
Conclusions
In this paper we have presented details about the implementation of an information retrieval system which was used to evaluate the task of focused retrieval of XML documents, in particular, in the Data-Centric track of the Initiative for the Evaluation of XML retrieval (INEX 2010).
226
D. Vilari˜ no et al.
We presented an indexing method based on an inverted index with XML tags embedded. For each category (movies, actors, producers, directors and others), we constructed an independent inverted index. The dictionary of the index considered both, the category and the indexed term which we assumed to be sufficient to correctly identify the specific part of the XML file associated to the topic. Based on the low scores obtained, we may conclude that a more detailed description in the dictionary (including more tags of the XML hierarchy) is needed in order to improve the precision of the information retrieval system presented.
References 1. Wang, Q., Li, Q., Wang, S., Du, X.: Exploiting semantic tags in XML retrieval. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2009. LNCS, vol. 6203, pp. 133–144. Springer, Heidelberg (2010) 2. Wang, Q., Trotman, A.: Task description of INEX 2010 Data-Centric track. In: Proc. of INEX 2010 (2010) 3. Amer-Yahia, S., Curtmola, E., Deutsch, A.: Flexible and efficient XML search with complex full-text predicates. In: Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data, pp. 575–586. ACM, New York (2006) 4. Manning, C.D., Raghavan, P., Sch¨ utze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2009)
Overview of the INEX 2010 Interactive Track Nils Pharo1, Thomas Beckers2, Ragnar Nordlie1, and Norbert Fuhr2 1
Faculty of Journalism, Library and Information Science, Oslo University College, Norway [email protected], [email protected] 2 Department of Computer Science and Applied Cognitive Science, University of Duisburg-Essen, Germany [email protected], [email protected]
Abstract. In the paper we present the organization of the INEX 2010 interactive track. For the 2010 experiments the iTrack has gathered data on user search behavior in a collection consisting of book metadata taken from the online bookstore Amazon and the social cataloguing application LibraryThing. The collected data represents traditional bibliographic metadata, user-generated tags and reviews and promotional texts and reviews from publishers and professional reviewers. In this year’s experiments we designed two search task categories, which were set to represent two different stages of work task processes. In addition we let the users create a task of their own, which is used as a control task. In the paper we describe the methods used for data collection and the tasks performed by the participants.
1 Introduction The INEX interactive track (iTrack) is a cooperative research effort run as part of the INEX Initiative for the Evaluation of XML retrieval [1]. The overall goal of INEX is to experiment with the potential of using XML to retrieve relevant parts of documents. In recent years, this has been done through the provision of a test collection of XML-marked Wikipedia articles. The main body of work within the INEX community has been the development and testing of retrieval algorithms. Interactive information retrieval (IIR) [2] aims at investigating the relationship between end users of information retrieval systems and the systems they use. This aim is approached partly through the development and testing of interactive features in the IR systems and partly through research on user behavior in IR systems. In the INEX iTrack the focus over the years has been on how end users react to and exploit the potential of IR systems that facilitate the access to parts of documents in addition to the full documents. The INEX interactive track was run for the first time in 2004, in the first two years the collection consisted of journal articles from IEEE computer science journals [3, 4]. In 2006/7 [5] and 2008 [6] the Wikipedia corpus was used. In 2009 [7] the iTrack switched to a collection consisting of book metadata compiled from the bookstore Amazon and the social cataloguing application LibraryThing. S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 227–235, 2011. © Springer-Verlag Berlin Heidelberg 2011
228
N. Pharo et al.
Throughout the years the design of the iTrack experiments has been quite similar: • • • •
a common subject recruiting procedure a common set of user tasks and data collection instruments such as interview guides and questionnaires a common logging procedure for user/system interaction an understanding that collected data should be made available to all participants for analysis
In this way the participating institutions have gained access to a rich and comparable set of data on user background and user behavior, with a relatively small investment of time and effort. The data collected has been subjected to both qualitative and quantitative analysis, resulting in a number of papers and conference presentations ([8], [9], [10], [11], [12], [13], [14], [15]). In 2009, it was felt that although the "common effort" quality of the previous years was valuable and still held potential as an efficient way of collecting user behavior data, the Wikipedia collection had exhausted its potential as a source for studies of user interaction with XML-coded documents. It was therefore decided to base the experiments on a new data collection with richer structure and more semantic markup than had previously been available. The collection was based on a crawl of 2.7 million records from the book database of the online bookseller Amazon.com, consolidated with corresponding bibliographic records from the cooperative book cataloguing tool LibraryThing. A sub-set of the same collection was used in this year’s experiments, with a change to a new IR system, of which two alternative versions were made available (a more specific description of the system and collection is given below). The records present book descriptions on a number of levels: formalized author, title and publisher data; subject descriptions and user tags; book cover images; full text reviews and content descriptions. New this year is that more emphasis is given to the distinction between publisher data and user-generated data. The two systems differ in that it is not possible to query the reviews nor the book abstracts in one of the two versions. The database was chosen with the intention to enable investigation of research questions concerning, for instance
• What is the basis for judgments on relevance in a richly structured and • • •
diverse material? What fields / how much descriptive text do users make use of / chose to see to be able to judge relevance? How do users understand and make use of structure (e.g. representing different levels of description, from highly formalized bibliographic data to free text with varying degrees of authority) in their search development? How do users construct and change their queries during search (sources of terms, use and understanding of tags, query development strategies ..)? How do users’ search strategies differ at different stages of their work task processes?
2 Tasks For the 2010 iTrack the experiment was designed with two categories of tasks constructed by the track organizers, from each of which the searchers were instructed
Overview of the INEX 2010 Interactive Track
229
to select one of three alternative search topics. In addition the searchers were invited to perform one semi-self-generated task, which would function as a control task. The two task categories were designed to be presented in contexts that reflect two different stages of a work task process [16]. The theory underlying our choice of tasks is that searchers at an early stage in the process will be in a more explorative and problemoriented mode, whereas at a later stage they will be focused towards more specific data collection. The first set of tasks was designed to let searchers use a broad selection of metadata, in particular combining topical searches with the use of review data. The tasks were thus designed to inspire users to create “polyrepresentative” [17] search strategies, i.e. to use explorative search strategies, which should give us data on query development, metadata type preference and navigation patterns. At the second stage we have attempted to simulate searchers that are in a rather mechanistic data gathering mode. The tasks have also been designed to focus on nontopical characteristics of the books. Information should typically be found in publisher's texts and possibly in user-provided tags. The self-selected task was intended to function as a “control” task for comparison with the performance of two others. The task groups are introduced in the following way: Task Group 1: The Explorative Tasks You are at an early stage of working on an assignment, and have decided to start exploring the literature of your topic. Your initial idea has led to one of the following three research needs: 1.
Find trustworthy books discussing the conspiracy theories which developed after the 9/11 terrorist attacks in New York. 2. Find controversial books discussing the climate change and whether it is man-made or not. 3. Find highly acclaimed novels that treat issues related to racial discrimination.
Task Group 2: The Data Gathering Tasks You are in a data gathering stage of an assignment and need to collect a series of books for further analysis. This has led to one of the following three research needs: 4. Find novels that won the Nobel Prize during the 1990's. 5. Find bestseller crime novels by female authors. 6. Find biographies on athletes active in the 1990's. The Semi Self-selected Task 7.
Try to find books about a specific topic or of a certain type, but do not look for a specific title you already know.
230
N. Pharo et al.
3 Participating Groups 3 research groups participated in this year’s track: Oslo University College, University of Duisburg-Essen, and University of Glasgow. Data from a total of 147 sessions performed by 49 test subjects were collected from October 2010 to January 2011. The participation was compensated for some participant with a EUR 12 Amazon voucher.
4 Research Design 4.1 Search System The experiments were conducted on a Java-based retrieval system built within the ezDL framework1, which resides on a server at and is maintained by the University of Duisburg-Essen. The collection was indexed with Apache Solr 1.4, which is based on Apache Lucene. Lucene applies a variation of the vector space retrieval model. The basis of the search system is similar to the interfaces used for previous iTracks, but the interface has been modified extensively to accommodate the new data set, and a set of new functionalities have been developed. Two versions (A and B) were developed for the experiments.
Fig. 1. The search system interface
1
http://ezdl.de, http://www.is.inf.uni-due.de/projects/ezdl/
Overview of the INEX 2010 Interactive Track
231
Figure 1 shows the interface of the system (A version). The main features available to the user are: •
•
• • •
•
• • •
The query interface provides a Google-like query field as well as additional query fields for title, author, year, abstract and reviews. When a search term is entered, the searcher can choose if he wants to search also in the reviews. The system can order the search results according to “relevance” (which books the system considers to be most relevant to your search terms), “year” (publication year of the book), or “average rating” (in the cases where quality ratings from readers were available). The system will show results twenty titles at a time, with features to assist in moving further forwards or backwards in the result list. A double click on an item in the result list will show the book details in the “Details” window. If the book has been reviewed, the reviews can be seen by clicking the “Reviews” tab at the bottom of this window. Each review shows the title, the rating, the date and the helpfulness rating. A simple click on a review extends the review by the full review text The users are instructed to determine the relevance of any examined book, as “Relevant”, “Partially relevant” or “Not relevant”, by clicking markers at the bottom of the screen. Any book decided to constitute part of the answer to the search task can be moved to a result basket by clicking the “Add to basket” button next to the relevance buttons. A “Query history” button in the right of the screen displays the query terms used so far in the current search session. A single click sets a query to the search tool. A double-click also executes this query A line of yellow dots above an item in the result list is used to indicate the system’s estimate of how closely related to the query the item is considered to be. Query terms are highlighted in the result list and the detail tool.
The B version of the search system did not allow the user to search in reviews or abstracts, i.e. no query fields for abstract and reviews were available to the user. 4.2 Document Corpus The collection contains metadata of 2 780 300 English-language books. The data has been crawled from the online bookstore Amazon and the social cataloging web site LibraryThing in February and March 2009 by the University of Duisburg-Essen. The MySQL database containing the crawled data has a size of about 190 GB.. Several millions of customer reviews were crawled. For this year’s run of the track we cleaned up the data by removing all records that do not have an image of the book cover. This was thought to be a good heuristic for removing records that only have very sparse data. After the clean-up, metadata from approximately 1.5 million books remained in the database. The records present book descriptions on a number of levels: formalized author, title and other bibliographic data; controlled subject descriptions and user-provided
232
N. Pharo et al.
content-descriptive tags; book cover images; full text reviews and publisher-supplied content descriptions. The following listing shows what items has been crawled from either Amazon or LibraryThing: Amazon ISBN, title, binding, label, list price, number of pages, publisher, dimensions, reading level, release date, publication date, edition, Dewey classification, title page images, creators, similar products, height, width, length, weight, reviews (rating, author id, total votes, helpful votes, date, summary, content) editorial reviews (source, content). LibraryThing Tags (including occurrence frequency), blurbs, dedications, epigraphs, first words, last words, quotations, series, awards, browse nodes, characters, places, subjects. 4.3 Online Questionnaires During the course of the experiment, the system presented the searchers with online questionnaires to support the analysis of the log data. The searchers were given a preexperiment questionnaire, with demographic questions such as searchers’ age, education and experience in information searching in general and in searching and buying books online. Each search task was preceded with a pre-task questionnaire, which concerned searchers’ perceptions of the difficulty of the search task, their familiarity with the topic etc. After each task, the searcher was asked to fill out a posttask questionnaire. The intention of the post-task questionnaire was to learn about the searchers’ use of and their opinion on various features of the search system, in relation to the just completed task. Each experiment sessions were closed with a postexperiment questionnaire, which elicited the searchers’ general opinion of the search system. 4.4 Relevance Assessments The searchers were instructed to indicate the relevance of the items in the result list, using a three-part relevance scale of “relevant”, “partly relevant” and “not relevant”. 4.5 “Shopping” Basket To simulate the purchase of relevant books the system provides a shopping basket feature in which searchers were asked to add books they would have purchased for solving the task. Books can be added and removed from the basket. 4.6 Logging All search sessions were logged and saved to a database. The logs register and time stamp the events in the session and the actions performed by the searcher, as well as the responses from the system.
Overview of the INEX 2010 Interactive Track
233
5 Experimental Procedure The experimental procedure for each searcher is outlined below. 1.
When recruiting searchers for the experiment, the experimenter gives the searchers the instructions for the self-selected task. 2. Experimenter briefs the searcher, and explains format of study. The searcher reads and signs the Consent Form. 3. The experimenter logs the searchers into the system. This presents the searcher with the task assignments and the questionnaire. The experimenter hands out and explains the User guidelines document. It is important to take good time to demonstrate and explain how the system works. A tutorial of the system with a training task is provided. 4. The experimenter answers questions from user. 5. The searcher selects his/her tasks from each of the two categories. In addition the self-selected task is input into the appropriate form. Tasks are rotated by the system, thus any of the three tasks may be the first to be solved by the searcher. 6. The searcher answers the Pre-experiment questionnaire provided by the system. 7. The searcher answers the Pre-task questionnaire provided by the system. 8. The task is started by clicking the link to the IR system. Each task has a duration of 15 minutes, at which point the system will tell the user time has run out. The IR system is closed by clicking the “End task” button. 9. The searcher answers the Post-task questionnaire provided by the system. 10. Steps 6-9 repeated for the two other tasks. 11. The searcher answers the Post-experiment questionnaire provided by the system. 12. At the end of the evaluation session the user presses the “Finish” button in the evaluation/questionnaire system to store his data into the database.
6 Results and Future Plans Table 1 shows the distribution of systems and tasks. As can be seen very few searchers chose task 4 (Nobel Prize winning novels), the other tasks were fairly evenly distributed. For some unknown reason one searcher performed two tasks in system A and one task in system B, although our distribution system was programmed to allocate three system B task for this user. This explains the system distribution being slightly skewed. We are currently analyzing our data, which will be presented in forthcoming conference and journal papers. A primary focus of the analysis will be searchers’ choice of sources of information for the completion of the tasks. Amongst the issues that we plan to look at is the effect of searchers’ topic knowledge and the influence of task types.
234
N. Pharo et al. Table 1. Distribution of systems and tasks Task
System
1
2
3
4
5
6
7
Total
A
14
10
2
6
8
11
26
77
B
11
10
2
13
5
6
23
70
25
20
4
19
13
17
49
147
Total
References [1] Malik, S., Trotman, A., Lalmas, M., Fuhr, N.: Overview of INEX 2006. In: Fuhr, N.,
[2] [3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
Lalmas, M., Trotman, A. (eds.) INEX 2006. LNCS, vol. 4518, pp. 1–11. Springer, Heidelberg (2007) Ruthven, I.: Interactive Information Retrieval. Annual Review of Information Science and Technology 42, 43–91 (2008) Tombros, A., Larsen, B., Malik, S.: The Interactive Track at INEX 2004. In: Fuhr, N., Lalmas, M., Malik, S., Szlávik, Z. (eds.) INEX 2004. LNCS, vol. 3493, pp. 410–423. Springer, Heidelberg (2005) Larsen, B., Malik, S., Tombros, A.: The Interactive Track at INEX 2005. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 398–410. Springer, Heidelberg (2006) Malik, S., Tombros, A., Larsen, B.: The Interactive Track at INEX 2006. In: Fuhr, N., Lalmas, M., Trotman, A. (eds.) INEX 2006. LNCS, vol. 4518, pp. 387–399. Springer, Heidelberg (2007) Pharo, N., Nordlie, R., Fachry, K.N.: Overview of the INEX 2008 Interactive Track. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2008. LNCS, vol. 5631, pp. 300–313. Springer, Heidelberg (2009) Pharo, N., Nordlie, R., Fuhr, N., Beckers, T., Fachry, K.N.: Overview of the INEX 2009 Interactive Track. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2009. LNCS, vol. 6203, pp. 303–311. Springer, Heidelberg (2010) Pharo, N., Nordlie, R.: Context Matters: An Analysis of Assessments of XML Documents. In: Crestani, F., Ruthven, I. (eds.) CoLIS 2005. LNCS, vol. 3507, pp. 238–248. Springer, Heidelberg (2005) Hammer-Aebi, B., Christensen, K. W., Lund, H., Larsen, B.: Users, structured documents and overlap: interactive searching of elements and the influence of context on search behaviour. In: Ruthven, I. et al. (eds.) Proceedings of Information Interaction in Context : International Symposium on Information Interaction in Context : IIIiX 2006, Copenhagen, Denmark, October 18-20, pp. 80-94, Royal School of Library and Information Science (2006) Pehcevski, J.: Relevance in XML retrieval: the user perspective. In: Trotman, A., Geva, S. (eds.) Proceedings of the SIGIR 2006 Workshop on XML Element Retrieval Methodology: Held in Seattle, Washington, USA, August 10, pp. 35–42. Department of Computer Science, University of Otago, Dunedin, New Zealand (2006)
Overview of the INEX 2010 Interactive Track
235
[11] Malik, S., Klas, C.-P., Fuhr, N., Larsen, B., Tombros, A.: Designing a user interface for
[12]
[13]
[14]
[15] [16] [17]
[18]
interactive retrieval of structured documents: lessons learned from the INEX interactive track? In: Gonzalo, J., Thanos, C., Verdejo, M.F., Carrasco, R.C. (eds.) ECDL 2006. LNCS, vol. 4172, pp. 291–302. Springer, Heidelberg (2006) Kim, H., Son, H.: Users Interaction with the Hierarchically Structured Presentation in XML Document Retrieval. In: Fuhr, N., Lalmas, M., Malik, S., Kazai, G. (eds.) INEX 2005. LNCS, vol. 3977, pp. 422–431. Springer, Heidelberg (2006) Kazai, G., Trotman, A.: Users’ perspectives on the Usefulness of Structure for XML Information Retrieval. In: Dominich, S., Kiss, F. (eds.) Proceedings of the 1st International Conference on the Theory of Information Retrieval, pp. 247–260. Foundation for Information Society, Budapest (2007) Larsen, B., Malik, S., Tombros, A.: A Comparison of Interactive and Ad-Hoc Relevance Assessments. In: Fuhr, N., Kamps, J., Lalmas, M., Trotman, A. (eds.) INEX 2007. LNCS, vol. 4862, pp. 348–358. Springer, Heidelberg (2008) Pharo, N.: The effect of granularity and order in XML element retrieval. Information Processing and Management 44(5), 1732–1740 (2008) Kuhlthau, C.C.: Seeking meaning: a process approach to library and information services, 2nd edn. Libraries Unlimited, Westport (2004) Larsen, B., Ingwersen, P., Kekäläinen, J.: The Polyrepresentation continuum in IR. In: Ruthven, I., et al. (eds.) Information Interaction in Context: International Symposium on Information Interaction in Context, IIiX 2006, pp. 148–162. ACM Press, New York (2006) Fuhr, N., Klas, C.-P., Schaefer, A., Mutschke, P.: Daffodil: An integrated desktop for supporting high-level search activities in federated digital libraries. In: Agosti, M., Thanos, C. (eds.) ECDL 2002. LNCS, vol. 2458, pp. 597–612. Springer, Heidelberg (2002)
Using Eye-Tracking for the Evaluation of Interactive Information Retrieval Thomas Beckers and Dennis Korbar Universit¨ at Duisburg-Essen Department of Computer Science and Applied Cognitive Science Information Engineering Duisburg, Germany {tbeckers,korbar}@is.inf.uni-due.de Abstract. In this paper we present the INEX 2010 Interactive Track experiments that have been performed with eye-tracking in addition to the other collected data (system logs and questionnaires). We present our tool AOILog for recording Areas of Interest (AOI) data for dynamic user interfaces. Finally, we show how these eye-tracking and AOI data could be used for further analysis of the user interaction with the search system.
1
Introduction
In this year’s run of the Interactive Track (iTrack) we wanted to investigate how users interact with an integrated interactive search system. The working tasks do not only rely on the classical topic aspect but also on other aspects such as book reviews or structure information. In addition to the standard experiments (see [1] for more detailed explanations of the experiment design and the data collection) in the 2010 run of the iTrack, we additionally used an eye-tracking system to record the user’s gaze data while interacting with the search system. Our main research goals are to check the assumptions of interactive IR models and to find out how users interact with an integrated search system.
2
System Description
The search system (see Figure 1) was developed at the University of DuisburgEssen. It is based on the digital library system ezDL1 . Pharo et al. provide a more detailed explanation (see [1]). The search tool offers a Google-like search field as well as advanced search fields for title, author, year, abstract and reviews (depends on the system version, see [1]). A combo box allows the user to search also in reviews with the Googlelike search field. Below this query panel the user can select fields for the sorting of the results. Furthermore, the user can choose the display style of the result 1
Live system: http://www.ezdl.de/ Developer site: http://www.is.inf.uni-due.de/projects/ezdl/
S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 236–240, 2011. c Springer-Verlag Berlin Heidelberg 2011
Using Eye-Tracking for the Evaluation of Interactive Information Retrieval
237
list. The lower half of the search tool contains the result list and the result page navigation buttons. A double-click on a result item shows book details in the detail tool. Users can indicate the relevance of an examined book as either relevant, partially relevant, or not relevant, by clicking markers at the bottom of the tool. A second tab shows reviews of the selected book. Initially the title, author, rating, date and the utility rating of the review is shown. By clicking on a review, the actual review text is added to the review. Users can mark any book as part of the answer to the search task by moving it to the basket tool. This can be performed either via drag-and-drop or by clicking the add to basket button next to the relevance buttons. A history of performed search queries is provided by the query history tool. Finally, the task tool shows the current working task.
Fig. 1. The search system interface (blue boxes: tools mentioned in the description in section 2)
3
Eye-Tracking for Interactive Information Retrieval Systems
In addition to the questionnaires and system log data we also used an eyetracking system2 to record the user’s eye gaze data. The search system provides a multi-panel layout that presents a lot of information to the user. The user does not need to explicitly interact with the system 2
SMI RED: http://www.smivision.com/en/gaze-and-eye-tracking-systems/ products/red-red250-red-500.html
238
T. Beckers and D. Korbar
to get certain information. Thus, system logs and questionnaires are not sufficient since they can only capture explicit interaction of the user with the search system (e.g. clicking on a button or selecting a tab). For analysing the eye-tracking data, the user interface of the system (see Figure 1) was divided into so called Areas of Interests (AOIs) (see Figure 2). AOIs define larger and logically connected gaze areas. These areas are used to capture not only fixations, but also more peripheral perceptions. Since some of the tools in the search system are on top of other tools and the position of user interface components changes dynamically, it is necessary to record the visibility information of tools to create dynamic AOIs automatically. The manual creation of AOIs would take too much time, thus it is generally not applicable in practice, especially when trying to analyze very dynamic components. Biedert et al. capture gaze data for elements in HTML pages to provide gazeaware text in web browsers [2]. For the ezDL desktop client a browser-based solution would not be applicable. We developed a framework called AOILog which automatically keeps track of position, visibility and size of registered Java Swing components, thus enabling us to consider even small and very dynamic areas of interest, such as result items in a scrollable list widget. Also included in the AOILog framework is a small converter application which will convert our AOI XML format into the proprietary SMI AOI format. This will enable us to use the recorded data in the statistical analysis software BeGaze which is part of SMI’s software suite. In order to use our framework with eye tracking devices by other manufacturers, one only has to implement an appropriate converter, the logging functionality on Java Swing based applications can be used out of the box. Figure 2 shows the AOIs of the search interface. We defined the following AOIs: – tools (search, details, task, query history) – query panel and query fields – search controls and result scrolling – result list – parts of the details – reviews and review sorting With these dynamic AOIs it will be possible for future analyses to investigate the use of tools and book details as well as reviews in general by the users.
4
A Model for Interactive Information Retrieval
In 2010 we started a new project called HIIR (Highly Interactive Information Retrieval)3 aiming to create an interactive information retrieval system based on efficient retrieval algorithms combined with a theoretic model for interactive IR, the IIR-PRP (Interactive Information Retrieval Probability Ranking Principle)[3]. The IIR-PRP tries to offer decision lists and information based on 3
http://www.is.inf.uni-due.de/projects/hiir/index.html.en
Using Eye-Tracking for the Evaluation of Interactive Information Retrieval
239
Fig. 2. Areas of Interest (screenshot from BeGaze analysis software)
a cost/benefit ratio. The basic idea is that users move from situation to situation. In every situation a list of choices is presented to the user, s/he then decides about each of these choices sequentially. Upon the first positive decision the user will move to a new situation. In order to use the IIR-PRP as a model to implement interactive IR systems, its assumptions have to be confirmed. Furthermore we will have to measure some of its constants in order to be able to apply the model’s ranking method. As we stated above, the IIR-PRP tries to sort available choices by use of a cost/benefit ratio. In order to apply the model to interactive IR systems we first have to measure the user’s effort for evaluating certain types of choices. By use of traditional evaluation methods, we would not be able to measure the time a user spends on evaluating a certain choice as there are always multiple choices available. By logging the time between two choices we would not be able to determine how much of this time has been spend to evaluate each of the available choices. Using an eye-tracker in conjunction with the AOI logging framework described above will enable us to measure the time a user spends on evaluating the offered choices. We will be able to measure exactly how much time a user spends gazing upon a certain available choice. Its implementation in the current INEX 2010 iTrack search system will provide us with a first indication about the validity of the IIR-PRP’s assumption concerning user behaviour while scanning a list of choices. In addition we might be able to determine if there is a connection between a result item’s textual length and the effort to evaluate that item. We will also try to analyze the effort to evaluate other objects provided by the search system (e.g. review items). This
240
T. Beckers and D. Korbar
might enable us to create a first measurement to calculate the expected effort for a given choice.
5
User Study @ iTrack 2010
For the 2010 run of the iTrack we recruited 24 participants for experiments with the additional eye-tracking support. The participation was compensated with an Amazon 12 EUR voucher. The experiments have been performed in our eye-tracking laboratory from November 2010 until December 2010. We plan to perform the analysis of the experiments later this year.
6
Conclusion and Outlook
We presented how eye-tracking data can be used for evaluation of interactive information retrieval systems. We plan to analyze the collected data (questionnaires, system logs and eyetracking data) to find out how users interact with interactive search systems. Our framework AOILog will be extended and included into ezDL. Furthermore it is planned to develop tools to make the analysis of the eye-tracking data and the linkage to the system logs easier.
References 1. Pharo, N., Beckers, T., Nordlie, R., Fuhr, N.: The INEX 2010 Interactive Track: An overview. In: Geva, S., et al. (eds.) INEX 2010. LNCS, vol. 6932, pp. 227–235. Springer, Heidelberg (2010) 2. Biedert, R., Buscher, G., Schwarz, S., M¨ oller, M., Dengel, A., Lottermann, T.: The Text 2.0 framework - writing web-based gaze-controlled realtime applications quickly and easily. In: Proceedings of the International Workshop on Eye Gaze in Intelligent Human Machine Interaction, EGIHMI (2010) 3. Fuhr, N.: A probability ranking principle for interactive information retrieval. Information Retrieval 11(3), 251–265 (2008)
Overview of the INEX 2010 Link the Wiki Track Andrew Trotman1, David Alexander1, and Shlomo Geva2 1
Department of Computer Science University of Otago Dunedin New Zealand 2 Faculty of Science and Technology Queensland University of Technology Brisbane, Australia
Abstract. The INEX 2010 Link-the-Wiki track examined link-discovery in the Te Ara collection, a previously unlinked document collection. Te Ara is structured more a digital cultural history than as a set of entities. With no links and no automatic entity identification, previous Link-the-Wiki algorithms could not be used. Assessment was also necessarily manual. In total 29 runs were submitted by 2 institutes. 70 topics were assessed, but only 52 had relevant target documents and only 45 had relevant links in the pool. This suggests that the pool was not diverse enough. The best performing run had a MAP of less than 0.1 suggesting that the algorithms tested in 2010 were not very effective.
1 Introduction and Motivation Keeping a rich hypermedia document collection such as the Wikipedia up-to-date is problematic. Each time a new document is added new links from that document into the collection are needed (and vice versa). Each time a document is deleted links to that document must be deleted. If a document changes then the links must be updated. This served as the motivation for the Link-the-Wiki track in 2007. Deletion of a document is a simple maintenance problem – simply remove link from documents that refer to the document being deleted. Update is analogous to deletion then reinsertion. Consequently the entire link-the-wiki problem can be examined from the perspective of document insertion. In fact, this is the approach taken by the track from 2007-2009. Choose a set of documents from within the collection; remove them from the collection; then measure the efficacy of automatic link discovery algorithms on those documents as if each were a new addition to the collection. These topic-documents were referred to as orphans. As early as 2007 two effective algorithms were seen, Geva’s algorithm [1] and Itakura’s algorithm [2]. Geva’s algorithm builds an index of titles of documents in the collection and searches for those titles in the topic-document. The algorithm is effective in the Wikipedia because Wikipedia documents are about entities and their names are essentially canonical and if they appear in the text they are probably accurate. S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 241–249, 2011. © Springer-Verlag Berlin Heidelberg 2011
242
A. Trotman, D. Alexander, and S. Geva
Itakura’s algorithm builds an index of links in the collection and orders these on the proportion of times the anchor-text seen as a link to the number of times it is seen as a phrase anywhere in the collection. This is essentially the strength of the anchortext as an anchor and the most likely target. The algorithms has proven problematic to implement –several implementations were seen at INEX 2009 but Trotman’s 2008 implementation remains the highest performing [3]. In 2007 the performance of algorithms was measured against the links that were in the collection before the document was orphaned. This was the same approach taken by others (for example Milne & Witten [5]). Results from evaluation in this way show extremely high performance. Otago, for example, achieved a MAP score of 0.734. Milne & Witten [5] see similar such scores using their machine learning approach. In 2008 and 2009 TREC-style manual assessment was performed. Runs were pooled and manually assessed (to completion) for accuracy. As a twist on the experiment the links present in the Wikipedia itself were added to the pools. The evaluation showed that Geva’s algorithm, Itakura’s algorithm and the Wikipedia were performing comparably and that MAP scores in previous years had been inflated by non-relevant links present in Wikipedia articles. It is not known which links those are or why there are there, but it has been speculated that the links might themselves be put in by bots using similar algorithms to those of Milne & Witten, Geva, and Itakira. During the running of the track the organizers became aware of an alternative link discovery scenario. The New Zealand Ministry for Culture and Heritage has an encyclopedia-like collection (Te Ara) that when complete “will be a comprehensive guide to the country’s peoples, natural environment, history, culture, economy, institutions and society”. It does not contain links between articles. Linking Te Ara is more complex than linking the Wikipedia for many reasons. The articles are often digital narratives and in being so do not represent entities – the controlled vocabulary of the Wikipedia is not present. Geva’s title-matching algorithm is unlikely to be effective. There are no links in the collection and so the machine learning algorithms of Milne & Witten[5] and of Itakura & Clarke [2] can’t be used.
2 Te Ara Test Set The document collection used in 2010 was the 2010 dump of Te Ara. It is a single 48MB XML file consisting of 36,715 documents. The task was to link each and every document. As such the topic set was the document collection itself. After runs were submitted a set of documents were chosen for manual assessment. First, the collection was ordered on the number of links in each document. This was then divided into 10 deciles. Finally, one work-set was built by randomly selecting one document from each decile. Seven such (non-ovelapping) work sets were assessed to completion resulting in a total of 70 assessed documents.
3 Runs In total 29 runs were submitted by 2 institutes. QUT submitted 5 runs and Otago submitted 24 runs.
Overview of the INEX 2010 Link the Wiki Track
243
3.1 Submission Format Results were submitted in the 2009 Link-the-Wiki Te Ara format, except that certain elements and attributes were made optional: • The root element, , had attributes for the participant's numeric ID, the run ID and the task (LTeAra). • The element give information about the machine on which the results were produced, and how long it took to produce them. This element was optional. • The gave an explanation of the linking algorithm. • The element contained a list of document collections used in the run. • Each topic was in a element which contained an element for each anchor-text. • One or more elements, within each element, gave the offset and the target document ID for the link. If the offset was specified, the target document ID was optional because the offsets were relative to the single XML file containing the collection. If no offset was specified it was assumed to be the start of the document. • Each topic could contain up-to 50 anchors, and each anchor could contain up-to 5 BEPs (each in different target documents). Example An example of a submission is: Intel Celeron 1.06GHz 1 1 128MB 3.04 seconds Describe the approach here, NOT in the run-id. TeAra_2010_Collection
244
A. Trotman, D. Alexander, and S. Geva
10151 12991 14270 10208 9363 DTD The DTD for the submission format was:
Overview of the INEX 2010 Link the Wiki Track
245
4 Assessment Tool A new cross-platform assessment tool was written in C/C++ using SQLite and GTK+. This tool was written in an effort to reduce the assessment time and this increase in the number of topics that could be assessed in a “reasonable” period of time. Of course, the assessment of “incoming” links was not necessary in 2010 as the entire collection was linked. A screenshot of the 2010 assessment tools is shown in Figure 1. Brief assessing instructions are given in the top window. On the left is source document with the current anchor highlighted in yellow. On the right at top is the list of links from the pool with assessed-relevant links marked in green and assessed-non-relevant links
Fig. 1. 2010 Link-the-Wiki assessment tool
246
A. Trotman, D. Alexander, and S. Geva
marked in red (un-assessed had a white background). On the right at the bottom is the target document with the best-entry-point (BEP) marked in yellow. Several changes were made to the assessment method. First, the assessor is shown the target document (and BEP). If this was not relevant then all anchors to that document must be non-relevant. This made it possible to assess several links in one click. If the target document was relevant then the assessor was shown all anchors for that document and asked to assess which were relevant. In this way the assessor was simply choosing whether the given anchor was an accurate way to link the two documents. The 2010 assessment tool does not ask the assessor to choose a BEP in the case were the document is relevant but the BEP was badly placed. There were several reasons for this decision, the strongest of which was that the results from the INEX ad hoc track BEP experiments show that BEPs are usually at (or very close to) the start of the given document (311.5 characters in 2009 [4]); but also because the pool was (supposed to be) assessed to completion and so the usefulness of the BEP could be determined by the assessor.
5 Metrics As is the informal convention at INEX, the metrics for the Link-the-Wiki track in 2010 were not published before the runs were submitted. As is also the informal convention, the metric changed in 2010. In a Link-the-Wiki run it is possible (and correct) to identify more than one anchor targeting the same document. It is also possible and correct to identify more than one target per anchor. Consequently metrics based on recall (such as un-interpolated Mean Average Precision (MAP)) are meaningless. If there is only one relevant target document, but the link-discovery algorithm identifies two different anchors for that target then what is the recall? Exactly this happens in this very document, the reference to Itakura’s algorithm and to the paper by Itakura & Clarke are different anchors for the same document. This also happens in the submitted runs. The runs were, consequently, de-duplicated by taking on the highest ranking instance of a target for a given topic and ignoring other instances of the target. The relevance of each target was then determined using the manual assessments. Then the score for each position in the results list was 1 if any target for that anchor was relevant and 0 otherwise. Mean un-interpolated Average Precision is then computed from this. This approach gives a highly optimistic evaluation because the run has 5 trials at each point in the results list and if any one trial is correct the run scores a relevant hit. It is also optimistic because the anchor and BEP are not considered (if multiple BEPs are seen in the assessments then if any-one is relevant the document is relevant). It also guarantees that recall cannot exceed 1.
Overview of the INEX 2010 Link the Wiki Track
247
6 Results In total 70 topics were assessed. 18 of these had no relevant target documents in the pool. The mean number of relevant targets per topic was 8.8 and the mean number of non-relevant targets per topic was 274.6. Topic 2919 had the most relevant targets (97). Figure 2 shows the distribution of the number of relevant targets documents per topic (solid line) and the number of relevant anchor-target pairs in the pool (dotted) ordered from most relevant targets to least. In some cases the number of relevant targets far exceeds the number of relevant links (for example, topic 25591 has 27 relevant targets but only one relevant anchor-target pair was in the pool). In cases where the number of relevant anchor-target pairs exceeds the number of relevant target documents (such as topic 15765) the assessor has identified more than one anchor as relevant to the same target document (in this case 66 relevant links to 62 relevant documents). On average there are 5.5 relevant links per document and only 11 topics have more 10 relevant links in them. Figure 3 shows the performance of all the runs submitted to the track. The best Otago run was Otago_LTeAraA2B_11 with a MAP of 0.0906. The best QUT run was QUT_LTeAraA2B_DH3 with a MAP of 0.0863. In the figure the MAP of each run is shown in brackets after the run name. As can be seen, the performance of the runs is not comparable to the high scores seen in the link-the-wiki track in previous years. Early precision is low and drops quickly – recall also that the evaluation is overly optimistic because the anchors are not evaluated (it is equivalent to previous year’s file-to-file evaluation) and that the score at each point in the results list is the best of the 5 possible targets seen there.
Fig. 2. Relevant targets per topic
248
A. Trotman, D. Alexander, and S. Geva
Fig. 3. Precision / Recall of all submitted runs
7 Conclusions and Further Work The INEX 2010 Link-the-Wiki track changed document collection from the Wikipedia to Te Ara. Te Ara is a far more difficult document collection because there are no prior links (so Itakura’s algorithms cannot be used), and the document titles are not canonical entity names (so Geva’s algorithm cannot be used). As a consequence of the increased difficulty the number of groups participating dropped to just two (Otago and QUT). In total 70 topics were manually assessed, but of those the assessors found only 52 with relevant target documents and only 45 with relevant links in the pool. This suggests that both the pool was not diverse enough and that the algorithms tested were not effective enough. This is echoed in the evaluation where the best scoring optimistic MAP is less than 0.1 and the runs perform poorly at all recall points.
Overview of the INEX 2010 Link the Wiki Track
249
Further work on Link-the-Wiki has already started at NTCIR with the crosslink track. That track is using a multi-lingual dump of the Wikipedia to examine algorithms that discover links from orphans in one language to targets in another language. This interested in further link-discovery research are referred to NTCIR.
References [1] Geva, S.: GPX: Ad-Hoc Queries and Automated Link Discovery in the Wikipedia. In: Focused Access to XML Documents, pp. 404–416. Springer, Heidelberg (2007) [2] Itakura, K.Y., Clarke, C.L.: University of Waterloo at INEX2007: Adhoc and Link-theWiki Tracks. In: Focused Access to XML Document, pp. 417–425. Springer, Heidelberg (2007) [3] Jenkinson, D., Leung, K.-C., Trotman, A.: Wikisearching and Wikilinking. In: Advances in Focused Retrieval, pp. 374–388. Springer, Heidelberg (2009) [4] Kamps, J., Geva, S., Trotman, A.: Analysis of the INEX 2009 ad hoc track results. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2009. LNCS, vol. 6203, pp. 26–48. Springer, Heidelberg (2010) [5] Milne, D., Witten, I.H.: Learning to link with wikipedia. In: Proceeding of the 17th ACM Conference on Information and Knowledge Management, pp. 509–518. ACM, Napa Valley (2008)
University of Otago at INEX 2010 Xiang-Fei Jia, David Alexander, Vaughn Wood, and Andrew Trotman Computer Science, University of Otago, Dunedin, New Zealand
Abstract. In this paper, we describe University of Otago’s participation in Ad Hoc, Link-the-Wiki Tracks, Efficiency and Data Centric Tracks of INEX 2010. In the Link-the-Wiki Track, we show that the simpler relevance summation method works better for producing Best Entry Points (BEP). In the Ad Hoc Track, we discusses the effect of various stemming algorithms. In the Efficiency Track, we compare three query pruning algorithms and discusses other efficiency related issues. Finally in the Data Centric Track, we compare the BM25 and Divergence ranking functions.
1
Introduction
In INEX 2010, University of Otago participated in the Ah Hoc, Link-the-Wiki Tracks, the Efficiency and Data Centric Tracks. In the Link-the-Wiki Track, we talk about how our linking algorithm works using the Te Ara collection and the newly developed assessment tool. In the Ad Hoc Track, we show the performance of our stemming algorithm using Genetic Algorithms and how it performs against other stemming algorithms. In the Efficiency Track, we discusses the performance of our three pruning algorithms; The first is the original topk (originally described in INEX 2009), an improved version of the topk and the heapk. Finally in the Data Centric Track, we compare the BM25 and Divergence ranking functions. In Section 2, related work is discussed. Section 3 explains how our search engine works. Section 4, 5, 6 and 7 talk about how we performed in the corresponding Tracks. The last section provides the conclusion and future work.
2 2.1
Related Work The Link-the-Wiki Track
The aim of the INEX Link-the-Wiki track is to develop and evaluate link recommendation algorithms for large hypertext corpora. Before 2009, Wikipedia was the only corpus used in the Link-the-Wiki track; the task was to link related Wikipedia documents to each other, with or without providing specific anchor locations in the source documents. In 2009, the Te Ara Encyclopedia of New Zealand was used alongside Wikipedia, and tasks included producing links within each of the two corpora, and linking articles in one corpus to articles in the other. Work has been done on the topic of hypertext link recommendation by a number of people both within the INEX Link-the-Wiki track and outside of S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 250–268, 2011. c Springer-Verlag Berlin Heidelberg 2011
University of Otago at INEX 2010
251
it. It is difficult to compare INEX-assessed algorithms with non-INEX-assessed algorithms because the assessment methodology plays a large part in the results, so this section will focus on algorithms from within INEX. For Wikipedia, the two most successful link-recommendation algorithms are due to Kelly Itakura [1] and Shlomo Geva [2]. Itakura’s algorithm chooses anchors in a new document by calculating the probability (γ) that each phrase, if found in the already-linked part of the corpus, would be an anchor. If γ exceeds a certain threshold (which may be based on the length of the document), the phrase is used as an anchor. The target for the link is chosen to be the most common target for that anchor among existing links. The formula for γ for a given phrase P is: γ=
number of occurrences of P in the corpus as a link number of occurrences of P in the corpus altogether
Geva’s algorithm simply searches for occurrences of document titles in the text of the orphan document. If such an occurrence is found, it is used as an anchor. The target of the link is the document whose title was found. 2.2
The Ad Hoc Track
In the Ad Hoc Track, we compare the performance of our stemming algorithm using Genetic Algorithms with other stemming algorithms. The S Stripper consists of three rules. These rules are given in Table 1. It uses only the first matching rule. It has improved MAP on previous INEX Ad Hoc collections, from 2006-2009. This serves as a baseline for stemmer performance, and is an example of a weak stemmer (It does not conflate many terms). Table 1. S Stripper rules. The first suffix matched on the left is replaced by the suffix on the right. ies → y es → s →
The Porter stemmer [3] has improved some runs for our search engine on previous INEX collections. It serves as an example of a strong stemmer. We use it here as a baseline for comparing stemmer performance. People have found ways to learn to expand queries using thesauruses generation or statistical methods. Jones [4] used clustering methods for query expansion. We have been unable to find any mention of symbolic learning used for stemming. A similar method for improving stemming by using term similarity information from the corpus was used by Xu and Croft [5]. Their work uses the Expected Mutual Information Measure. Instead we have used Pointwise Mutual Information and the Jaccard Index. These were chosen as the best out of a larger group of measures.
252
2.3
X.-F. Jia et al.
The Efficiency Track
The following discussion of the related work is taken from our published paper in ADCS 2010 [6]. Disk I/O involves reading query terms from a dictionary (a vocabulary of all terms in the collection) and the corresponding postings lists for the terms. The dictionary has a small size and can be loaded into memory at start-up. However, due to their large size, postings are usually compressed and stored on disk. A number of compression algorithms have been developed and compared [7,8]. Another way of reducing disk I/O is caching, either at application level or system level [9,10]. Since the advent of 64-bit machines with vast amounts of memory, it has become feasible to load both the dictionary and the compressed postings into main memory, thus eliminating all disk I/O. Reading both dictionary and postings lists into memory is the approach taken in our search engine. The processing (decompression and similarity ranking) of postings and subsequent sorting of accumulators can be computationally expensive, especially when queries contain frequent terms. Processing of these frequent terms not only takes time, but also has little impact on the final ranking results. Postings pruning at query time is a method to eliminate unnecessary processing of postings and thus reduce the number of non-zero accumulators to be sorted. A number of pruning methods have been developed and proved to be efficient and effective [11,12,13,14,15,16]. In our previous work [16], the topk pruning algorithm partially sorts the static array of accumulators using an optimised version of quick sort [17] and statically prunes postings. In this paper, we present an improved topk pruning algorithm and an new pruning algorithm based on heap data structure. Traditionally, term postings are stored in pairs of pairs. However, postings should be impact ordered so that most important postings can be processed first and the less important ones can be pruned using pruning methods [18,14,15]. One approach is to store postings in order of term frequency and documents with the same term frequency are grouped together [18,14]. Each group stores the term frequency at the beginning of the group followed by the compressed differences of the document numbers. The format of a postings list for a term is a list of the groups in descending order of term frequencies. Another approach is to pre-compute similarity values and use these pre-computed impact values to group documents instead of term frequencies [15]. Pre-computed impact values are positive real numbers. In order to better compress these numbers, they are quantised into whole numbers [19,15]. Three forms of quantisation method have been proposed (Left.Geom, Uniform.Geom, Right.Geom) and each of the methods can better preserve certain range of the original numbers [15]. In our search engine, we use pre-computed BM25 impact values to group documents and the differences of document numbers in each group are compressed using Variable Byte Coding by default. We choose to use the Uniform.Geom quantisation method for transformation of the impact values, because the Uniform.Geom quantisation method preserves the original distribution of the numbers, thus no decoding is required at query time. Each impact value is quantised into an 8-bit whole number.
University of Otago at INEX 2010
253
Since only partial postings are processed in query pruning, there is no need to decompress the whole postings lists. Skipping [12] and blocking [20] allow pseudorandom access into encoded postings lists and only decompress the needed parts. Further research work [21,22] represent postings in fixed number of bits, thus allowing full random access. Our search engine partially decompress postings list based on the worst case of the static pruning. Since we know the parameter value of the static pruning and the biggest size of an uncompressed impact value (1 byte), we can add these together to find the cut point for decompression. We can simply hold decompression after that number of postings have been decompressed.
3 3.1
System Overview Indexer
Memory management is a challenge for fast indexing. Efficient management of memory can substantially reduce indexing time. Our search engine has a memory management layer above the operating system. The layer pre-allocates large chunks of memory. When the search engine requires memory, the requests are served from the pre-allocated pool, instead of calling system memory allocation functions. The sacrifice is that some portion of pre-allocated memory might be wasted. The memory layer is used both in indexing and in query evaluation. As shown previously in [16], only a very small portion of memory is actually wasted. The indexer uses hashing with a collision binary tree for maintaining terms. We tried several hashing functions including Hsieh’s super fast hashing function. By default, the indexer uses a very simple hashing function, which only hashes the first four characters of a term and its length by referencing a pre-defined look-up table. A simple hashing function has less computational cost, but causes more collisions. Collisions are handled by a simple unbalanced binary tree. We will examine the advantages of various hashing and chaining algorithms in future work. Postings lists can vary substantially in length. The indexer uses various sizes of memory blocks chained together. The initial block size is 8 bytes and the re-size factor is 1.5 for the subsequent blocks. The indexer supports either storing term frequencies or pre-computed impact values. A modified BM25 is used for pre-computing the impact values. This variant does not result in negative IDF values and is defined thus: N (k1 + 1) tftd RSVd = log · df d t k1 (1 − b) + b × LLavg + tftd t∈q here, N is the total number of documents, and dft and tftd are the number of documents containing the term t and the frequency of the term in document d, and Ld and Lavg are the length of document d and the average length of all documents. The empirical parameters k1 and b have been set to 0.9 and 0.4 respectively by training on the previous INEX Wikipedia collection.
254
X.-F. Jia et al.
In order to reduce the size of the inverted file, we always use 1 byte to store term frequencies and pre-computed impact values. This limits to a maximum value of 255. Term frequencies which have values larger than 255 are simply truncated. Truncating term frequencies could have an impact on long documents. But we assume long documents are rare in a collection and terms with high frequencies in a document are more likely to be common words. Pre-computed impact values are transformed using the Uniform.Geom quantisation method. As shown in Figure 1, the index file has five levels of structure. In the top level, original documents in compressed format can be stored. Storing original documents is optional, but is required for focused retrieval.
!"
#
!"
#
$
$
$
Fig. 1. The index structures
Instead of using the pair of for postings, we group documents with the same term frequency (or the impact value) together and store the term frequency (or the impact value) at the beginning of each group. By grouping and impacting order documents according to term frequencies (or impact values), during query evaluation we can easily process documents with potential high impacts first and prune the less important documents at the end of the postings list. The difference of document ids in each group are then stored in increasing order and each group ends with a zero. Postings are compressed with Variable Byte coding. The dictionary of terms is split into two parts. Terms with the same prefix are grouped together in a term block. The common prefix (only the first four characters) is stored in the first level of the dictionary and the remaining are stored in the term block in the second level. The number of terms in the block is
University of Otago at INEX 2010
255
stored at the beginning of the block. The term block also stores the statistics for the terms, including collection frequency, document frequency, offset to locate the postings list, the length of the postings list stored on disk, the uncompressed length of the postings list, and the position to locate the term suffix which is stored at the end of the term block. At the very end of the index file, the small footer stores the location of the first level dictionary and other values for the management of the index. 3.2
Query Evaluation
At start-up, only the the first-level dictionary is loaded into memory by default. To process a query term, two disk reads have to be issued; The first reads the second-level dictionary. Then the offset in that structure is used to locate postings. The search engine also supports a command line option which allows loading the whole index into memory, thus totally eliminating I/O at query time. An array is used to store the accumulators. We used fixed point arithmetic on the accumulators because it is faster than the floating point. For last year INEX, we developed the topk algorithm for fast sorting of the accumulators. It uses a special version of quick sort [17] which partially sorts the accumulators. A command line option (lower-k) is used to specify how many top documents to return. Instead of explicit sorting of all the accumulators, we have developed an improved version of topk. During query evaluation, it keeps track of the current top documents and the minimum partial similarity score among the top documents. The improved topk uses an array of pointers to keep track of top documents. Two operations are required to maintain the top documents, i.e. update and insert. If a document is in the top documents and gets updated to a new score, the improved topk simply does nothing. If a document is not in the top k and gets updated to a new score which is larger than the minimum score, the document needs to be inserted into the topk. The insert operation is accomplished by two linear scans of the array of pointers; (1) the first scan locates the document which has the minimum score and swap the minimum document with the newly updated document, (2) the second finds the current minimum similarity score. Based on the topk algorithm, we have further developed a new algorithm called heapk. It uses a minimum heap to keep track of the top documents. Instead of using the minimum similarity score, heapk uses bit strings to define if a document is among the top k. The heap structure is only built once which is when the number of top slots are fully filled. If a document is in the heap and gets updated to a new score, heapk first linearly scans the array to locate the document in the heap and then partially updates the structure. If a document is not in the heap and the newly updated score is larger than the minimum score (the first pointer) in the heap, heapk partially inserts the document into the heap. The upper-K command line option is used for static pruning of postings. It specifies a value, which is the number of postings to be processed. Since only part of the postings lists is processed, there is no need to decompress the whole list. Our search engine partially decompress postings lists based on the worst
256
X.-F. Jia et al.
cast. Since we know the parameter value of upper-K and the biggest size of an uncompressed impact value (1 byte), we can add these together to find the cut point for decompression.
4
The Link-the-Wiki Track
In this year’s Link-the-Wiki track, the only corpus used was the Te Ara Encyclopedia of New Zealand. Wikipedia was abandoned as a corpus because it had become too easy for algorithms to score highly according to the metrics used by INEX. This is believed to be because of characteristics of Wikipedia that Te Ara does not possess. Te Ara is therefore of interest because it presents challenges that Wikipedia does not. It is also of interest because its maintainers (New Zealand’s Ministry of Culture and Heritage) have asked for links to be incorporated into the official, public version of their encyclopedia. This is an opportunity for these linking algorithms to be tested in a real-world application. Our participation in the Link-the-Wiki track is detailed in the rest of this section. First, the differences between Wikipedia and Te Ara are outlined, as well as the possible ways to develop linking algorithms for Te Ara. Then, our own linking algorithm is explained, and its assessment results given. Finally, our contribution to the Link-the-Wiki assessment process is explained. 4.1
Differences between Wikipedia and Te Ara
The most important difference between Wikipedia and Te Ara is that Te Ara has no existing links. The Link-the-Wiki Track has always been to take a single “orphan” (a document whose incoming and outgoing links have been removed) and produce appropriate links to and from it, using the remainder of the corpus (including any links that do not involve the orphan) as input if desired. This meant that algorithms could statistically analyse the anchors and targets of the existing links in the corpus, using that information to decide what kind of links would be appropriate for the orphan document. Itakura’s algorithm (described in Section 2) is an example of one that does so, and it has been consistently successful on Wikipedia. In Te Ara this is not possible. The problem is not merely the lack of links, but that the encyclopedia was not written with links in mind. In any body of writing there are a number of different ways to refer to a given topic, but in a hypertext corpus such as Wikipedia, writers tend to use existing article titles as “canonical names” to refer to the topics of those articles. The absence of this in Te Ara renders an approach such as Geva’s algorithm less effective. Wikipedia and Te Ara are also organised in very different ways. Te Ara is primarily a record of New Zealand history, and the discussion of any given topic may be spread among several articles, each of which may discuss other topics as well. This is especially true of topics that are relevant to both the indigenous and colonial inhabitants of New Zealand; and also topics that have been relevant over a long period of time. In Wikipedia, even such wide-ranging topics are typically centred around a single article.
University of Otago at INEX 2010
4.2
257
Adapting to the Differences in Te Ara
Without the possibility of using previous years’ best-performing algorithms directly on Te Ara, we were left with two options: we could either find a way to “map” Wikipedia documents to their closest Te Ara counterparts, and then translate Wikipedia links into Te Ara links; or we could devise a new linking algorithm that did not rely on existing links at all. We chose the latter option because, as discussed above, Te Ara is organised very differently from Wikipedia, and finding a suitable mapping would have been difficult. The algorithm we used is described below. 4.3
Algorithm
The main premise behind our linking algorithm is that Te Ara documents are less “to-the-point” than Wikipedia documents (that is, a single Te Ara article tends to touch on numerous related topics in order to “tell a story” of some sort), and therefore it is important to take into account the immediate context of a candidate anchor or entry-point, as well as the more general content of the two documents being linked. Three sets of files were created and indexed using our search engine (described in Section 3). In the first, each document was contained within a separate file. In the second, each section of each document was contained within a separate file. The third was the same, but only included the section headings rather than the body text of each section. In this way, we were able to vary the level of target-document context that was taken into account when searching for possible entry-points for a given link. Within each source document, candidate anchors were generated. Every maximal sequence of consecutive words containing no stopwords or punctuation marks was considered as a candidate anchor. The purpose of this was to avoid using large portions of sentences as anchors merely because all the words appear in the target document. For each candidate anchor, various levels of context around the anchor (document, paragraph, sentence, and clause) were extracted from the source document. Each anchor context, as well as the anchor text itself, was used to query for possible targets against whichever one of the three target file-sets provided the level of context closest in size to the source context. If a particular document (or section) appeared in the query results for the anchor text itself, and for at least one of the chosen contexts, it was used as a target for that anchor. The target was given a relevance score, which was a weighted average of the relevance scores given by BM25 for each of the different contexts’ queries, based on our estimate of their importance. 24 runs were produced by varying the following 4 parameters: – Full-document anchor context Whether or not the entire source document of an anchor was used as one of its contexts. If not, the largest level of context was the paragraph containing the anchor.
258
X.-F. Jia et al.
– Relevance summation method How the total relevance score for a link was added up. In one method, the relevance scores for a target, queried from all levels of context and from the anchor itself, were simply averaged using the predetermined weights. In the other method, the values averaged were the squared differences between the relevance scores for each context and from the anchor. The rationale for the second method was that if a target was much more relevant to the anchor context than the anchor, then a nearby anchor would probably be better than the current one. – Relevance score contribution Whether all of the weights for the anchor contexts were non-zero, or just the weight for the largest context. When a context has a weight of zero, it still contributes to the choice of targets for an anchor, but not to their scores. – Target contexts Which target contexts the anchor texts themselves were directly queried against (headings, sections or both). 4.4
Results
This section details the results of assessing the 24 runs described in Section 4.3. Figure 2 shows the mean average precisions for the BEPs produced by each run. Precision/Recall graphs are included in the track overview paper. All the full document contexts runs outperformed the paragraph context runs. This result suggests that context is important when predicting links for Te Ara, and a generalisation of the result that context matters in Focused Retrieval in general. The difference squared method for summation always worked better than the sum method, and the single relevance context worked best (in that order). This suggests that although context is important in identifying links, the best link to use is determined by using just one context. The best target context to use is the heading, followed by heading and section, then just section. This results suggests that headings are important for identifying targets – something that was show to be the case with the Wikipedia link-the-wiki. 4.5
Assessment Tool
Apart from submitting runs to Link-the-Wiki, we also took over the task of maintaining the assessment tool. Improvements have been made to the assessment tool every year. However, it is crucial to the quality of our results that the manual assessment process is made as easy as possible — it is difficult for assessors to produce reliable results if they cannot understand what they are being asked, if they do not have readily available all the information that they need to make an assessment, if they need to perform unnecessarily repetitive tasks to make assessments, or if the tool responds too slowly. Therefore, we decided to make further improvements. We rewrote the assessment tool from scratch in C++ using the cross-platform GUI library GTK+, with SQLite databases for storing assessment information.
University of Otago at INEX 2010
259
Run Context Summation Contribution Element MAP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Article Article Article Article Article Article Article Article Article Article Article Article Paragraph Paragraph Paragraph Paragraph Paragraph Paragraph Paragraph Paragraph Paragraph Paragraph Paragraph Paragraph
Diff Diff Diff Diff Diff Diff Sum Sum Sum Sum Sum Sum Diff Diff Diff Diff Diff Sum Sum Sum Sum Sum Sum Sum
Single Single Single Average Average Average Single Single Single Average Average Average Single Single Single Average Average Average Single Single Single Average Average Average
Heading Both Section Heading Both Section Heading Both Section Heading Both Section Heading Both Section Heading Both Section Heading Both Section Heading Both Section
0.0906 0.0906 0.0868 0.0868 0.0863 0.0768 0.0767 0.0703 0.0700 0.0481 0.0481 0.0136 0.0136 0.0102 0.0102 0.0102 0.0102 0.0102 0.0102 0.0102 0.0102 0.0102 0.0102 0.0102
Fig. 2. Results of the Otago runs in INEX 2010 Link-the-Wiki
This has resulted in a tool that responds to the user’s requests quickly, even for large documents containing many links to be assessed. We also made some changes to the layout of the GUI. The previous GUI only showed information about one target document at a time, whereas the new one shows a list of the titles of all target documents to be assessed, and shows the contents of the selected target document. Rather than having every link assessed, as was done previously, we only ask the assessor to assess links whose BEPs they have deemed relevant (the assumption being that an anchor cannot be relevant if its BEP is not). Figure 3 shows a screenshot of the new GUI. As well as improving the quality of assessments in 2010, we hope that our changes to the assessment tool will reveal further areas for improvement in 2011. Our assessment tool collects usage statistics, the analysis of which should help us improve the tool. Even before analysing these statistics we have been able to identify one possible area for improvement. It became clear while doing the assessment that the process would have been greatly sped up if “hints” had been provided to the assessor about whether a target was likely to be relevant. As the assessment for a particular topic progressed, the assessor could build up a list of “relevant” and
260
X.-F. Jia et al.
Fig. 3. An annotated screenshot of the 2010 assessment tool
“non-relevant” words for that topic, which would be highlighted whenever they appeared in a candidate target document, just as the Ah Hoc tool does. The assessor could ignore this if necessary, but it would help in many cases. However, it would be very important to use such a feature carefully so as not to bias the assessment process.
5 5.1
The Ad Hoc Track Learning Stemmers
We previously learnt suffix rewriting stemmers using Genetic Algorithms. The stemmer referred to as the Otago Stemmer is one created part way through this work. Here we use it to address one problem with using assessments to learn recall enhancing methods like stemming. Pooled collections rely on the result lists of the participants to restrict the list of documents to assess. When we later try to learn a recall enhancing method, finding documents which were not found by any participant cannot be rewarded by increases in Mean Average Precision. The goal of submitting runs with the Otago stemmer is to compare performance with the baselines where we can add documents to the pool. The rules of the Otago stemmer are shown in Table 2. Each rule of the stemmer uses a measure condition to ensure the length of the word is sufficient for a suffix to exist. This is taken from the Porter stemmer, and is an attempt to count the number of syllables. The measure of the word must be greater than or equal to the value for the rule. As an efficiency measure, any word to be stemmed must be longer than 3 characters. It also partitions the rules into sections. Only the first
University of Otago at INEX 2010
261
Table 2. The Otago Stemmer. Rule sections are separated by lines. Measure 0 2 4 1 4 0 0 0 4 0 4 3 0 3 4 1 1 0 0 1 2 0 2 1 3 0
Match this Replace with this shi ej ngen i dops nes sy ics e ii sr ito ng rs tie q al in ar ice s ic rs tie s f uow f uow q s que sy sl anu e f ague dz ean
successful rule in a section is used. This was learnt on the INEX 2008 Wikipedia collection. 5.2
Refining Stemmers
We sought to improve the sets of terms that stemmers conflate. Additional terms found by the stemmer are only conflated if they are similar enough to the query term. We found a threshold value for several measures using an adaptive grid search on the INEX 2008 Wikipedia collection. Pointwise Mutual Information (PMI) and the Jaccard Index were found to aid performance, and we submitted runs using them to improve the Otago stemmer. For both measures we used the term occurrences in documents as the probability distributions or sets to compare. For PMI, a threshold of 1.43 was found to give the best improvement. Only terms with similarity scores greater or equal to this were conflated. The PMI for two distributions x and y: P M I(x, y) = log
P (x, y) P (x)P (y)
262
X.-F. Jia et al.
The Jaccard Index used a parameter of 0.00023 and is given between two sets of documents A and B by: J(A, B) = 5.3
A∩B A∪B
Experimental Results
For the INEX 2010 Ad Hoc track we submitted 7 runs. Their performance is given in Table 3. These runs are combinations of stemmers and stemmer refinement. The best run uses just the S Stripper. We find the Otago stemmer provides decent performance, and Porter to hurt performance a lot. Our baseline of no stemming occurs between the Otago and Porter stemmers. Using PMI to improve the Otago stemmer proved successful. The Jaccard index on the same was less so. On the S stripper the Jaccard Index was found to harm performance excessively. We forgot to submit one run, the PMI used on the S stripper. This run has been performed locally and gives a slight decrease in performance to just using the S stripper.
Table 3. Stemming runs Rank 47 54 58 59 61 74 105
6 6.1
MAP 0.3012 0.2935 0.2898 0.2894 0.2789 0.2556 0.1102
Run Name v_sstem v_otago_w_pmi v_ostem_w_jts v_otago_stem_1 v_no_stem v_porter v_sstem_w_jts
Features S Stripper Otago Stemmer with PMI refinement Otago Stemmer with Jaccard Index refinement Otago Stemmer No Stemming Porter Stemmer S Stripper with Jaccard Index refinement
The Efficiency Track Experiments
We conducted our experiments on a system with dual quad-core Intel Xeon E5410 2.3 GHz, DDR2 PC5300 8 GB main memory, Seagate 7200 RPM 500 GB hard drive, and running Linux with kernel version 2.6.30. We conducted three sets of experiments, one for each of the topk, improved topk, and heapk algorithms. For the sets of experiments on the original topk, we used the same settings as our experiments conduced in INEX 2009. We want to compare the performance of the original topk with our improved topk and heapk algorithms. The collection used in the INEX 2010 Efficiency Track is the INEX 2009 Wikipedia collection [23].
University of Otago at INEX 2010
263
Table 4. (a) Summary of INEX 2009 Wikipedia Collection using term frequencies as impact values and without stemming. (b) Summary of INEX 2009 Wikipedia Collection using pre-computed BM25 as impact values and S-Striping for stemming. (a)
(b)
Collection Size 50.7 GB Documents 2666190 Avg Document Length 880 words Unique Words 11437080 Total Worlds 2347132312 Postings Size 1.2 GB Dictionary Size 399 MB
Collection Size 50.7 GB Documents 2666190 Avg Document Length 880 words Unique Words 11186163 Total Worlds 2347132312 Postings Size 1.5 GB Dictionary Size 390 MB
The collection was indexed twice, one for the original topk and one for improved topk and heapk. For the original topk, term frequencies were used as impact values, no words were stopped and stemming was not used. For the improved topk and heapk, pre-computed BM25 similarity scores were used as impact values and S-String stemming was used. Table 4a and 4b show the summary of the document collection and statistics for the index file. The Efficiency Track used 107 topics in INEX Ad Hoc 2010. Only title was used for each topic. All topics allow focused, thorough and article query evaluations. For the Efficiency Track, we only evaluated the topics for article ContentOnly. During query evaluation, the terms for each topic were sorted in order of the maximum impact values of the terms. For the sets of experiments on the improved topk and heapk, the whole index was loaded into memory, thus no I/O was involved at query evaluation time. For the original topk, only first-level dictionary was loaded into memory at start-up. For the three sets of experiments, we specified lower-k parameter with k = 15, 150 and 1500 as required by the Efficiency Track. For each iteration of the lower-k, we specified the upper-K of 10, 100, 1 000, 10 000, 100 000, 1 000 000. In total we submitted 54 runs. The lists of run IDs and the associated lower-k and upper-K values are shown in Table 5. Officially we submitted the wrong runs for the heapk. The runs has been corrected and are used in this paper and the MAiP measures are generated using the official assessment tools. 6.2
Results
This section talks about the evaluation and performance of our three sets of the runs, obtained from the official Efficiency Track (except for the heapk ). Figure 4 shows the MAiP measures for the original topk, improved topk and heapk. When upper-K has values of 150 and 1500, MAiP measures are much better than the upper-K 15. In terms of lower-k, MAiP measures approach constant at a value of 10 000. The best runs are 09topk-18 with a value of 0.2151, 10topk-18 with a value of 0.2304 and 10heapk-18 with a value of 0.2267 for the three algorithms respectively.
264
X.-F. Jia et al.
Table 5. The lists of run IDs and the associated lower-k and upper-K values Lower-k Upper-K Original Topk Improved Topk 15 15 15 15 15 15 150 150 150 150 150 150 1500 1500 1500 1500 1500 1500
10 100 1000 10000 100000 1000000 10 100 1000 10000 100000 1000000 10 100 1000 10000 100000 1000000
09topk-1 09topk-2 09topk-3 09topk-4 09topk-5 09topk-6 09topk-7 09topk-8 09topk-9 09topk-10 09topk-11 09topk-12 09topk-13 09topk-14 09topk-15 09topk-16 09topk-17 09topk-18
10topk-1 10topk-2 10topk-3 10topk-4 10topk-5 10topk-6 10topk-7 10topk-8 10topk-9 10topk-10 10topk-11 10topk-12 10topk-13 10topk-14 10topk-15 10topk-16 10topk-17 10topk-18
Heapk 10heapk-1 10heapk-2 10heapk-3 10heapk-4 10heapk-5 10heapk-6 10heapk-7 10heapk-8 10heapk-9 10heapk-10 10heapk-11 10heapk-12 10heapk-13 10heapk-14 10heapk-15 10heapk-16 10heapk-17 10heapk-18
The MAiP measures are about the same for the improved topk and heapk. The subtle differences are when documents have the same similarity scores and the order of these documents can be different between the improved topk and heapk. The MAiP measures of the original topk are quite different from the other two algorithms. Using term frequencies as impact values have better MAiP measures when the values of lower-k and upper-K are small while pre-computed BM25 impact values have better MAiP measures when upper-K has a value larger than 10 000. To have a better picture of the time cost for the three sets of the runs, we plotted the total evaluation times (including both CPU and I/O times) of all runs in Figure 5. The total times of both the improved topk and heapk are simply the CPU costs since the index was load into memory. For the original topk, the total times were dominated by the I/O times. Regardless of the values used for lower-k and upper-K, the same number of postings were retrieved from disk, thus causing all runs to have the same amount of disk I/O. We also plotted the CPU times of the original topk since we want to compare it with the other algorithms in terms of CPU cost. The differences of the CPU times between the original topk and the other two algorithms are the times taken for decompression of the postings lists and sorting of the accumulators. First, partial decompression was used in improved topk and heapk while the original topk did not. Second, the original topk used a special version of quick sort to
Fig. 5. Efficiency comparison
00
15
−
1
10
0
00
10 0
−
00
15
−
00
15
1
−
0
00
0
●
00
0 0 0
00
00
00
0
10
0
0
0
10
−
00
00
00
10 10
−
10
−
00 00
15
15
−
00 00
15
0
00
15
1
0
10
10
−
0.15
MAiP 0.2
0.25
09topk 10topk 10heapk
00 0
0
00
0
00
●
−
0
0
10 0
●
00
−
●
15
0
10
10
−
−
0
0
00
10
10
−
0
10
●
15 00
15 0
−
−
0
15
0
15
−
0
−
00
0
0
●
00
0
00 0
●
00
0
00
00
0
00
●
15
1
10 0
10
15
0
15
0
0
00 15
15
1
00
00
00
0.1
●
15
−
−
−
●
15 0
0
15
●
15 0
●
10 00
●
−
0
10
10
−
00 0
0
10
10
−
10
0
10
10
●
−
0
15
00 0
−
−
−
−
−
0.05
●
15 0
1
0
00
15
15
15
15
15
15
0
●
15 0
−
0
0
00
10100 12100
●
15
10
10
−
●
−
10 00
8100
●
15
−
10
10 0
6100
Milliseconds
4100
●
15
−
−
2100 ●
15
15
15
100
University of Otago at INEX 2010
● ● ●
● ●
● ●
● ●
● ●
●
Fig. 4. MAiP measures for the original topk, improved topk and heapk
● ● ●
● ●
09topk (total time) 09topk (CPU Time Only) 10topk 10heapk
265
266
X.-F. Jia et al.
partially sort all accumulators while the improved topk and heapk only keep track of the top documents only the final top documents got sorted. For the original topk, the value of lower-k has no effect on the CPU cost, and values of 10 000 or above for upper-K causes more CPU usage. For the improved topk, it performs the best when lower-k has a value of 15 and 150. However, for the set of the runs where the value of lower-k is 1500, the performance of the improved topk grows exponentially. This is caused by the linearly scans of the array of pointers to insert a new document into the top k. For the runs when lower-k has a value of 15 and 150, the heapk has a small overhead compared with the improved topk, especially when upper-K has a large value. Well, the heapk performs the best when both lower-k and upper-K have large values.
7
The Data Centric Track
The collection used in the INEX 2010 Efficiency Track is the 2010 IMDB collection. The collection was indexed twice. The first index used pre-computed BM25 similarity scores as the impact values and the second used pre-computer Divergence similarity scores [24] as the impact values. For both indexes, no words were stopped and S-String stemming was used. Table 6 shows the results. With the ranking shown as (position / total runs), the results suggest that BM25 is a better ranking function than Divergence from Randomness for this collection, it consistently performed better regardless of the measure. They also suggest that BM25 whole document ranking is effective with our best run consistently in the top 6 regardless of how it is measured. We believe that, as is already the case in the ad hoc track, BM25 document ranking should be used as a baseline in future years in the document centric track. Table 6. Effectiveness measure for the Data Centric Track Run ID
MAgP
MAiP
MAP
DC-BM25 0.2491 (#1/14) 0.1550 (#6/29) 0.3397 (#5/29) DC-DIVERGENCE 0.1561 (#5/14) 0.1011 (#3/29) 0.2103 (#14/29)
8 8.1
Conclusion and Future Work The Link-the-Wiki Track
We have generated a number of runs for Te Ara. Given the inapplicability of Itakura and Geva’s algorithms to Te Ara (see Section 4.1), we believe that this year’s results are a step in the right direction towards a successful solution of what is still an unsolved problem: link recommendation in a corpus that has no existing links.
University of Otago at INEX 2010
8.2
267
The Ad Hoc Track
We find that the S stripper is hard to beat. However it is possible to use machine learning to create a good stemmer. Furthermore such stemmers seem amenable to improvement using collection statistics. Of those PMI is a good measure to use. It was also found to be the best locally. This also confirms previous findings that Porter can have a variable effect on performance. Improvement using term similarity can also harm performance. We had seen this before when finding the parameters to use, so perhaps that might have been the consequence for the Jaccard Index. Of course, this refinement can only prevent terms from being stemmed together, so using it on such a weak stemmer would be expected to not do so well. 8.3
The Efficiency Track
We compared three of our query pruning algorithms. The original topk uses a special version of quick sort to sort all accumulators and return the top k documents. Instead of explicitly sorting all accumulators, the improved topk keeps tracks of the current top k documents and finally the top k documents are sorted and returned. Based on the improved topk, we have developed heapk which essentially is a minimum heap structure. The heapk algorithm has small overhead compared with the improved topk when the values of lower-k and upper-K are small. However, the heapk outperforms the improve topk for large values of lower-k and upper-K.
References 1. Huang, D., Xu, Y., Trotman, A., Geva, S.: Overview of inex 2007 link the wiki track. In: Fuhr, N., Kamps, J., Lalmas, M., Trotman, A. (eds.) INEX 2007. LNCS, vol. 4862, pp. 373–387. Springer, Heidelberg (2008) 2. Geva, S.: Gpx: Ad-hoc queries and automated link discovery in the wikipedia. In: Fuhr, N., Kamps, J., Lalmas, M., Trotman, A. (eds.) INEX 2007. LNCS, vol. 4862, pp. 404–416. Springer, Heidelberg (2008) 3. Porter, M.: An algorithm for suffix stripping. Program 14(3), 130–137 (1980) 4. Spärck Jones, K.: Automatic Keyword Classification for Information Retrieval. Archon Books (1971) 5. Xu, J., Croft, W.B.: Corpus-based stemming using cooccurrence of word variants. ACM Trans. Inf. Syst. 16(1), 61–81 (1998) 6. Jia, X.F., Trotman, A., O’Keefe, R.: Efficient accumulator initialisation. In: Proceedings of the 15th Australasian Document Computing Symposium (ADCS 2010), Melbourne, Australia (2010) 7. Trotman, A.: Compressing inverted files. Inf. Retr. 6(1), 5–19 (2003) 8. Anh, V.N., Moffat, A.: Inverted index compression using word-aligned binary codes. Inf. Retr. 8(1), 151–166 (2005) 9. Baeza-Yates, R., Gionis, A., Junqueira, F., Murdock, V., Plachouras, V., Silvestri, F.: The impact of caching on search engines. In: SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 183–190. ACM, New York (2007)
268
X.-F. Jia et al.
10. Jia, X.F., Trotman, A., O’Keefe, R., Huang, Z.: Application-specific disk I/O optimisation for a search engine. In: PDCAT 2008: Proceedings of the 2008 Ninth International Conference on Parallel and Distributed Computing, Applications and Technologies, pp. 399–404. IEEE Computer Society, Washington, DC (2008) 11. Buckley, C., Lewit, A.F.: Optimization of inverted vector searches, pp. 97–110 (1985) 12. Moffat, A., Zobel, J.: Self-indexing inverted files for fast text retrieval. ACM Trans. Inf. Syst. 14(4), 349–379 (1996) 13. Tsegay, Y., Turpin, A., Zobel, J.: Dynamic index pruning for effective caching, pp. 987–990 (2007) 14. Persin, M., Zobel, J., Sacks-Davis, R.: Filtered document retrieval with frequencysorted indexes. J. Am. Soc. Inf. Sci. 47(10), 749–764 (1996) 15. Anh, V.N., de Kretser, O., Moffat, A.: Vector-space ranking with effective early termination, pp. 35–42 (2001) 16. Trotman, A., Jia, X.F., Geva, S.: Fast and effective focused retrieval. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2009. LNCS, vol. 6203, pp. 229–241. Springer, Heidelberg (2010) 17. Bentley, J.L., Mcilroy, M.D.: Engineering a sort function (1993) 18. Persin, M.: Document filtering for fast ranking, pp. 339–348 (1994) 19. Moffat, A., Zobel, J., Sacks-Davis, R.: Memory efficient ranking. Inf. Process. Manage. 30(6), 733–744 (1994) 20. Moffat, A., Zobel, J., Klein, S.T.: Improved inverted file processing for large text databases, pp. 162–171 (1995) 21. Anh, V.N., Moffat, A.: Random access compressed inverted files. In: Australian Computer Science Comm.: Proc. 9th Australasian Database Conf. ADC, vol. 20(2), pp. 1–12 (February 1998) 22. Anh, V.N., Moffat, A.: Compressed inverted files with reduced decoding overheads, pp. 290–297 (1998) 23. Schenkel, R., Suchanek, F., Kasneci, G.: YAWN: A semantically annotated wikipedia xml corpus (March 2007) 24. Amati, G., Van Rijsbergen, C.J.: Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Trans. Inf. Syst. 20(4), 357–389 (2002)
Overview of the INEX 2010 Question Answering Track (QA@INEX) Eric SanJuan1 , Patrice Bellot1 , V´eronique Moriceau2 , and Xavier Tannier2 1
LIA, Universit´e d’Avignon et des Pays de Vaucluse (France) {patrice.bellot,eric.sanjuan}@univ-avignon.fr 2 LIMSI-CNRS, University Paris-Sud 11 (France) {moriceau,xtannier}@limsi.fr
Abstract. The INEX Question Answering track (QA@INEX) aims to evaluate a complex question-answering task using the Wikipedia. The set of questions is composed of factoid, precise questions that expect short answers, as well as more complex questions that can be answered by several sentences or by an aggregation of texts from different documents. Long answers have been evaluated based on Kullback Leibler (KL) divergence between n-gram distributions. This allowed summarization systems to participate. Most of them generated a readable extract of sentences from top ranked documents by a state-of-the-art document retrieval engine. Participants also tested several methods of question disambiguation. Evaluation has been carried out on a pool of real questions from OverBlog and Yahoo! Answers. Results tend to show that the baselinerestricted focused IR system minimizes KL divergence but misses readability meanwhile summarization systems tend to use longer and standalone sentences thus improving readability but increasing KL divergence.
1
Introduction
The INEX QA 2009-2010 track [1] aimed to compare the performance of QA, XML/passage retrieval and automatic summarization systems on special XML enriched dumps of the Wikipedia : the 2008 annotated Wikipedia [2] used in the INEX ad-hoc track in 2009 and 2010. Two types of questions were considered. The first type was factual questions which require a single precise answer to be found in the corpus if it exists. The second type consisted of more complex questions whose answers required a multidocument aggregation of passages with a maximum of 500 words exclusively. Like for the 2010 ad-hoc restricted focus task, systems had to make a selection of the most relevant information, the maximal length of the abstract being fixed. Therefore focused IR systems could just return their top ranked passages meanwhile automatic summarization systems need to be combined with a document IR engine. The main difference between the QA long type answer task and the ad-hoc restricted focus one is that in QA, readability of answers[3] is as important as the informative content. Both need to be evaluated. Therefore answers S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 269–281, 2011. c Springer-Verlag Berlin Heidelberg 2011
270
E. SanJuan et al.
cannot be any passage of the corpus, but at least well formed sentences. As a consequence, informative content of answers cannot be evaluated using standard IR measures since QA and automatic summarization systems do not try to find all relevant passages, but to select those that could provide a comprehensive answer. Several metrics have been defined and experimented with at DUC [4] and TAC workshops [5]. Among them, Kullback-Leibler (KL) and Jenssen-Shanon (JS) divergences have been used [6] to evaluate the informativeness of short summaries based on a bunch of highly relevant documents. In this edition we used the KL one to evaluate the informative content of the long answers by comparing their n-gram distributions with those from 4 highly relevant Wikipedia pages. In 2009 a set of encyclopedic questions about ad-hoc topics was released[1]. The idea was that informativeness of answers of encyclopedic questions could be evaluated based on the ad-hoc qrels[7]. This year, a set of “real” questions from Over-Blog1 website logs not necessarily meant for the Wikipedia was proposed. A state of the art IR engine powered by Indri was also made available to participants. It allowed the participation of seven summarization systems for the first time at INEX. These systems only considered long type answers and have been evaluated on the 2010 subset. Only two standard QA systems participated to the factual question sub-track. Therefore most of QA@INEX 2010 results are about summarization systems versus a state of the art restricted focused IR system. The rest of the paper is organized as follows. First, the focused IR system used as baseline is introduced in Section 2. It is described in (§2.1) and evaluated in (§2.2). Section 3 details the collection of questions (§3.1-3.3) and available reference texts for assessments (§3.5). Section 4 explains the final choice of metrics used for the evaluation of the informativeness of long answers after several experiments. Results are reported in Section 5. Finally, Section 6 discusses our findings and draws perspectives for next year edition.
2
Baseline System: Restricted Focused IR System
Several on-line resources have been made available to facilitate participation and experiment the metrics. These resources available via a unique web interface at http://termwatch.es/Term2IR included: 1. 2. 3. 4.
a document index powered by Indri, a sentence and Part of Speech tagger powered by the TreeTagger, a summarization and Multi-Word Term extractor powered by TermWatch, a tool for automatic evaluation of summary informativeness powered by FRESA, 5. links to document source on the TopX web interface. 2.1
Features
The system allows to test on the INEX 2009 ad-hoc corpus the combination of a simple IR passage retrieval system (Indri Language Model) with a baseline summarization system (a fast approximation of Lexrank). 1
http://www.over-blog.com/
Overview of the INEX 2010 Question Answering Track (QA@INEX)
271
Different outputs are available. The default is a selection of relevant sentences with a link towards the source document in TopX. Sentences have been selected following approximated LexRank scores among the 20 top ranked passages returned by Indri using a Language Model over INEX 2008 corpus. Multiword terms extracted by shallow parsing are also highlighted. A second possible output gives a baseline summary with less than 500 words, made of the top ranked sentences. The Kullback-Leibler divergence between distributions of n-grams in the summary and in the passages retrieved by Indri are also shown. They are computed using the FRESA package. It is also possible to test any summary against this baseline. Finally, the passages retrieved by Indri are available, in several formats: raw results in native INEX XML format, raw text, POS tagged text with TreeTagger. Questions and queries can be submitted in plain text or in Indri language. The following XML tags have been indexed and can be used in the query: b, bdy, category, causal agent, country, entry, group, image, it, list, location, p, person, physical entity, sec, software, table, title. These are examples of correct queries: – Who is Charlie in the chocolate factory? – #1(Miles davis) #1(Charles Mingus) collaboration – #1(Charles Mingus).p, #combine[p](Charles Mingus) 2.2
Evaluation on the 2010 Restricted Focus Ad-Hoc Task
Let us first give some details on this restricted focus system. As stated before it starts by retrieving n documents using an Indri language model. These sentences are then segmented into sentences using shallow parsing. Finally sentences are ranked using a fast approximation of LexRank. Basically, we only consider sentences that are at distance two from the query in the intersection graph of sentences. These are sentences that share at least one term with the query, or with another sentence that shares it. The selected sentences are then ranked by entropy. We evaluated this baseline system on the Ad-hoc restricted focused task, by setting n = 100. We then retrieve for each sentence all passages in which the same word sequence appears, with possible insertions. We return the first 1000 characters. The precision/recall function of this system starts high compared to other participant runs. It gets among automatic runs, the third char precision (0.3434) and the best iP[0.01] with a value of 0.15 (0.1822 for the best manual run).
3
Sets of Questions and References
A total set of 345 questions has been made available. There are four categories of questions: 1. factual and related to 2009 ad-hoc topics (151), 2. complex and related to 2009 ad-hoc topics (85),
272
E. SanJuan et al.
3. factual from Over-Blog logs (44), 4. complex from Over-Blog logs (70) This year evaluation has been carried out on the fourth category. Answers for the first category are available. A run is also available for categories 1 and 3. Informativeness of answers to questions in category 3 can be partially evaluated based on qrel from ad-hoc 2009 INEX track. 3.1
Encyclopedic vs. General Questions
236 questions are related to 2009 INEX ad-hoc topics. Most of the remaining questions come from a sample of the log files from the search engine on OverBlog. These are real questions submitted to their website by visitors looking for answers among the blogs hosted on their website. We have selected a subset of these questions such that there exists at least a partial answer in the Wikipedia 2008. Then we have mixed these questions with others from Yahoo! Answers website2 . We considered three different types of questions: short single, short multiple and long. 3.2
Short Type Questions
Those labeled short single or short multiple are 195 and both require short answers, i.e. passages of a maximum of 50 words (strings of alphanumeric characters without spaces or punctuations) together with an offset indicating the position of the answer. Short single questions should have a single correct answer, e.g. question 216: Who is the last olympic champion in sabre? whereas multiple type questions will admit multiple answers (question 209: What are the main cloud computing service providers? ). For both short types, participants had to give their results as a ranked list of maximum 10 passages from the corpus together with an offset indicating the position of the answer. Passages had to be self-contained to decide if the answer is correct or not. Besides, we collected manually answers for the first category (factual and related to 2009 ad-hoc topics) in the Wikipedia INEX collection. These answers will be made available as a development set for 2011 campaign. Moreover, a sample run is available for all factual questions. This run has been produced by FIDJI, an open-domain QA system for French and English [8]. This system combines syntactic information with traditional QA techniques such as named entity recognition and term weighting in order to validate answers through different documents. Question analysis in FIDJI turns the question into a declarative sentence. It also aims to identify the syntactic dependencies, the expected type(s) of the answer (named entity type) and the question type (factoid, definition, complex, list questions). 2
http://answers.yahoo.com/
Overview of the INEX 2010 Question Answering Track (QA@INEX)
3.3
273
Long Type Questions
Long type questions require long answers up to 500 words that must be selfcontained summaries made of passages extracted from the INEX 2009 corpus. Are considered as words any sequence of letters and digits. An example of a long type question is (#196): What sort of health benefit has olive oil? There can be questions of both short and long types, for example a question like Who was Alfred Nobel? can be answered by “a chemist” or by a short biography. However, most of the selected long type questions are not associated with obvious name entities and require at least one sentence to be answered. 3.4
Format Submission
The submission format has been simplified to follow INEX TREC ad-hoc format: Q0
where: – the first column is the topic number. – the second column currently unused and should always be Q0. – the third column is the file name (without .xml) from which a result is retrieved, which is identical to the ¡id¿ of the Wikipedia document. – the fourth column is the rank the result is retrieved, and fifth column shows the score (integer or floating point) that generated the ranking. – the sixth column is called the ”run tag” and should be a unique identifier for the group AND for the method used. The remaining three columns depend on the question type (short or long) and on the chosen format (text passage or offset). For textual content, raw text is given without XML tags and without formatting characters. The resulting word sequence has to appear in the file indicated in the third field. This is an example of such output: 1 Q0 3005204 1 0.9999 I10Run1 The Alfred [...] societies. 1 Q0 3005204 2 0.9998 I10Run1 The prize [...] Engineers. 1 Q0 3005204 3 0.9997 I10Run1 It has [...] similar spellings. An Offset Length format (FOL) can also be used. In this format, passages are given as offset and length calculated in characters with respect to the textual content (ignoring all tags) of the XML file. File offsets start counting from 0 (zero). Previous example would be the following in FOL format: 1 Q0 3005204 1 0.9999 I10Run1 256 230 1 Q0 3005204 2 0.9998 I10Run1 488 118 1 Q0 3005204 3 0.9997 I10Run1 609 109 This would mean that results are from article 3005204. The first passage starts at the 256th character (so 257 characters beyond the first character), and has a length of 230 characters. In the case of short type questions, we use an extra field that indicates the position of the answer in the passage. This position is given by counting the number of words before the detected answer.
274
E. SanJuan et al.
3.5
Reference Texts
For each question we have selected four highly relevant Wikipedia pages from which we have extracted the most relevant sections. Questions for which there were too few relevant passages were not submitted to participants. These passages that were not publicly available have been then used as reference text to evaluate long type answers using KL divergence.
4
Evaluation of Long Answers
Only long answer evaluation is presented. As short answer runs come from organizers’ systems, we decided not to evaluate them but they will be made available for future participants. The informative content of the long type answers are evaluated by comparing the several n-gram distributions in participant extracts and in a set of relevant passages selected manually by organizers. We followed the experiment in [6] done on TAC 2008 automatic summarization evaluation data. This allows to evaluate directly summaries based on a selection of relevant passages. Given a set R of relevant passages and a text T , let us denote by pX (w) the probability of finding an n-gram w from the Wikipedia in X ∈ {R, T }. We use standard Dirichlet smoothing with default µ = 2500 to estimate these probabilities over the whole corpus. Word distributions are usually compared using one of these functions: – Kullback Leibler (KL) divergence:
KL(pT , pR ) =
w∈R∪T
pT (w) × log2
pT (w) pR (w)
– Jensen Shannon (JS) divergence: JS(pT , pR ) =
1 (KL(pT , pT ∪R ) + KL(pR , pT ∪R )) 2
In [6], the metric that obtained best correlation scores with ROUGE semiautomatic evaluations of abstracts used in DUC and TAC was JS. However, we have observed that JS is too sensitive to abstract size; therefore we finally used KL divergence to evaluate informative content reference texts or passages. We used the FRESA package3 to compute both KL and JS divergences between n-grams (1 ≤ n ≤ 4). This package also allows to consider skip n-grams. Evaluating informative content without evaluating readability does not make sense. It clearly appears that if readability is not considered then the best summarizer would be the random summarizer on n-grams which certainly minimizes KL divergence but produces incomprehensible texts. The readability and coherence are evaluated according to “the last point of interest” in the answer which is the counterpart of the “best entry point” in 3
http://lia.univ-avignon.fr/fileadmin/axes/TALNE/Ressources.html
Overview of the INEX 2010 Question Answering Track (QA@INEX)
275
Table 1. Cumulative KL divergence for best runs per participant ID 98 92 860 857 855 943 557
Specificity Focused IR MWT expansion System combination Question reformulation Semantic expansion Long sentences JS minimization
unigrams 1599.29 1617.35 1617.6 1621.14 1625.76 1642.93 1631.29
bigrams 4 skip grams 2207.56 2212.49 2226.61 2231.56 2227.37 2232.43 2234.57 2239.85 2235.21 2240.35 2252.28 2257.22 2237.61 2242.83
average 2006.45 2025.17 2025.8 2031.85 2033.77 2050.81 2037.24
readability 1/5 2/5 3/5 3/5 3/5 4/5 3/5
INEX ad-hoc task. It requires a human evaluation by organizers and participants where the assessor indicates where he misses the point of the answers because of highly incoherent grammatical structures, unsolved anaphora, or redundant passages.
5
Results
We received runs for long type questions from seven participants. All of these participants generate summaries by sentence extraction. This helps readability even if it does not ensure general coherence. Extracts made of long sentences without anaphora are often more coherent but have higher KL scores. To retrieve documents, all participants used the IR engine powered by Indri, available at track resources webpage4 . Table 1 shows results based on KL divergence on long-type questions from OverBlog logs. The cumulative divergence is the sum of KL scores between participant extracts and selected pages. As expected, baseline-restricted focused IR system (98) minimizes KL divergence but the resulting readability is poor. Meanwhile the system (943) having best readability favors long sentences and gets highest divergence figures. The most sophisticated summary approach is the Cortex system (860) which reaches a compromise between KL divergence and readability. But query formulation to retrieve documents looks also important, the approach based on query enrichment with related MultiWord Terms (92) automatically extracted from top ranked documents, gets similar divergence scores. Meanwhile this is a system slightly adapted from the focused IR system used in previous INEX 2008 and 2009 ad-hoc track [9,10]. Surprisingly sentence JS minimization (557) does not seem to minimize overall KL divergence. This system ranks sentences in decreasing order according to their JS divergence with the query and the retrieved documents. Only score differences between the baseline and the other systems are significant, as shown in Table 2.
4
http://qa.termwatch.es/
276
E. SanJuan et al. Table 2. Probabilities of signed t-tests over KL divergence scores ID 92 860 857 855 943 557 98 * 0.0400 * 0.0149 ** 0.0028 * 0.0172 **** 0.0005 *** 0.0000 92 0.3777 0.1037 0.2304 0.0821 0.0529 860 0.1722 0.4442 0.1497 0.1104 857 0.2794 0.5047 0.1788 943 0.1798 0.1013 557 0.1857
The standard deviation among systems KL divergences varies. The ten question minimizing standard deviation and, therefore, getting most similar answers among systems are: 2010044 2010107 2010096 2010066 2010062 2010083 2010046 2010047 2010005 2010049
What happened to the president of Rwanda death? What are the symptoms of a tick bite? How to make rebellious teenager obey you? How much sadness is normal breakup? How much is a typical sushi meal in japan? What are the Refugee Camps in DRC? How to get Aljazera sports? How to be a chef consultant? Why did Ronaldinho signed for Barcelona? Where can I find gold sequined Christain Louboutin shoes?
All these questions contain at least one named entity that refers to a Wikipedia page. Therefore, the systems mostly built their answer based on the textual content of this page and KL divergence is not accurate enough to discriminate among them. On the contrary, the 10 following questions are the top ten that maximized standard deviation and have the greatest impact in the ranking of the systems: 2010093 Why is strategy so important today? 2010114 What is epigenetics and how does it affect the DNA/genes in all of our cells? 2010009 What does ruddy complexion mean? 2010066 What do nice breasts look like? 2010022 How to get over soul shock? 2010092 How to have better sex with your partner? 2010080 How to be physically attractive and classy? 2010014 Why is it so difficult to move an mpeg into imovie? 2010010 What do male plants look like? 2010075 WHAT IS A DUAL XD ENGINE? Clearly, these questions are not encyclopedic ones and do not refer to particular Wikipedia pages. Meanwhile partial answers exist in the Wikipedia but they are spread among several articles.
Overview of the INEX 2010 Question Answering Track (QA@INEX)
277
Figure 1 shows the KL divergence values for 4-skip n-grams over the most discriminative questions. It can be observed that cumulative KL divergence varies along questions. These variations reveal the gaps between reference documents and textual content extracted by participant systems.
! " #$% & '(
Fig. 1. KL divergence per system over the most discriminative questions
Figure 2 shows the same values but scaled in order to reveal main differences between systems.
6
Plans for Next QA Track 2011 Edition
In 2011, we will keep on addressing real-world focused information needs formulated as natural language questions using special XML annotated dumps of the Wikipedia, but we will mix questions requiring long answers and those not. 6.1
Merging Short and Long Answer Task
Contrary to long answer task, we had too few participant teams interested in short answer evaluation in 2010. We think that this is mainly due to the fact that this task required too many modifications to traditionalINEX participant systems. To avoid this problem, we plan to make the task more manageable for questions where short answers are expected. Consequently, potential short answers will be tagged in the collection and considered as traditional XML retrieval units. The provided collection will thus be enriched with named entity annotations. These entities are generally the most
278
E. SanJuan et al.
!
" # $ %
Fig. 2. scaled KL divergence per system over the most discriminative questions
Overview of the INEX 2010 Question Answering Track (QA@INEX)
279
relevant for short answers. However, the type of entities (persons, locations, organizations, etc.) will not be provided. Moreover, the information concerning the short/long type of expected answers will be removed from the input question format. Given a question, the decision whether the answer should be rather short or long is then left to the system. Concerning the evaluation, many different answers can be considered as relevant, and a manual assessment is necessary. As a short answer is now an XML tag, the global methodology should not differ from long answer task. Finally, by restricting output to XML elements and not any passage, we shall improve readability. Indeed, we have observed that a significant number of readability issues were due to incorrect sentence segmentation or mixed document structure elements like lists, tables and titles. 6.2
Towards Contextualization of Questions
In 2011, we shall build and make available a new dump of the Wikipedia adapted for QA where only document structure, sentences and named entities will be annotated. For each question, participants will have to guess the context category (for example, ”expert”, ”pupil”, ”journalist”...), provide a context as a query oriented readable summary and a list of possible answers. Summary and answers will be made of XML elements from Wikipedia dump. Real questions will be selected from OverBlog 5 website logs and Twitter.6 Participants system will have to fisrt detect the research context. Then, they shall adapt their strategy to select document nodes that match the query, by computing, for example, complexity or readability measures, expertise level scores or genre. The assumption is that context could improve the performances of Information Retrieval Systems. Modelling context in IR is considered as a long-term challenge by the community [11]. It is defined as the combination of retrieval technologies and knowledge on the context of the query and the user in a single model to provide the most suitable response compared to an information need. The contextual aspect refers to tacit or explicit knowledge concerning the intentions of users, the environment of users and the system itself. Modelling context is not an end in itself. The system must be able to decide the most adequate technologies compared to a given context, i.e.: to adapt the searching methods to the context. Of course, the user does not provide the system with the knowledge on the context of the required search. Classical IR approaches suppose a common objective which is topic relevance of top ranked documents. Opposite to this, approaches investigated by [12] allow to integrate scores orthogonal to the relevance score such as complexity. Document complexity must be matched to both users capability and level of specialisation in the target knowledge area. Different measures of difficulty can be incorporated to the relevance scores. 5 6
http://en.over-blog.com This track relates to Contextual Information Retrieval and will be supported by the French National Research Agency ANR via the project CAAS (Contextual Analysis and Adaptive Search - ANR 2010 CORD 001 02).
280
E. SanJuan et al.
The international community in information retrieval is indeed interested in research in contextual information retrieval since few years. The group IRiX (Information Retrieval in Context), proposed by Pr. Rijsbergen organizes an annual workshop on this topic since 2004. In the framework of the 6th PRCD (European commission), the network of excellence DELOS (Network of Excellence on Digital Libraries7 ) is interested in the access to information for users of various categories: individuals, specialists or population. Project PENG (Personalized News Content Programming8) is interested in defining an environment for programming and modelling the contents which makes it possible to offer interactive and personalized tools for multimedia information access to specialists or simple users.
7
Conclusion
In 2010 we provided a reusable resource for QA evaluation based on INEX adhoc 2009-2010 wikipedia document collection, including an original evaluation approach and software implementation. In 2011 we will try to deal with two types of challenges. The first challenge is defining the query features that could help in predicting the query type and extracting it automatically from the query formulation or from the environment of the user. The second challenge is to be able to help the system to disambiguate the users expectation and use it for answering the queries.
References 1. Moriceau, V., SanJuan, E., Tannier, X., Bellot, P.: Overview of the 2009 qa track: Towards a common task for qa, focused ir and automatic summarization systems. In: [7], pp. 355–365 (2009) 2. Schenkel, R., Suchanek, F.M., Kasneci, G.: Yawn: A semantically annotated wikipedia xml corpus. In: Kemper, A., Sch¨ oning, H., Rose, T., Jarke, M., Seidl, T., Quix, C., Brochhaus, C. (eds.) BTW. LNI, vol. 103, pp. 277–291 GI (2007) 3. Pitler, E., Louis, A., Nenkova, A.: Automatic evaluation of linguistic quality in multi-document summarization. In: ACL, pp. 544–554 (2010) 4. Nenkova, A., Passonneau, R.: Evaluating content selection in summarization: The pyramid method. In: Proceedings of HLT-NAACL, vol. 2004 (2004) 5. Dang, H.: Overview of the TAC 2008 Opinion Question Answering and Summarization Tasks. In: Proc. of the First Text Analysis Conference (2008) 6. Louis, A., Nenkova, A.: Performance confidence estimation for automatic summarization. In: EACL, The Association for Computer Linguistics, pp. 541–548 (2009) 7. Geva, S., Kamps, J., Trotman, A. (eds.): Focused Retrieval and Evaluation, 8th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2009, Brisbane, Australia, December 7-9, LNCS, vol. 6203. Springer, Heidelberg (2010) (Revised and selected papers) 7 8
http://www.delos.info/ http://www.peng-project.org/
Overview of the INEX 2010 Question Answering Track (QA@INEX)
281
8. Moriceau, V., Tannier, X.: FIDJI: Using Syntax for Validating Answers in Multiple Documents. Information Retrieval, Special Issue on Focused Information Retrieval 13, 507–533 (2010) 9. SanJuan, E., Ibekwe-Sanjuan, F.: Combining language models with nlp and interactive query expansion. In: [7], pp. 122–132 10. Ibekwe-Sanjuan, F., SanJuan, E.: Use of multiword terms and query expansion for interactive information retrieval. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2008. LNCS, vol. 5631, pp. 54–64. Springer, Heidelberg (2009) 11. Allan, J., Aslam, J., Belkin, N.J., Buckley, C., Callan, J.P., Croft, W.B., Dumais, S.T., Fuhr, N., Harman, D., Harper, D.J., Hiemstra, D., Hofmann, T., Hovy, E.H., Kraaij, W., Lafferty, J.D., Lavrenko, V., Lewis, D.D., Liddy, L., Manmatha, R., McCallum, A., Ponte, J.M., Prager, J.M., Radev, D.R., Resnik, P., Robertson, S.E., Rosenfeld, R., Roukos, S., Sanderson, M., Schwartz, R.M., Singhal, A., Smeaton, A.F., Turtle, H.R., Voorhees, E.M., Weischedel, R.M., Xu, J., Zhai, C.: Challenges in Information retrieval and language modeling: report of a workshop held at the center for intelligent information retrieval, university of massachusetts amherst. SIGIR Forum 37(1), 31–47 (2002) 12. Gao, J., Qi, H., Xia, X., Nie, J.Y.: Linear discriminant model for information retrieval, pp. 290–297. ACM, New York (2005)
The GIL Summarizers: Experiments in the Track QA@INEX’10 Edmundo-Pavel Soriano-Morales, Alfonso Medina-Urrea, Gerardo Sierra Mart´ınez, and Carlos-Francisco M´endez-Cruz Instituto de Ingenier´ıa Universidad Nacional Aut´ onoma de M´exico, Mexico {esorianom,amedinau,gsierram,cmendezc}@iingen.unam.mx http://www.iling.unam.mx
Abstract. In this paper we briefly describe two summarizers: Res´ umeME and GIL-UNAM-3. Both are used to extract utterances from a set of documents retrieved by means of synonym modified queries. That is, we modify each query by obtaining from the WordNet database word synonyms for each of its words. The queries are provided by the QA@INEX 2010 task. The results of the experiments are evaluated automatically by means of the FRESA system. Keywords: INEX, Automatic summarization system, QuestionAnswering system, WordNet synonyms, Res´ umeME, GIL-UNAM-3 summarizer, FRESA system.
1
Introduction
A wide variety of text extraction techniques for summarizing documents exists (see, for instance, [3,11]). Our experience with summarization systems has included mainly word information content, sentence or utterance position, and standard deviation of utterance position measurements ([2,4,5]). In this paper, we explore the implementation of two of our summarization systems: Res´ umeME and GIL-UNAM-3. In order to conduct the experiments, these summarizers rely on a) query enrichment by means of word synonyms and b) token occurrence in both the query and the document summarized. Both systems, Res´ umeME and GIL-UNAM-3, are evaluated in the context of the question-answering track of the INEX 2010 task,1 which provides questions either with short answers or with complex answers dealing with one or more utterances, possibly very long ones. Our summarizers are proposed as tools for dealing with experiments on this second challenge. The corpus used contains all English texts of Wikipedia. The idea is to retrieve relevant documents by means of synonym enriched queries. The documents are retrieved by the search engine INDRI.2 Then, by using separately Res´ umeME 1 2
http://www.inex.otago.ac.nz/ http://www.lemurproject.org/indri/
S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 282–289, 2011. c Springer-Verlag Berlin Heidelberg 2011
The GIL Summarizers: Experiments in the Track QA@INEX’10
283
and the GIL-UNAM-3 summarizer, we provide answers which are extracts from these retrieved documents. Each extract has no more than 500 words. We then evaluate the answers automatically by means of the FRESA system ([6,8,9]). The organization of this paper is as follows: in Section 2 the system Res´ umeME is described; in Section 3 we give details on the GIL-UNAM-3 summarizer; in Section 4 we present the algorithm used in order to modify queries; the experimental settings and results are presented in Section 5; and, finally, in Section 6, we briefly present our conclusions and future work.
2
The Res´ umeME Summarizer
Res´ umeME is a single-document extractive summarization system. Several unsupervised statistical measurements are used to identify relevant utterances from source text. As expected, each of them is assigned with a score: hence utterances with highest score are going to be part of the final summary. This score was calculated by means of two measures: TF-IDF and S-I. The latter is a combination of information content and standard deviation. For each word of an utterance the two measurements are calculated. Consequently, the score of an utterance is based on an average of word weights. Formally, the score of an utterance is: scoreT F −IDF (utterance) + scoreS−I (utterance) 2 Afterwards, the system chooses as many utterances with highest score as the ratio compression value allows. They are presented in the final extract in the same order as they appear in the source text. The final utterance score is calculated by means of a smoothed average of utterance words. This average gives higher value to long utterances. The subjacent idea is that long utterances contain more information. Let n be the total number of words in an utterance. The total utterance score is defined as: n T F − IDFt,e × n scoreT F −IDF = t=1 log n score(utterance) =
The S-I measurement combines word information content and standard deviation of word position as a measurement of text dispersion. This weight assigns a higher score to words with higher information content (higher standard deviation means words occur dispersed). The subjacent idea is that the most relevant words are the least frequent and most dispersed in a text. Let E be the set of utterances in the source text and T the set of words in the same text. If e E and t T , the S-I weight of a word is defined as: weightS−I (t) =
st + I(t) 2
284
E.-P. Soriano-Morales et al.
where st is the standard deviation of the positions of t and I(t) is the information content of t.
3
The GIL-UNAM-3 Summarizer
The GIL-UNAM-3 summarizer is an utterance extraction system for singledocuments. The method proposed deals with two key aspects: a document representation scheme and an utterance weighting procedure. The former is accomplished by vector representation and the latter by means of a greedy optimization algorithm. The summary is build by concatenating utterances exhibiting the highest weightings according to the algorithm. In order to accomplish this, the GIL-UNAM-3 summarizer is composed of the following stages: document transformation to vector representation, calculation of utterance weightings, and summary generation. In the first stage, a stemmer and a stop-list, both tailor-made for the English language, are used in order to filter the document tokens so that same-root forms are conflated and function words are eliminated. In the second stage, the optimization algorithm is applied to build a matrix and to estimate utterance weightings. These weightings are calculated by means of the following two values: 1. Cosine measurements of the angles related between each utterance vector and the modified query vector. These values are normalized to fall in the interval [0,1]. 2. Normalized word frequencies in both the document and the modified query (also in [0,1]). Furthermore, each utterance weighting is simply the arithmetic mean of both of these values calculated for each utterance, which is also a value in the interval [0,1]. In the last stage, the relevant utterances, those with the highest weightings, are extracted and concatenated in order to build the summary.
4
Query Modification
Query expansion is a general technique in the information retrieval field for generating alternative queries (query enrichment). For our experiments, we used it to replace terms in the original query with synonyms, in order to take advantage of additional information from the documents. Furthermore, in information extraction the most common query expansion approach is accomplished by means of thesaurus. Here we use WordNet3 , a common resource for these purposes, for obtaining lexical relations based in sets of synonyms (synsets). 3
http://wordnet.princeton.edu/
The GIL Summarizers: Experiments in the Track QA@INEX’10
285
Thus, to modify the queries, all content words in the original query were replaced by those synonyms found in the synsets of the lexical database WordNet. In fact, since reliance on WordNet for obtaining synonyms is questionable, we are actually dealing with candidate synonyms rather than real synonyms. The algorithm for replacing each word of the question with the first different word found in the synsets obtained from WordNet was implemented with Python 2.7,4 using the Natural Language Toolkit (NLTK)5 suite of libraries. This allows fast tokenization and easy access to the WordNet database. The steps of the algorithm are: 1. For each query: (a) Tokenize the original query via regular expressions keeping the order of appearance of each token. (b) Remove all tokens that contain non alphabetic characters. Also remove tokens contained in a stoplist. (c) For each remaining token: i. Search in WordNet the token and keep the first different word found in the synsets. this word will be the new token to be exchanged with the original one. (d) Replace the old tokens from the original query with the new ones, keeping the same order of appearance. This can be also described in pseudocode: Input: List of original queries OQ Output: List of modified queries M Q foreach originalquery ∈ OQ do OriginalT okens ← Tokenize(originalquery) ; N otF unctional ← RemoveFunctionalTokens(OriginalT okens); foreach token ∈ N otF unctional do Synonym ← GetSynonymFromWordNet(token); N ewT okens += Synonym; end M odif iedQuery ← ReplaceTokens(OriginalT okens, N ewT okens); N ewQueries += M odif iedQuery; end OQ ← N ewQueries; return OQ;
Algorithm 1. ModifyOriginalQueries(OQ) The most interesting method, GetSynonymFromWordNet, is described also: 4 5
http://www.python.org/about/ http://www.nltk.org/
286
E.-P. Soriano-Morales et al.
Input: token T Output: synonym S Result ← LookupW ordInW ordN et(T ); S ← T; if Result is not empty then foreach synset ∈ Result do foreach word ∈ synset do if word = T then S ← word; return S; end end end else return S; Algorithm 2. GetSynonymFromWordNet(T)
5
Experiments Settings and Results
As mentioned above, in order to evaluate the performance of both, the Res´ umeME and GIL-UNAM-3 summarizers, applied on the QA@INEX corpus, we used the FRESA system, which does not rely on human produced summaries or human validation. The results of the experiments can be observed in Tables 1, 2, 3 and 4. Values represent divergence of the summaries with respect to the original documents. Tables 1 and 2 refer to the experiment with Res´ umeME, whereas Tables 3 and 4 exhibit results of the GIL-UNAM-3 summarizer. Table 1. FRESA results for modified query number 2009071 using the Res´ umeMe summarizer Distribution type Baseline summary Empty baseline Random unigram Random 5-gram Submitted summary
unigram 14.46978 19.32973 11.94329 10.777 16.94064
bigram with 2-gap 22.27342 22.19415 27.98158 27.87586 20.80448 20.57957 18.08401 18.30523 25.61087 25.5306
Average 19.64579 25.06239 17.77578 15.72208 22.69404
Both tables for the first experiment show similar tendencies. Also, both tables for the second one exhibit the best results. It is interesting to notice, on the one hand, that random summaries (unigram and 5-gram) exhibit smaller divergence values than the summaries generated by both our summarizers. However, these summaries are unintelligible (given their randomness).
The GIL Summarizers: Experiments in the Track QA@INEX’10
287
Table 2. FRESA results for modified query number 2010062 using the Res´ umeMe summarizer Distribution type Baseline summary Empty baseline Random unigram Random 5-gram Submitted summary
unigram 16.49457 21.63229 14.95259 13.75724 20.21189
bigram with 2-gap 23.52351 23.6694 29.20944 29.34705 22.61202 22.68446 20.61196 21.02727 27.79291 27.92865
Average 21.22916 26.72959 20.08302 18.46549 25.31115
Table 3. FRESA results for modified query number 2009071 using the GIL-UNAM-3 summarizer Distribution type Baseline summary Empty baseline Random unigram Random 5-gram Submitted summary
unigram 14.46978 19.32973 11.94329 10.777 13.92082
bigram with 2-gap 22.27342 22.19415 27.98158 27.87586 20.80448 20.57957 18.08401 18.30523 21.7732 21.7724
Average 19.64579 25.06239 17.77578 15.72208 19.15548
On the other hand, the divergence values for the summarizers differ with respect to the baseline. Namely, while the Res´ umeME system exhibited a higher divergence than this baseline, the GIL-UNAM-3 summarizer showed a slightly lower one. Specifically, it can be observed that all divergence values of the summaries generated by Res´ umeME are greater than the values of the baseline, but smaller than those of the empty baseline summaries (which consist of the words in the original documents not included in our summaries). Contrastingly, it can be observed that all divergence values of the summaries generated by the GILUNAM-3 summarizer are somehow smaller than the values of the baseline and noticeably smaller than those of the empty baseline summaries. Finally, it is important to mention that the Res´ umeME summarizer is tailor made for short texts of around 10 pages and the test documents for this experiment are significantly longer. This is relevant because the minimum percentage of reduction for Res´ umeME is 10%, which generates extracts considerably longer Table 4. FRESA results for modified query number 2010062 using the GIL-UNAM-3 summarizer Distribution type Baseline summary Empty baseline Random unigram Random 5-gram Submitted summary
unigram 16.4946 21.6323 14.9526 13.7572 16.1715
bigram with 2-gap 23.52351 23.6694 29.20944 29.34705 22.61202 22.68446 20.61196 21.02727 23.47908 23.62243
Average 21.22916 26.72959 20.08302 18.46549 21.091
288
E.-P. Soriano-Morales et al.
than the required ones. Thus, we applied Res´ umeME iteratively to each document until it generated an extract of the required size. Perhaps, this accounts for the poor performance of this summarizer with respect to the performance of the GIL-UNAM-3 summarizer.
6
Conclusions
In this brief paper, we have presented experiments on the document sets made available during the Initiative for the Evaluation of XML Retrieval (INEX) 20106, in particular on the QA@INEX 2010 Task (QA@INEX) http://www. inex.otago.ac.nz/tracks/qa/qa.asp. We have described both, the Res´ umeME system and the GIL-UNAM-3 summarizer, which extract utterances from sets of documents retrieved by means of synonym modified queries. That is, we obtain from the WordNet database word synonyms for each word of the query. The systems apply several utterance selection metrics in order to extract those most likely to summarize a document; namely, TF-IDF, a combination of information content and standard deviation of utterance position, normalized cosine measurements and normalized word frequencies. As mentioned before, the queries are provided by the QA@INEX 2010 task. The results of the experiments are evaluated automatically by means of the FRESA system. Interestingly, our summarizers exhibit somehow different result tendencies. On the one hand, the one based on TF-IDF and a combination of information content and standard deviation of utterance position (Res´ umeME) performs not as well as the baseline (although, it performs better than the empty one). On the other one, the one based on normalized cosine measurements and normalized word frequencies (GIL-UNAM-3) performs somehow better than the baselines. Many adjustments can be made to these experiments. For instance, as future work, it would be interesting to modify the queries by adding synonyms, rather than replacing the query words by them, like it was done here.
References 1. Boudin, F., Torres Moreno, J.M.: NEO-CORTEX: A performant user-oriented multi-document summarization system. In: Gelbukh, A. (ed.) CICLing 2007. LNCS, vol. 4394, pp. 551–562. Springer, Heidelberg (2007) 2. Vasques, G., Ximena, M.: Sistema de resumen extractivo autom´ atico. In: Facultad de Ingenier´ıa, UNAM, Mexico (2010) 3. Mani, I., Maybury, M.T.: Automatic Text Summarization. The MIT Press, Cambridge (1999) 4. Medina, U., Alfonso: De la palabra gr´ afica al texto: sobre la extracci´ on de enunciados para el resumen autom´ atico. In: V´ azquez Laslop, M.E., Zimmermann, K., Segovia, F. (eds.) De la lengua por s´ olo la extra˜ neza: estudios de lexicolog´ıa, norma ling¨ u´ıstica, historia y literatura en homenaje a Luis Fernando Lara, El Colegio de M´exico, Mexico (forthcoming) 6
http://www.inex.otago.ac.nz/
The GIL Summarizers: Experiments in the Track QA@INEX’10
289
5. M´endez, C., Carlos, F., Medina, U., Alfonso: Extractive Summarization Based on Word Information and Sentence Position. In: Gelbukh, A. (ed.) CICLing 2005. LNCS, vol. Alfonso, pp. 653–656. Springer, Heidelberg (2005) 6. Saggion, H., Torres-Moreno, J.M., da Cunha, I., SanJuan, E., Vel´ asquez-Morales, P.: Multilingual Summarization Evaluation without Human Models. In: Proceedings of the 23rd International Conference on Computational Linguistics COLING 2010, Beijing (2010) 7. Torres Moreno, J.M., St-Onge, P.L., Gagnon, M., El-B´eze, M., Bellot, P.: Automatic Summarization System coupled with a Question-Answering System (QAAS), CoRR (2009) 8. Torres-Moreno, J.M., Saggion, H., da Cunha, I., SanJuan, E.: Summary Evaluation with and without References. Polibits Research Journal on Computer Science and Computer Engineering and Applications 42 (2010) 9. Torres-Moreno, J.M., Saggion, H., da Cunha, I., Vel´ azquez-Morales, P., SanJuan, ´ E.: Evaluation automatique de r´esum´es avec et sans r´ef´erence. In: 17e Conf´erence sur le Traitement Automatique des Langues Naturelles. TALN, Montreal (2010) 10. Torres-Moreno, J.M., Vel´ azquez-Morales, P., Jean-Guy, M.: Cortex: un algorithme pour la condensation automatique des textes. In: La Cognition Entre Individu et Soci´et´e ARCo 2001, Lyon (2001) 11. Weiss, S.M., Indurkhya, N., Zhang, T., Damerau, F.: Text Mining Predictive Methods for Analizing Unstructured Information. Springer, Heidelberg (2005)
The Cortex Automatic Summarization System at the QA@INEX Track 2010 Juan-Manuel Torres-Moreno1,2 and Michel Gagnon2 1
Universit´e d’Avignon et des Pays de Vaucluse - LIA 339, chemin des Meinajari`es, Agroparc BP 91228, 84911 Avignon Cedex 9 France 2 ´ Ecole Polytechnique de Montr´eal - D´epartement de g´enie informatique CP 6079 Succ. Centre Ville H3C 3A7 Montr´eal (Qu´ebec), Canada [email protected], [email protected] http://www.lia.univ-avignon.fr
Abstract. The Cortex system is constructed of several different sentence selection metrics and a decision module. Our experiments have shown that the Cortex decision on the metrics always scores better than each system alone. In the INEX@QA 2010 task of Long Questions, Cortex strategy system obtained very good results in the automatic evaluations FRESA. Keywords: INEX, Automatic Answering system, Cortex.
1
summarization
system,
Question-
Introduction
Automatic summarization is indispensable to cope with ever increasing volumes of valuable information. An abstract is by far the most concrete and most recognized kind of text condensation [1]. We adopted a simpler method, usually called extraction, that allow to generate summaries by extraction of pertinence sentences [2,3]. Essentially, extracting aims at producing a shorter version of the text by selecting the most relevant sentences of the original text, which we juxtapose without any modification. The vector space model [4,5] has been used in information extraction, information retrieval, question-answering, and it may also be used in text summarization. Cortex1 is an automatic summarization system, recently developed [6] which combines several statistical methods with an optimal decision algorithm, to choose the most relevant sentences. An open domain Question-Answering system (QA) has to precisely answer a question expressed in natural language. QA systems are confronted with a fine and difficult task because they are expected to supply specific information and not whole documents. At present there exists a strong demand for this kind of text processing systems on the Internet. A QA system comprises, a priori, the following stages [7]: 1
COndens´es et R´esum´es de TEXte (Text Condensation and Summarization).
S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 290–294, 2011. c Springer-Verlag Berlin Heidelberg 2011
The Cortex Automatic Summarization System at the QA@INEX Track 2010
291
– Transform the questions into queries, then associate them to a set of documents; – Filter and sort these documents to calculate various degrees of similarity; – Identify the sentences which might contain the answers, then extract text fragments from them that constitute the answers. In this phase an analysis using Named Entities (NE) is essential to find the expected answers. Most research efforts in summarization emphasize generic summarization [8,9,10]. User query terms are commonly used in information retrieval tasks. However, there are few papers in literature that propose to employ this approach in summarization systems [11,12,13]. In the systems described in [11], a learning approach is used (performed). A document set is used to train a classifier that estimates the probability that a given sentence is included in the extract. In [12], several features (document title, location of a sentence in the document, cluster of significant words and occurrence of terms present in the query) are applied to score the sentences. In [13] learning and feature approaches are combined in a two-step system: a training system and a generator system. Score features include short length sentence, sentence position in the document, sentence position in the paragraph, and tf.idf metrics. Our generic summarization system includes a set of eleven independent metrics combined by a Decision Algorithm. Query-based summaries can be generated by our system using a modification of the scoring method. In both cases, no training phase is necessary in our system. This paper is organized as follows. In Section 2 we explain the methodology of our work. Experimental settings and results are presented in Section 3. Section 4 exposes the conclusions of the paper and the future work.
2
The CORTEX System
Cortex (COndensation et R´esum´es de Textes) [14,15] is a single-document extract summarization system using an optimal decision algorithm that combines several metrics. These metrics result from processing statistical and informational algorithms on the document vector space representation. The INEX 2010 Query Task evaluation is a real-world complex question (called long query) answering, in which the answer is a summary constructed from a set of relevant documents. The documents are parsed to create a corpus composed of the query and the the multi-document retrieved by Indri. The idea is to represent the text in an appropriate vectorial space and apply numeric treatments to it. In order to reduce complexity, a preprocessing is performed to the question and the document: words are filtered, lemmatized and stemmed. The Cortex system uses 11 metrics (see [16] for a detailed description of these metrics) to evaluate the sentence’s relevance. – The frequency of words (F). – The overlap between the words of the question (R). – The entropy the words (E).
292
J.-M. Torres-Moreno and M. Gagnon
– The shape of text (Z). – The angle between question and document vectors (A). – The sum of Hamming weights of words per segment times the number of different words in a sentence. – The sum of Hamming weights of the words multiplied by word frequencies. – ... The system scores each sentence with a decision algorithm that relies on the normalized metrics. Before combining the votes of the metrics, these are partitionned into two sets: one set contains every metric λi > 0.5, while the other set contains every metric λi < 0.5 (values equal to 0.5 are ignored). We then calculate two values α and β, which give the sum of distances (positive for α and negative for β) to the threshold 0.5 (the number of metrics is Γ , which is 11 in our experiment): α=
Γ
(λi − 0.5); λi > 0.5
i=1
β=
Γ
(0.5 − λi ); λi < 0.5
i=1
The value given to each sentence is calculated with: if(α > β) then Scorecortex (s, q) = 0.5 + α/Γ else Scorecortex (s, q) = 0.5 − β/Γ In addition to this score, two other measures are used: the question-document similarity and the topic-sentence overlap. The Cortex system is applied to each document of a topic set and the summary is generated by concatenating higher score sentences. The specific similarity measure [17] between the question and the corpus allows us to re-scale the sentence scores according to the relevance of the document from which they are extracted. This measure is the normalized scalar product of the tf.idf vectorial representations of the document and the question q. Let d = (d1 . . . dn ) and q = (q1 . . . qn ) be the vectors representing the document and the question, respectively. The definition of the measure is: n i=1 di qi Similarity(q, d) = n n 2 2 i=1 di + i=1 qi The last measure, the topic-sentence overlap, assigns a higher ranking for the sentences containing question words and makes selected sentences more relevant. The overlap is defined as the normalized cardinality of the intersection between the uestion word set T and the sentence word set S. Overlap(T, S) =
card(S ∩ T ) card(T )
The Cortex Automatic Summarization System at the QA@INEX Track 2010
293
The final score of a sentence s from a document d and a question q is the following: cortex Score = α1 Score (s, q) + α2 Overlap(s, q) + α3 Similarity(d, q) where i αi = 1
3
Experiments Settings and Results
In this study, we used the document sets made available during the Initiative for the Evaluation of XML retrieval (INEX) 20102, in particular on the INEX 2010 QA Track (QA@INEX) http://www.inex.otago.ac.nz/tracks/qa/qa.asp. To evaluate the efficacity of Cortex on INEX@QA corpus, we used the FRESA package3. INEX queries. No pre-processing or modification was applied on queries set. Cortex used the query as a title of a big document retrieved by Indri. Table 1 shows an example of the results obtained by Cortex system using 50 documents as input. The query that the summary should answer in this case was the number 2010111: What is considered a beautiful body shape?. This table presents Cortex results in comparison with an the INEX baseline (Baseline summary), and three baselines, that is, summaries including random n-grams (Random unigram) and 5-grams (Random 5-gram) and empty baseline. We observe that our system is always better than Baseline summary and empty baseline. Table 1. Example of Cortex Summarization results Summary type 1-gram Baseline summary 26.679 Empty baseline 31.715 Random unigram 25.063 Random 5-gram 23.168 Cortex summary 26.421
4
2-gram 34.108 39.452 32.822 30.644 33.935
SU4-gram FRESA average 34.218 31.668 39.534 36.901 32.852 30.246 30.838 28.217 34.030 31.462
Conclusions
We have presented the Cortex summarization system that is based on the fusion process of several different sentence selection metrics. The decision algorithm obtains good scores on INEX-2010, indicating that the decision process is a good strategy for preventing overfitting on the training corpus. In the INEX2010 corpus, Cortex system obtained very good results in the automatic FRESA evaluations. 2 3
http://www.inex.otago.ac.nz/ FRESA package is disponible at web site: http://lia.univ-avignon.fr/ fileadmin/axes/TALNE/downloads/index_fresa.html
294
J.-M. Torres-Moreno and M. Gagnon
References 1. ANSI. American National Standard for Writing Abstracts. Technical report, American National Standards Institute, Inc., New York, NY (ANSI Z39.14.1979) (1979) 2. Luhn, H.P.: The Automatic Creation of Literature Abstracts. IBM Journal of Research and Development 2(2), 159 (1958) 3. Edmundson, H.P.: New Methods in Automatic Extracting. Journal of the ACM (JACM) 16(2), 264–285 (1969) 4. Salton, G.: The SMART Retrieval System - Experiments un Automatic Document Processing, Englewood Cliffs (1971) 5. Salton, G., McGill, M.: Introduction to Modern Information Retrieval. McGrawHill, New York (1983) 6. Torres-Moreno, J.M., Velazquez-Morales, P., Meunier, J.-G.: Condens´es automatiques de textes. Lexicometrica. L’analyse de donn´ees textuelles: De l’enquˆete aux corpus litt´eraires, Special (2004), www.cavi.univ-paris3.fr/lexicometrica 7. Jacquemin, C., Zweigenbaum, P.: Traitement automatique des langues pour l’acc`es au contenu des documents. Le Document en Sciences du Traitement de l’information 4, 71–109 (2000) 8. Abracos, J., Lopes, G.P.: Statistical Methods for Retrieving Most Significant Paragraphs in Newspaper Articles. In: Mani, I., Maybury, M.T. (eds.) ACL/EACL 1997-WS, Madrid, Spain, July 11 (1997) 9. Teufel, S., Moens, M.: Sentence Extraction as a Classification Task. In: Mani, I., Maybury, M.T. (eds.) ACL/EACL 1997-WS, Madrid, Spain (1997) 10. Hovy, E., Lin, C.Y.: Automated Text Summarization in SUMMARIST. In: Mani, I., Maybury, M.T. (eds.) Advances in Automatic Text Summarization, pp. 81–94. The MIT Press, Cambridge (1999) 11. Kupiec, J., Pedersen, J.O., Chen, F.: A Trainable Document Summarizer. In: Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 68–73 (1995) 12. Tombros, A., Sanderson, M., Gray, P.: Advantages of Query Biased Summaries in Information Retrieval. In: Hovy, E., Radev, D.R. (eds.) AAAI1998-S, Stanford, California, USA, March 23-25, pp. 34–43. The AAAI Press, Menlo Park (1998) 13. Schlesinger, J.D., Backer, D.J., Donway, R.L.: Using Document Features and Statistical Modeling to Improve Query-Based Summarization. In: DUC 2001, New Orleans, LA (2001) 14. Torres-Moreno, J.M., Velazquez-Moralez, P., Meunier, J.: CORTEX, un algorithme pour la condensation automatique de textes. In: ARCo, vol. 2, p. 365 (2005) 15. Torres-Moreno, J.M., St-Onge, P.-L., Gagnon, M., El-B`eze, M., Bellot, P.: Automatic summarization system coupled with a question-answering system (qaas). CoRR, abs/0905.2990 (2009) 16. Torres-Moreno, J.M., Velazquez-Morales, P., Meunier, J.G.: Condens´es de textes par des m´ethodes num´eriques. JADT 2, 723–734 (2002) 17. Salton, G.: Automatic text processing, 9th edn. Addison-Wesley Longman Publishing Co. Inc., Amsterdam (1989)
The REG Summarization System with Question Reformulation at QA@INEX Track 2010 Jorge Vivaldi1 , Iria da Cunha1 , and Javier Ram´ırez2 1
Instituto Universitario de Ling¨ u´ıstica Aplicada - UPF Barcelona 2 Universidad Aut´ onoma Metropolitana-Azcapotzalco Mexico {iria.dacunha,jorge.vivaldi}@upf.edu,[email protected] http://www.iula.upf.edu
Abstract. In this paper we present REG, a graph approach to study a fundamental problem of Natural Language Processing: the automatic summarization of documents. The algorithm models a document as a graph, to obtain weighted sentences. We applied this approach to the INEX@QA 2010 task (question-answering). To do it, we have extracted the terms and name entities from the queries, in order to obtain a list of terms and name entities related with the main topic of the question. Using this strategy, REG obtained good results regarding performance (measured with the automatic evaluation system FRESA) and readability (measured with human evaluation), being one of the seven best systems into the task. Keywords: INEX, Automatic Summarization Answering System, REG.
1
System, Question-
Introduction
Nowadays automatic summarization is a very prominent research topic. We can define summary as “a condensed version of a source document having a recognizable genre and a very specific purpose: to give the reader an exact and concise idea of the contents of the source” (Saggion and Lapalme, 2002: 497). Summaries can be divided into “extracts”, if they contain the most important sentences extracted from the original text (ex. Edmunson, 1969; Nanba and Okumura, 2000; Gaizauskas et al., 2001; Lal and Reger, 2002; Torres-Moreno et al., 2002) and “abstracts”, if these sentences are re-written or paraphrased, generating a new text (ex. Ono et al., 1994; Paice, 1990; Radev, 1999). Most of the automatic summarization systems are extractive. These systems are useful in several domains: medical (ex. Johnson et al., 2002 Afantenos et al., 2005; da Cunha et al., 2007; Vivaldi et al., 2010), legal (ex. Farzindar et al., 2004), journalistic (ex. Abracos and Lopes, 1997; Fuentes et al., 2004), etc. One of the tasks where these extractive summarization systems could help is question-answering. The objective of the INEX@QA 2010 track is to evaluate S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 295–302, 2011. c Springer-Verlag Berlin Heidelberg 2011
296
J. Vivaldi, I. da Cunha, and J. Ram´ırez
a difficult question-answering task, where questions are very precise (expecting short answers) or very complex (expecting long answers, including several sentences). In this second task is where automatic summarization systems could help. The used corpus in this track contains all the texts included into the English Wikipedia. The expected answers are automatic summaries of less than 500 words exclusively made of aggregated passages extracted from the Wikipedia corpus. The evaluation of the answers will be automatic, using the automatic evaluation system FRESA (Torres-Moreno et al., 2010a, 2010b, Saggion et al., 2010), and manual (evaluating syntactic incoherence, unsolved anaphora, redundancy, etc.). To carry out this task, we have decided to use REG (Torres-Moreno and Ram´ırez, 2010; Torres-Moreno et al., 2010), an automatic summarization system based on graphs. We have performed some expansions of the official INEX@QA 2010 queries, detecting the terms and name entities they contain, in order to obtain a list of terms related with the main topic of all the questions. This paper is organized as follows. In Section 2 we show REG, the summarization system we have used for our experiments. In Section 3 we explain how we have carried out the terms and name entities extraction of the queries. In Section 4 we present the experimental settings and results. Finally, in Section 5, we expose some conclusions.
2
The REG System
REG (Torres-Moreno and Ram´ırez, 2010; Torres-Moreno et al. 2010) is an Enhanced Graph summarizer (REG) for extract summarization, using a graph approach. The strategy of this system has two main stages: a) to carry out an adequate representation of the document and b) to give a weight to each sentence of the document. In the first stage, the system makes a vectorial representation of the document. In the second stage, the system uses a greedy optimization algorithm. The summary generation is done with the concatenation of the most relevant sentences (previously scored in the optimization stage). REG algorithm contains three modules. The first one carries out the vectorial transformation of the text with filtering, lemmatization/stemming and normalization processes. The second one applies the greedy algorithm and calculates the adjacency matrix. We obtain the score of the sentences directly from the algorithm. Therefore, sentences with more score will be selected as the most relevant. Finally, the third module generates the summary, selecting and concatenating the relevant sentences. The first and second modules use CORTEX (Torres-Moreno et al., 2002), a system that carries out an unsupervised extraction of the relevant sentences of a document using several numerical measures and a decision algorithm. The complexity of REG algorithm is O(n2 ). Nevertheless, there is a limitation, because it includes a fast classification algorithm which can be used only for short instances; this is the reason it is not very efficient for long texts.
The REG System at QA@INEX Track 2010
3
297
Terms and Name Entity Extraction
The starting point of this work is to consider that the terms or name entities included into the queries are representative of the main subject of these queries. If this assumption is true, the results obtained by a search engine would be optimized if we give it as input the list of terms or name entities of the queries, instead the complete queries. The first procedure for obtaining the query terms has been to found the main topic of the questions. This has been obtained by finding the terms candidate and the name entitites present in every query. On the one hand, terms are usually defined as lexical units to designate concepts in a domain. The notion of term that we have adopted in this work is based on the “Communicative Theory of Terminology” (Cabr´e, 1999): a term is a lexical unit (single/multiple word) that designates a concept in a given specialized domain. This means that terms have a designative function and, obviously not all the words in a language have such function. To detect which are the words that in a given text have this designative function is the main task of term detector/extractors (see Cabr´e et al. (2001) for a review of this subject). The detection of these units is a complex task mainly because terms adopt, with a few exceptions, all word formation rules in a given language (Pearson, 1998; Sager, 1999). Also, as mentioned in the term definition itself, it is necessary to confirm that a given lexical unit belong to the domain of interest. Due to the difficulties to verify this condition it is usual to refer the results obtained by an extractor as term candidates instead of just “terms”. Some terminological extractors are described in Barr´on-Cede˜ no et al. (2009), Wong et al. (2008), Sclano et al. (2007), Vivaldi and Rodr´ıguez (2001a, 2001b), Vivaldi (2001) or Bourigault and Jacquemin (1999), among others. In this context we have used the basic procedure for obtaining term candidates in the field of term extraction. Such candidates are typically obtained by using the morphosyntactic terminological patterns for any given language (see Cabr´e et al., 2001; Pazienza et al., 2005), English in this case. As the queries do not belong to any specific domain it is not possible determine the termhood of the retrieved candidates. We consider termhood as it has been defined in Kageura and Umino (1996): “the degree that a linguistic unit is related to domain-specific concepts”. On the other hand, the main objective of name entity recognition systems is to found automatically some units into texts. These units can be names of persons, names of organizations, locations, etc. Some works about this subjects are Jun’ichi and Kentaro (2007), Leong and Tou (2002) or Volk and Clematide (2001), among many others. As we have seen, nowadays several term extraction systems and name entity recognition systems exist for English. Nevertheless, their performances are not still perfect, so if we employ these systems in our work, their mistakes and the mistakes of the system we present here would be mixed. Moreover, term extractors are usually designed for a specialized domain, as medicine, economics, law, etc, but the topics of the queries provided by INEX@QA 2010 are several, that is, they do not correspond to an unique domain. As the INEX@QA 2010 organizers state (SanJuan et al., 2010: 2):
298
J. Vivaldi, I. da Cunha, and J. Ram´ırez
“Long type questions require long answers up to 500 words that must be selfcontained summaries made of passages extracted from the INEX 2009 corpus. Among the 150 long type questions, 80 are related to 2009 ad-hoc topics and the remaining 70 come from Nomao and Yahoo! Answers”. These facts, the performance of term and name entity extractors, and the diversity of domains of the queries, made us to carry out the term and name entity extraction not using a specific term extraction tool but just using a POS tagger. In practice, we mainly extract nominal sentences. Doing the task in this way, we are sure that the results we obtain are precise and we can evaluate our system properly. Considering that questions are very short, only a few candidates are obtained by such procedure; therefore, they have a high probability to be the main topic of the question. For example, for the query “How does GLSL unify vertex and fragment processing in a single instruction set?”, we consider that the terms “glsl”, “vertex processing”, “fragment processing” and “single instruction set” are useful to find the right answer. But for the query “Who is Eiffel?”, there are not any term, only the name entity “Eiffel?”.
4
Experiments Settings and Results
In this study, we used the document sets made available during the Initiative for the Evaluation of XML retrieval (INEX) 20101, in particular on the INEX 2010 QA Track (QA@INEX). These sets of documents where provided by the search engine Indri.2 REG has produced multidocument summaries using sets of 30, 40 and 50 of the documents provided by Indri using all the queries of the track. To evaluate the efficiency of REG over the INEX@QA corpus, we have used the FRESA package. This evaluation framework (FRESA –FRamework for Evaluating Summaries Automatically-) includes document-based summary evaluation measures based on probabilities distribution, specifically, the Kullback Leibler (KL) divergence and the Jensen Shannon (JS) divergence. As in the ROUGE package (Lin, 2004), FRESA supports different n-grams and skip n-grams probability distributions. The FRESA environment has been used in the evaluation of summaries produced in several European languages (English, French, Spanish and Catalan), and it integrates filtering and lemmatization in the treatment of summaries and documents. FRESA is available in the following link: http://lia.univ-avignon.fr/fileadmin/axes/TALNE/Ressources.html. Table 1 shows an example of the results obtained by REG using 50 documents as input. The query that the summary should answer in this case was the number 2009006 (“What are the similarities and differences between mean average precision and reciprocal rank used in Information Retrieval?”). The extracted 1 2
http://www.inex.otago.ac.nz/ Indri is a search engine from the Lemur project, a cooperative work between the University of Massachusetts and Carnegie Mellon University in order to build language modelling information retrieval tools: http://www.lemurproject.org/indri/
The REG System at QA@INEX Track 2010
299
Table 1. Example of REG results using 50 documents as input Distribution type Baseline summary Random unigram Random 5-gram Submitted summary
unigram 22.64989 18.18043 17.47178 22.77755
bigram with 2-gap 31.70850 32.07926 28.25213 28.44528 26.33253 27.03882 32.06325 32.53706
Average 28.81255 24.95928 23.61437 29.12595
terms for this query were: “similarities”, “differences”, “mean average precision”, “reciprocal rank” and “information retrieval”. This table presents REG results in comparison with an intelligent baseline (Baseline summary), and two simple baselines, that is, summaries including random n-grams (Random unigram) and 5-grams (Random 5-gram). We observe that our system is always better than these two simple baselines, but in comparison with the first one the performance is variable. Readability and coherence evaluation required human evaluation. Humans evaluated the correctness of the answers detecting incoherent grammatical structures, unsolved anaphora or redundant passages. For more information about the human evaluation methodology that INEX@QA 2010 organizers used, see SanJuan et al. (2010). Table 2 includes the final results of the task. There were nearly 100 participants into the INEX@QA 2010 task, using different approaches. Our system (number 857) is one of the best seven systems. It obtains 2031.85 average performance and average readability. Regarding performance, it is the fourth best system and, regarding readability, its results are similar to four other systems. The best performance was obtained by the system 98 (2006.45), based on focused Information Retrieval (IR), but it obtains a bad readability (the worst readability into the evaluation). The best system regarding readability was the 943, but it was the worst regarding average performance (2050.81). We consider that the results obtained by our system are positive: neither performance nor readability are bad. Moreover, results on both categories (performance and readability) are good. We think that the fact our system is able to maintain a good level in both categories means that our system is robust. Table 2. Evaluation results for the seven best participants into the INEX@QA 2010 task: Cumulative KL divergence ID method unigrams bigrams 4 skip grams 943 long sentences 1642.93 2252.28 2257.22 857 question reformulation 1621.14 2234.57 2239.85 98 focused IR 1599.29 2207.56 2212.49 92 MWT expansion 1617.35 2226.61 2231.56 860 system combination 1617.6 2227.37 2232.43 855 semantic expansion 1625.76 2235.21 2240.35 557 JS minimization 1631.29 2237.61 2242.83
average readability 2050.81 good 2031.85 average 2006.45 bad 2025.17 average 2025.8 average 2033.77 average 2037.24 average
300
5
J. Vivaldi, I. da Cunha, and J. Ram´ırez
Conclusions
We have presented the REG summarization system, an extractive summarization algorithm that models a document as a graph, to obtain weighted sentences. We applied this approach to the INEX@QA 2010 task, extracting the terms and the name entities from the queries, in order to obtain a list of terms and name entities related with the main topic of the question. We consider that, over the INEX-2010 corpus, REG obtained good results in the automatic evaluations. The experiments have shown that the performance and readability of our system are very good, in comparison to the other participants in the INEX@QA 2010 task: it is the fourth best system. Neither performance nor readability are bad, and we consider that this fact confirms the robustness of the system. We think that the main problem of our methodology it that some queries are long and they have several terms or name entities we could extract, but there are some queries that are very short and the extraction is not possible or very limited. As future work, we would like to include more semantic aspects to solve this problem. Eventually, we may use Wikipedia for selecting the most relevant nominal sentences found in the question. In this way, some nominal sentences may be no used in the query, improving the results obtained from the search engine. In the case of very short questions, the query may be expanded using information obtained from this resource. We would like to use in the future another similar but more efficient algorithm (like merge sort or heapsort, with complexity O(nlgn); see Cormen et al., 2005), before the greedy algorithm, when the sentences ranking is performed.
References 1. Abracos, J., Lopes, G.: Statistical methods for retrieving most significant paragraphs in newspaper articles. In: Proceedings of the ACL/EACL 1997 Workshop on Intelligent Scalable Text Summarization, Madrid, pp. 51–57 (1997) 2. Afantenos, S., Karkaletsis, V., Stamatopoulos, P.: Summarization of medical documents: A survey. Artificial Intelligence in Medicine 33(2), 157–177 (2005) 3. Barr´ on-Cede˜ no, A., Sierra, G., Drouin, P., Ananiadou, S.: An Improved Automatic Term Recognition Method for Spanish. In: Gelbukh, A. (ed.) CICLing 2009. LNCS, vol. 5449, pp. 125–136. Springer, Heidelberg (2009) 4. Bourigault, D., Jacquemin, C.: Term Extraction + Term Clustering: an integrated platform for computer-aided terminology. In: Proceedings of EACL, pp. 15–22 (1999) 5. Cabr´e, M.T.: La terminolog´ıa. Representaci´ on y comunicaci´ on. IULA-UPF, Barcelona (1999) 6. Cabr´e, M.T., Estop` a, R., Vivaldi, J.: Automatic term detection: a review of current systems. In: Bourigault, D., Jacquemin, C., L’Homme, M.C. (eds.) Recent Advances in Computational Terminology, pp. 53–87. John Benjamins, Amsterdam (2001) 7. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to algorithms, 2nd edn. The MIT Press, Cambridge (2005)
The REG System at QA@INEX Track 2010
301
8. da Cunha, I., Wanner, L., Cabr´e, M.T.: Summarization of specialized discourse: The case of medical articles in Spanish. Terminology 13(2), 249–286 (2007) 9. Edmunson, H.P.: New Methods in Automatic Extraction. Journal of the Association for Computing Machinery 16, 264–285 (1969) 10. Farzindar, A., Lapalme, G., Descl´es, J.-P.: R´esum´e de textes juridiques par identification de leur structure th´ematique. Traitement Automatique des Langues 45(1), 39–64 (2004) 11. Fuentes, M., Gonzalez, E., Rodriguez, H.: Resumidor de noticies en catala del projecte Hermes. In: Proceedings of II Congr´es d’Enginyeria en Llengua Catalana (CELC 2004), Andorra, pp. 102–102 (2004) 12. Gaizauskas, R., Herring, P., Oakes, M., Beaulieu, M., Willett, P., Fowkes, H., Jonsson, A.: Intelligent access to text: Integrating information extraction technology into text browsers. In: Proceedings of the Human Language Technology Conference, San Diego, pp. 189–193 (2001) 13. Johnson, D.B., Zou, Q., Dionisio, J.D., Liu, V.Z., Chu, W.W.: Modeling medical content for automated summarization. Annals of the New York Academy of Sciences 980, 247–258 (2002) 14. Jun’ichi, K., Kentaro, T.: Exploiting Wikipedia as External Knowledge for Name Entity Recognition. In: Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 698–707 (2007) 15. Kageura, K., Umino, B.: Methods of automatic term recognition: A review. Terminology 3(2), 259–289 (1996) 16. Lal, P., Reger, S.: Extract-based Summarization with Simplication. In: Proceedings of the 2nd Document Understanding Conference at the 40th Meeting of the Association for Computational Linguistics, pp. 90–96 (2002) 17. Leong Chieu, H., Tou Ng, H.: Named entity recognition: a maximum entropy approach using global information. In: Proceedings of the 19th International Conference on Computational Linguistics, pp. 1-7 (2002) 18. Lin, C.-Y.: ROUGE: A Package for Automatic Evaluation of Summaries. In: Proceedings of Text Summarization Branches Out: ACL 2004 Workshop, pp. 74–81 (2004) 19. Nanba, H., Okumura, M.: Producing More Readable Extracts by Revising Them. In: Proceedings of the 18th International Conference on Computational Linguistics (COLING 2000), Saarbrucken, pp. 1071–1075 (2000) 20. Ono, K., Sumita, K., Miike, S.: Abstract generation based on rhetorical structure extraction. In: Proceedings of the International Conference on Computational Linguistics, Kyoto, pp. 344–348 (1994) 21. Paice, C.D.: Constructing literature abstracts by computer: Techniques and prospects. Information Processing and Management 26, 171–186 (1990) 22. Pazienza, M.T., Pennacchiotti, M., Zanzotto, F.M.: Terminology Extraction: An Analysis of Linguistic and Statistical Approaches. In: Studies in Fuzziness and Soft Computing, vol. 185, pp. 255–279 (2005) 23. Pearson, J.: Terms in context. John Benjamin, Amsterdam (1998) 24. Radev, D.: Language Reuse and Regeneration: Generating Natural Language Summaries from Multiple On-Line Sources. New York, Columbia University [PhD Thesis] (1999) 25. Sager, J.C.: In search of a foundation: Towards a theory of terms. Terminology 5(1), 41–57 (1999) 26. Saggion, H., Lapalme, G.: Generating Indicative-Informative Summaries with SumUM. Computational Linguistics 28(4), 497–526 (2002)
302
J. Vivaldi, I. da Cunha, and J. Ram´ırez
27. Saggion, H., Torres-Moreno, J.-M., da Cunha, I., SanJuan, E., Vel´ azquez-Morales, P., SanJuan, E.: Multilingual Summarization Evaluation without Human Models. In: Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010), Pekin (2010) 28. SanJuan, E., Bellot, P., Moriceau, V., Tannier, X.: Overview of the 2010 QA Track: Preliminary results. In: Geva, S., et al. (eds.) INEX 2010. LNCS, vol. 6932, pp. 269–281. Springer, Heidelberg (2010) 29. Sclano, F., Velardi, P.: Termextractor: a web application to learn the shared terminology of emergent web communities. In: Proceedings of the 3rd International Conference on Interoperability for Enterprise Software and Applications, pp. 287– 298 (2007) 30. Torres-Moreno, J.-M., Saggion, H., da Cunha, I., SanJuan, E., Vel´ azquez-Morales, P., SanJuan, E.: Summary Evaluation With and Without References. Polibitis: Research Journal on Computer Science and Computer Engineering with Applications 42 (2010a) 31. Torres-Moreno, J.-M., Saggion, H., da Cunha, I., Vel´ azquez-Morales, P., SanJuan, E.: Ealuation automatique de r´esum´es avec et sans r´ef´erence. In: Proceedings of the 17e Conf´erence sur le Traitement Automatique des Langues Naturelles (TALN), Universit´e de Montr´eal et Ecole Polytechnique de Montr´eal, Montreal Canada (2010) 32. Torres-Moreno, J-M., Ram´ırez, J.: REG: un algorithme glouton appliqu´e au r´esum´e automatique de texte. In: JADT 2010, Roma, Italia (2010) 33. Torres-Moreno, J-M., Ram´ırez, J.: Un resumeur a base de graphes, ind´ependant de la langue. In: Proceedings of the International Workshop African HLT 2010, Djibouti (2010) 34. Torres-Moreno, J.M., Vel´ azquez-Morales, P., Meunier, J.G.: Condens´es de textes par des m´ethodes num´eriques. In: Proceedings of the 6th International Conference on the Statistical Analysis of Textual Data (JADT), St. Malo, pp. 723–734 (2002) 35. Vivaldi, J., da Cunha, I., Torres-Moreno, J.M., Vel´ azquez, P.: Automatic Summarization Using Terminological and Semantic Resources. In: En Actas del 7th International Conference on Language Resources and Evaluation (LREC 2010), Valletta, Malta (2010) 36. Vivaldi, J.: Extracci´ on de candidatos a t´ermino mediante combinaci´ on de estrategias heterog´eneas. Ph.D. thesis, Universitat Politcnica de Catalunya, Barcelona (2001) 37. Vivaldi, J., Rodr´ıguez, H.: Improving term extraction by combining different techniques. Terminology 7(1), 31–47 (2001a) 38. Vivaldi, J., M` arquez, L., Rodr´ıguez, H.: Improving term extraction by system combination using boosting. In: Flach, P.A., De Raedt, L. (eds.) ECML 2001. LNCS (LNAI), vol. 2167, pp. 515–526. Springer, Heidelberg (2001b) 39. Volk, M., Clematide, S.: Learn-filter-apply-forget. Mixed approaches to name entity recognition. In: Proceedings of the 6th International Workshop on Applications of Natural Language for Informations Systems, Madrid, Spain (2001) 40. Won, W., Liu, W., Bennamoun, M.: Determination of Unithood and Termhood for Term Recognition. In: Song, M., Wu, Y. (eds.) Handbook of Research on Text and Web Mining Technologies. IGI Global (2008)
Overview of the INEX 2010 Focused Relevance Feedback Track Timothy Chappell1 and Shlomo Geva2 1 2
Queensland University of Technology [email protected] Queensland University of Technology [email protected]
Abstract. The INEX 2010 Focused Relevance Feedback track offered a refined approach to the evaluation of Focused Relevance Feedback algorithms through simulated exhaustive user feedback. As in traditional approaches we simulated a user-in-the loop by re-using the assessments of ad-hoc retrieval obtained from real users who assess focused ad-hoc retrieval submissions. The evaluation was extended in several ways: the use of exhaustive relevance feedback over entire runs; the evaluation of focused retrieval where both the retrieval results and the feedback are focused; the evaluation was performed over a closed set of documents and complete focused assessments; the evaluation was performed over executable implementations of relevance feedback algorithms; and finally, the entire evaluation platform is reusable. We present the evaluation methodology, its implementation, and experimental results obtained for nine submissions from three participating organisations.
1
Introduction
This paper presents an overview of the INEX 2010 Focused Relevance Feedback track. The purpose behind the track is to evaluate the performance of focused relevance feedback plugins in comparison to each other against unknown data. The data used for this track is the document collection and the assessments collected for the INEX 2010 Ad Hoc track. Organisations participated in the track by submitting their algorithms in the form of dynamic libraries implementing a ranking function capable of receiving relevance information from the evaluation platform and acting on it to improve the quality of future results. The interface also allows the algorithms to provide back more detailed information, such as the section or sections within a document that it believes are most relevant, enabling focused results to be returned. The result of running the algorithms against a set of topics is a set of relevance assessments, which can then be scored against the same assessments used to provide feedback to the algorithms. The result is a reflection of how well the algorithms were able to learn from the relevance information they were given. S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 303–312, 2011. c Springer-Verlag Berlin Heidelberg 2011
304
2
T. Chappell and S. Geva
Focused Relevance Feedback
The relevance feedback approach that is the focus of this track is a modified form of traditional approaches to relevance feedback, which typically involved nominating whole documents as either relevant or not relevant. The end user would typically be presented with a list of documents which they would mark as relevant or not relevant before returning this input to the system which would search the remainder of the collection for similar documents and present them to the user. Due to a fundamental paradigm change in how people use computers since these early approaches to relevance feedback, a more interactive feedback loop where the user continues to provide relevance information as they go through the results is now possible. We adopted a refined approach to the evaluation of relevance feedback algorithms through simulated exhaustive incremental user feedback. The approach extends evaluation in several ways relative to traditional evaluation. First, it facilitates the evaluation of retrieval where both the retrieval results and the feedback are focused. This means that both the search results and the feedback are specified as passages, or as XML elements, in documents - rather than as whole documents. Second, the evaluation is performed over a closed set of documents and assessments, and hence the evaluation is exhaustive, reliable and less dependent on the specific search engine in use. By reusing the relatively small topic assessment pools, having only several hundred documents per topic, the search engine quality can largely be taken out of the equation. Third, the evaluation is performed over executable implementations of relevance feedback algorithms rather than being performed over result submissions. Finally, the entire evaluation platform is reusable and over time can be used to measure progress in focused relevance feedback in an independent, reproducible, verifiable, uniform, and methodologically sound manner.
3
Evaluation
The Focused Relevance Feedback track is concerned with the simulation of a user interacting with an information retrieval system, searching for a number of different topics. The quality of the results this user receives is then used to evaluate the relevance feedback approach. The INEX Ad-Hoc track, which evaluates ranking algorithms, makes use of user-collected assessments on which portions of documents are relevant to users searching for particular topics. These assessments are perfect, not just for the evaluation of the rankings produced by the algorithms, but also for providing Focused Relevance Feedback algorithms with the relevance information they need. As such, a Focused Relevance Feedback algorithm can be mechanically evaluated without a need of a real user by simulating one, looking up the appropriate assessments for each document received from the algorithm and sending back the relevant passages.
Overview of the INEX 2010 Focused Relevance Feedback Track
305
To be able to accurately evaluate and compare the performance of different focused relevance feedback algorithms, it is necessary that the algorithms not be trained on the exact relevance assessments they are to receive in the evaluation. After all, a search engine isn’t going to know in advance what the user is looking for. For this reason, it becomes necessary to evaluate an algorithm with data that was not available at the time the algorithm is written. Unlike in the Ad-Hoc track, the relevance submissions used to evaluate the plugins are also required for input to the plugins, so there is no way to provide participating organisations with enough information for them to provide submissions without potentially gaining an unrealistic advantage. There are at least two potential ways of rectifying this. One is to require the submission of the algorithms a certain amount of time (for example, one hour) after the assessments for the Ad Hoc track were made available. This approach, however, is flawed as it allows very little margin for error and that it will unfairly advantage organisations that happen to be based in the right time zones, depending on when the assessments are released. In addition, it allows the relevance feedback algorithm to look ahead at relevance results it has not yet received in order to artificially improve the quality of the ranking. These factors make it unsuitable for the running of the track. The other approach, and the one used in the Focused Relevance Feedback track, is to have the participating organisations submit the algorithms themselves, rather than just the results. The algorithms were submitted as dynamic libraries written in Java, chosen for its cross-platform efficiency. The dynamic libraries were then linked into an evaluation platform which simulated a user searching for a number of different topics, providing relevance results on each document given. The order in which the documents were submitted to the platform was then used to return a ranking, which could be evaluated like the results of any ranking algorithm.
4 4.1
Task Overview
Participants were asked to create one or more Relevance Feedback Modules intended to rank a collection of documents with a query while incrementally responding to explicit user feedback on the relevance of the results presented to the user. These Relevance Feedback Modules were implemented as dynamically linkable modules that implement a standard defined interface. The Evaluation Platform interacts with the Relevance Feedback Modules directly, simulating a user search session. The Evaluation Platform instantiates a Relevance Feedback Module object and provides it with a set of XML documents and a query. The Relevance Feedback Module responds by ranking the documents (without feedback) and returning the ranking to the Evaluation Platform. This is so that the difference in quality between the rankings before and after feedback can be compared to determine the extent of the effect the relevance feedback has on the results. The Evaluation Platform is then asked for the next most relevant
306
T. Chappell and S. Geva
document in the collection (that has not yet been presented to the user). On subsequent calls the Evaluation Platform passes relevance feedback (in the form of passage offsets and lengths) about the last document presented by the Relevance Feedback Module. This feedback is taken from the qrels of the respective topic, as provided by the Ad-Hoc track assessors. The simulated user feedback may then be used by the Relevance Feedback Module to re-rank the remaining unseen documents and return the next most relevant document. The Evaluation Platform makes repeated calls to the Relevance Feedback Module until all relevant documents in the collection have been returned. The Evaluation Platform retains the presentation order of documents as generated by the Relevance Feedback Module. This order can then be evaluated as a submission to the ad-hoc track in the usual manner and with the standard retrieval evaluation metrics. It is expected that an effective dynamic relevance feedback method will produce a higher score than a static ranking method (i.e. the initial baseline rank ordering). Evaluation is performed over all topics and systems are ranked by the averaged performance over the entire set of topics, using standard INEX and TREC metrics. Each topic consists of a set of documents (the topic pool) and a complete and exhaustive set of manual focused assessments against a query. Hence, we effectively have a ”classical” Cranfield experiment over each topic pool as a small collection with complete assessments for a single query. The small collection size allows participants without an efficient implementation of a search engine to handle the task without the complexities of scale that the full collection presents. 4.2
Submission Format
Participating organisations submitted JAR files that implemented the following specification: package rf; public interface RFInterface { public Integer[] first(String[] documentList, String query); public Integer next(); public String getFOL(); public String getXPath(); public void relevant(Integer offset, Integer length, String Xpath, String relevantText); } In the call to first, the algorithm is given the set of documents and the query used to rank them and must return an initial ranking of the documents. The purpose of this is to quantify the improvement gained from providing the relevance assessments to the Relevance Feedback Module. The Evaluation Platform then calls next to request the next document from the algorithm, making a call to relevant to provide feedback on any relevant passages in the document. The optional methods getFOL and getXPath, if implemented, allow the Relevance
Overview of the INEX 2010 Focused Relevance Feedback Track
307
Feedback Module to provide more focused results to the Evaluation Platform in order to gain better results from the focused evaluation. None of the submitted algorithms implemented these methods, however. A subset of 10 topics that assessments were created for in the Ad-Hoc track were selected to evaluate the topics. These topics were chosen by taking the first 10 topics that were manually assessed during the assessment stage of the Ad Hoc track. Between them these topics covered a total of 7488 documents (with some overlap), of which 1421 were marked as relevant (containing at least one passage marked as relevant.) On average, in the relevant documents 36.43% of the text was marked as relevant, with the shortest relevant passage being 29 characters long. Each Relevance Feedback Module receives a call to first for each topic, but is not otherwise reinitialised, giving the module opportunity to implement some form of learning to rank approach. It is difficult to tell if any of the submissions made use of this.
5 5.1
Results Submissions
Three groups submitted a total of nine Relevance Feedback Modules to the INEX 2010 Relevance Feedback track. While this submission pool is small, it was to be expected given that this is the first time the track has been run. QUT resubmitted the reference Relevance Feedback Module. The Indian Statistical Institute (ISI) submitted three modules and Peking University submitted five. To provide a starting point for participating organisations, a reference Relevance Feedback Module, both in source and binary form, was provided by QUT. This reference module used the ranking engine Lucene[2] as a base for a modified Rocchio[3] approach. The approach used was to provide the document collection to Lucene for indexing, then construct search queries based on the original query but with terms added from those selections of text nominated as relevant. A scrolling character buffer of constant size was used, with old data rolling off as new selections of relevant text were added to the buffer, and popular terms (ranked by term frequency) added to the search query. The highest ranked document not yet returned is then presented to the Evaluation Platform and this cycle continues until the collection is exhausted. The reference algorithm does not provide focused results and as such does not implement the getFOL or getXPath methods. The Indian Statistical Institute (ISI) submitted a relevance feedback algorithm that was designed around finding non-overlapping word windows in the relevant passages and modifying the query to include these terms. Peking University submitted an algorithm that used a Rocchio-based algorithm revised to include negative feedback and criterion weight adjustments.
308
T. Chappell and S. Geva
Table 1. Average precision and R-precision for submitted modules Run Reference 1367 1366 1365 1368 1373 1369 1371 1370
Organisation Average Precision R-Precision QUT 0.4827 0.5095 ISI 0.3770 0.3845 ISI 0.3617 0.3503 ISI 0.3445 0.3307 Peking 0.2275 0.2171 Peking 0.2210 0.2015 Peking 0.2196 0.2015 Peking 0.2185 0.2015 Peking 0.2177 0.1962
Fig. 1. Recall-precision comparison of Relevance Feedback Modules
Table 2. Average precision and R-precision for submitted modules, before and after feedback Run Reference 1367 1366 1365 1368 1373 1369 1371 1370
Organisation QUT ISI ISI ISI Peking Peking Peking Peking Peking
Without feedback Average R-Precision 0.2796 0.3240 0.2771 0.2718 0.2771 0.2718 0.2768 0.2718 0.2286 0.2031 0.2231 0.2158 0.2231 0.2158 0.2292 0.2159 0.2292 0.2159
With feedback Average R-Precision 0.4827 0.5095 0.3770 0.3845 0.3617 0.3503 0.3445 0.3307 0.2275 0.2171 0.2210 0.2015 0.2196 0.2015 0.2185 0.2015 0.2177 0.1962
Overview of the INEX 2010 Focused Relevance Feedback Track
Fig. 2. Recall-precision comparison for track submissions
309
310
T. Chappell and S. Geva
Fig. 3. Recall-precision comparison for track submissions
Overview of the INEX 2010 Focused Relevance Feedback Track
Fig. 4. Recall-precision comparison for track submissions
311
312
5.2
T. Chappell and S. Geva
Evaluation
The Relevance Feedback Modules submitted by participating organisations were run through the Evaluation Platform. As none of the submitted Relevance Feedback Modules returned focused results, trec eval [1] was used to evaluate the results. The two measures used in this evaluation are average precision and R-precision. R-precision is calculated as the precision (number of relevant documents) after R documents have been seen, where R is the number of relevant documents in the collection. Average precision is calculated from the sum of the precision at each recall point (a point where a certain fraction of the documents in the collection have been seen) divided by the number of recall points. To see how the different Relevance Feedback Modules compared over the entire evaluation, a recall-precision curve can be plotted to show how they stack up. These recall-precision curves were created with trec eval and consist of precision values at 11 points from 0.0 to 1.0 recall. The precision value used is the highest precision within that recall point. The ranking results returned by the Relevance Feedback Modules show how each module would have ranked the documents if feedback was not available. This information can then be used to show how use of feedback changes the results from each algorithm: The results can also be compared on a recall-precision curve to show how the use of feedback data changed the results of each algorithm.
6
Conclusion
We have presented the Focused Relevance Feedback track at INEX 2010. Despite the limited pool of participating organisations, the track has provided a good starting point for further work in this area and it is hoped that the Focused Relevance Feedback track will be more successful at INEX 2011. While the INEX Ad Hoc track—used as the source of assessment data for the Focused Relevance Feedback track—will not be running in INEX 2011, there is plenty of assessment data that has not so far been used and as such future runs of this track will use assessments from previous runs of the Ad Hoc track. Acknowledgements. We would like to thank all the participating organisations for their contributions and hard work.
References 1. Buckley, C.: The trec eval IR evaluation package (2004) (Retrieved January 1, 2005) 2. Goetz, B.: The Lucene search engine: Powerful, flexible, and free. Javaworld (2002), http://www.javaworld.com/javaworld/jw-09-2000/jw-0915-lucene.html 3. Rocchio, J.J.: Relevance feedback in information retrieval. In: Salton, G. (ed.) The SMART Retrieval System: Experiments in Automatic Document Processing. Prentice-Hall Series in Automatic Computation, vol. 14, pp. 313–323. Prentice-Hall, Englewood Cliffs (1971)
Exploring Accumulative Query Expansion for Relevance Feedback Debasis Ganguly, Johannes Leveling, and Gareth J.F. Jones CNGL, School of Computing, Dublin City University, Dublin 9, Ireland {dganguly,jleveling,gjones}@computing.dcu.ie
Abstract. For the participation of Dublin City University (DCU) in the Relevance Feedback (RF) track of INEX 2010, we investigated the relation between the length of relevant text passages and the number of RF terms. In our experiments, relevant passages are segmented into non-overlapping windows of fixed length which are sorted by similarity with the query. In each retrieval iteration, we extend the current query with the most frequent terms extracted from these word windows. The number of feedback terms corresponds to a constant number, a number proportional to the length of relevant passages, and a number inversely proportional to the length of relevant passages, respectively. Retrieval experiments show a significant increase in MAP for INEX 2008 training data and improved precisions at early recall levels for the 2010 topics as compared to the baseline Rocchio feedback.
1
Introduction
Query expansion (QE) is a popular technique to improve information retrieval effectiveness by extending the original query. The Relevance Feedback (RF) track at INEX 2010 attempts to simulate user interaction by communicating true relevance information between a Controller module, with access to the qrels file and simulates RF from a user, and a Feedback module. This allows re-ranking results by changing the set of retrieved documents in every retrieval iteration. In the RF track, the incremental reporting of relevant text segments from full documents allows the development of a feedback algorithm choosing feedback terms in different ways, compared to standard Blind Relevance Feedback (BRF). The exchange of relevance information between user and system denotes a retrieval iteration and can be repeated multiple times for the same query. In each iteration, the feedback algorithm or its parameters can be adapted to improve retrieval performance. In this paper, we investigate the relationship between the length of relevant passages and the number of feedback terms. We explore three variants of selecting the number of feedback terms depending on the length of relevant test segments: i) choosing a constant number of feedback terms, ii) choosing a number directly proportional to the lengths of the relevant segments, and iii) choosing a number inversely proportional to the lengths of the relevant segments. All three approaches can be justified in their own way. One might want to choose more S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 313–318, 2011. c Springer-Verlag Berlin Heidelberg 2011
314
D. Ganguly, J. Leveling, and G.J.F. Jones
terms from a smaller relevant segment in the hope that it has less or no noisy terms. It might be more effective to choose more terms from larger relevant segments on the assumption that the likelihood of finding useful expansion terms increases with the the length of a relevant section. Finally, the length of a relevant passage may be unrelated to the best number of feedback terms so that a constant number of feedback terms is the best choice. The rest of this paper is organized as follows: Section 2 describes the motivation of the RF experiments and related work, our RF algorithm is introduced in Section 3, Section 4 reports our results in the RF track and analyzes the results and we conclude the paper with directions for future work in Section 5.
2
Related Work
One of the problems of BRF is that all terms which meet the selection criterion for feedback terms are used for QE. This includes terms which are not related to the query, for example semantically unrelated, but highly frequent terms from long (pseudo-)relevant documents or text segments.1 A number of experiments using small text passages instead of full documents for BRF have been conducted [1,2,3,4,5]. One assumption behind using small passages is that long documents can contain a wider range of discourse and noisy terms would be added to the original query, which can result in a topic shift. A wide range of discourse in long documents means that the relevant portion of such a document may be quite small and feedback terms should be extracted from relevant portions only. Another assumption behind these approaches is that even non-relevant documents can contain passages with useful feedback terms [6]. In contrast, our experiments for the RF track at INEX 2010 aim at investigating if true relevant text passages also contain noise so that a segmentation into smaller textual units (in this case: word windows) will improve IR effectiveness. The motivation behind our method is the assumption that even large true relevant text passages contain harmful terms for QE. Furthermore, the RF track provides the opportunity to explore how to use true relevant passages for QE. LCA [7] involves decomposing the feedback documents into fixed length word windows to overcome the problem of choosing terms from unrelated portions of a long document. The word windows are ranked by a score which depends on the co-occurrence of a word with the query term. Similar to LCA, we presume that terms in close proximity to query terms are good candidates for QE. In our RF method, we select feedback terms from word windows which are maximally similar to the query, the similarity being measured by Lucene’s default similarity which is a variant of tf · idf, thus achieving the same effect of filtering out potentially irrelevant parts of a longer document as in LCA. A major difference with respect to LCA is that we do not compute term co-occurrences explicitly. 1
We employ the term segment in its most general sense, denoting sentences, paragraphs, and other small text units such as word windows.
Exploring Accumulative Query Expansion for Relevance Feedback
3
315
System Setup
In contrast to other evaluation tracks in IR, submissions to the RF track comprise of an implemented software module (a JAVA .jar file). We submitted 3 RF modules for the RF track at INEX 2010. As a baseline, we use standard Rocchio feedback [8], which was packaged as a default feedback module implementation by the INEX organizers. We use the Lucene API2 for indexing and retrieval. The RF track simulates a user highlighting relevant passages if any for each document presented to him. The feedback module re-ranks the initial results based on relevance information. The Term Selection Algorithm. We propose the following basic algorithm for RF. Three variations of this algorithm are realized by choosing the terms ti in different ways (Step 6 of the algorithm). 1. For the ith request of the next document to return, repeat Steps 2-7. 2. Let R be the accumulated string of relevant passages from the last document returned. 3. Tokenize the string R into words and break it up into fixed length windows of m words after applying stopword removal and stemming. 4. Let tf(ti ) be the term frequency of the ith term in the window and idf(ti ) the inverse document frequency of ti . 1 For each window w = (w1 , . . . wn ), where wi = tf(ti ) 2 log idf(ti ), compute the cosine similarity of w with q = (q1 , . . . qn ), where qi = tf(ti ). 5. Rank all windows w by similarity score and choose top p windows. 6. Extract the most frequent T terms from these windows and add them to the query. The three variants for choosing T terms are as follows: – RFconst : T = t, where t is a constant ∀i. i) – RFinvrsl : T = (LiL−r t, where t is a constant, Li is the length of the ith i document and ri is the length of the relevant section of the ith document. – RFrsl : T = Lrii t, with t, Li and ri defined as before. 7. Re-retrieve with the expanded query and return the topmost similar document not returned previously. The first variant (RFconst ) chooses a constant number of terms regardless of the segment length. For RFinvrsl and RFrsl the number of terms added is inversely and directly proportional to the length of the relevant section, which corresponds to choosing a greater number of terms from shorter relevant segments, and choosing a smaller number of terms from shorter segments, respectively. The default Rocchio feedback implementation serves as a baseline with parameters (α, β, γ) = (1, 0.75, 0) to weight original terms, positive, and negative feedback terms. The Rocchio feedback uses T = 20 terms for query expansion. Two major differences between the baseline module and our implementation are: a) the baseline method adds expansion terms to the original query at each iteration, whereas we add expansion terms for the ith iteration obtained during 2
http://www.apache.org/dyn/closer.cgi/lucene/java/
D. Ganguly, J. Leveling, and G.J.F. Jones
0.70
0.70
0.60
0.60
0.50
0.50 Precision
Precision
316
0.40 0.30 0.20 0.10 0.00 0.00
0.30 0.20
Rocchio const invrsl rsl 0.20
0.40
0.10 0.40 0.60 Recall
0.80
1.00
0.00 0.00
Rocchio const invrsl rsl 0.20
0.40 0.60 Recall
0.80
1.00
Fig. 1. Interpolated Precision-Recall graphs for INEX 2008 (left) and 2010 data (right)
the (i−1)th iteration; b) the step-size of the incremental feedback for the baseline method is 5, i.e. it expands the original query after every 5 iterations whereas our method uses a step-size of 1, i.e. we update the query after every iteration. Thus, our query expansion accumulates terms at every retrieval iteration in contrast to the baseline method, which generates a new query in each iteration. This is in contrast to the “save nothing” strategy [9] which was shown to be ineffective for incremental feedback. Training the System. The parameters as outlined in the feedback algorithm are the window length m, the number p of most similar windows to restrict the expansion terms to, and the variable T , which represents the number of feedback terms. After conducting a range of experiments on INEX 2008 topic set we chose the optimal settings of (m, p, T ) = (30, 10, 5). The results of these training experiments with the above settings are outlined in Table 1. Wilcoxon tests on the 11 point precision-recall curves reveal that the improvements for the three proposed methods over RFRocchio are statistically significant.
4
Results and Analysis
For our official submission to the RF track, we used the optimal parameters obtained from INEX 2008 training topics. The graphs of Figure 1 show the document level interpolated precision-recall curves for the four approaches on INEX 2008 and 2010 topics. The graphs of Figure 2 show the interpolated geometric means of per-topic precision values measured at 11-point recall levels on 2008 and 2010 data. The left graph of Figure 1 reveals some interesting characteristics of the two feedback methods RFrsl and RFinvrsl . While it can be seen that RFrsl yields low precision for lower levels of recall, it outperforms RFinvrsl for higher levels of recall which suggests that it might be worth trying a combination of the above two techniques as a part of our future work. The right graph of Figure 1 suggests that RFRocchio starts off with a better precision and outperforms the focused methods until a recall level of 80% is reached. For the focused methods, although the initial retrieval precision (precision at less than 10% recall level) is
Exploring Accumulative Query Expansion for Relevance Feedback
317
Table 1. Results for Relevance Feedback on INEX 2008 training topics Methodology
Evaluation Metric MAP
No feedback RFRocchio RFconst RFinvrsl RFrsl
0.80
0.3087 0.4292 0.4687 0.4805 0.4596
0.60
0.60 Precision
0.40 0.30
0.50 0.40 0.30
0.20
0.20
0.10
0.10 0.20
0.40
0.60
Recall
0.80
Rocchio const invrsl rsl
0.70
0.50
0.00 0.00
0.3952 0.5011 0.5519 0.5611 0.5477
0.80
Rocchio const invrsl rsl
0.70
Precision
0.3610 0.4744 0.5366 0.5442 0.5307
GMAP MAiP
1.00
0.00 0.00
0.20
0.40
0.60
0.80
1.00
Recall
Fig. 2. RIC curves for INEX 2008 (left) and INEX 2010 data (right)
lower, precision picks-up steadily and does not suffer from a steep down-hill as observed for the Rocchio method. For the RIC metric, we see a different trend for the INEX 2010 topics. The focused methods have a higher precision (thus suggesting that it is more appropriate for precision oriented retrieval tasks such as the focused task) at recall levels of less than 20% after which the Rocchio feedback outperforms each of them. The fact that the focused RF methods are outperformed by the Rocchio feedback as measured by the standard document level retrieval metric MAP, leads to the question of what changes in the characteristics of the topics and the relevant set, if any, from 2008 to 2010, caused this trend reversal. The corresponding answers should explain the differences in the training results and the official submissions. A possibility is that the average length of relevant passages (average being computed by accumulating the number of relevant characters per document averaged over the number of topics) for the INEX 2010 topic set is higher (453.6 characters) as compared to INEX 2008 (409.9 characters), which means that further reducing the length of relevant passages may be required. This suggests using smaller values for the number of windows, e.g. decreasing p, could possibly improve results.
318
5
D. Ganguly, J. Leveling, and G.J.F. Jones
Conclusions and Future Work
For our participation in the RF track at INEX 2010, we implemented a new feedback method which selects feedback terms from maximally similar word windows extracted from reported relevant text passages. The proposed method significantly outperforms the baseline Rocchio feedback method on the INEX 2008 training data and yields better precision at early recall levels when measured with the RIC metric, but does not show improvement for the INEX 2010 data when evaluated with MAP. Future work includes optimizing feedback parameters m and p keeping in mind that the average length of relevant segments is higher for INEX 2010 topics. In addition based on the observation from INEX-2010 results that Rocchio gives better precision at early recall levels and our method gives better precision at higher recall levels, we plan to explore a combination of feedback strategies, i.e. selecting or switching the feedback strategies at some retrieval iteration. Acknowledgments. This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (CNGL) project.
References 1. Callan, J.P.: Passage-level evidence in document retrieval. In: SIGIR 1994, pp. 302– 310. ACM/Springer (1994) 2. Allan, J.: Relevance feedback with too much data. In: SIGIR 1995, pp. 337–343. ACM Press, New York (1995) 3. Ganguly, D., Leveling, J., Jones, G.J.F.: Exploring sentence level query expansion in the language model. In: Proceedings of ICON 2010, pp. 18–27 (2010) 4. Murdock, V.: Aspects of Sentence Retrieval. PhD thesis, University of Massachusetts - Amherst (2006) 5. Losada, D.E.: Statistical query expansion for sentence retrieval and its effects on weak and strong queries. Inf. Retr. 13, 485–506 (2010) 6. Wilkinson, R.: Effective retrieval of structured documents. In: SIGIR 1994, pp. 311– 317. Springer-Verlag Inc., New York (1994) 7. Xu, J., Croft, W.B.: Query expansion using local and global document analysis. In: SIGIR 1996, pp. 4–11. ACM, New York (1996) 8. Rocchio, J.J.: Relevance feedback in information retrieval. In: The SMART retrieval system – Experiments in automatic document processing, pp. 313–323. PrenticeHall, Englewood Cliffs (1971) 9. Allan, J.: Incremental relevance feedback for information filtering. In: SIGIR 1996, pp. 270–278. ACM, NY (1996)
Combining Strategies for XML Retrieval Ning Gao1 , Zhi-Hong Deng1 2 , Jia-Jian Jiang1 , Sheng-Long Lv1 , and Hang Yu1
1 Key Laboratory of Machine Perception (Ministry of Education), School of Electronic Engineering and Computer Science, Peking University 2 The State Key Lab of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing 100190, China
Abstract. This paper describes Peking University’s approaches to the Ad Hoc, Data Centric and Relevance Feedback track. In Ad Hoc track, results for four tasks were submitted, EÆciency, Restricted Focused, Relevance In Context and Restricted Relevance In Context. To evaluate the relevance between documents and a given query, multiple strategies, such as Two-Step retrieval, MAXLCA query results, BM25, distribution measurements and learn-to-optimize method are combined to form a more eective search engine. In Data Centric track, to gain a set of closely related nodes that are collectively relevant to a given keyword query, we promote three factors, correlation, explicitnesses and distinctiveness. In Relevance Feedback track, to obtain useful information from feedbacks, our implementation employs two techniques, a revised Rocchio algorithm and criterion weight adjustment. Keywords: INEX, Ad Hoc, Data Centric, Relevance Feedback.
1 Introduction INEX Ad Hoc Track [1] aims at evaluating performance in retrieving relevant results (e.g. XML elements or documents) to a certain query. In Ad Hoc 2010, four di erent tasks are addressed: (1) EÆciency task requires a thorough run to estimate the relevance of documents components. (2)Relevant in Context task requires a ranked list of nonoverlap XML elements or passages grouped by their corresponding parent articles. (3) Restricted Focused task limits results (elements or passages) ranked in relevance order up to a maximal length of 1,000 characters per topic. (4) Restricted Relevant in Context tasks requires a ranked list of elements or passages. And for each element a ranked list result covers its relevant material, at most 500 characters for each result document. Initially we only consider the e ectiveness of retrieval, regardless of the eÆciency and restricted length. Five di erent querying strategies, BM25 [2], Two-Step retrieval, Maximal Lowest Common Ancestor (MAXLCA)[3] query results, distribution measurements and learn-to-optimize method, are combined to form a more eÆcient search engine. Detailed definitions of these technologies is introduced in section 2. The thorough retrieval results are submitted to eÆciency task. Furthermore, the results for other The corresponding author. S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 319–331, 2011. c Springer-Verlag Berlin Heidelberg 2011
320
N. Gao et al.
three tasks are all obtained based on thorough run task. In Restricted Focused task, each topic limits the answers up to a maximal length of 1,000 characters. Hence, we only return the result elements with top relevance-length ratio, in which relevance score for each results are computed by the aforementioned combined strategies module. For Relevance in Context task, the process scans the thorough runs and integrates the elements belonging to the same document. The orders for these integrated elements in ranked list are determined by the elements with maximal relevance score in each set. To obtain the results of Restricted Relevance in Context task, each result in the Relevance in Context is pruned to maximal 500 characters. Similar to the Restricted Focused task, only passages with top relevance-length ratio are retrieved. Data Centric track is to provide a common forum for researchers or users to compare di erent retrieval techniques on data-centric XML documents, whose structure is plentiful and carries important information about objects and their relationships. In order to generate a good snippet [4] of an XML document, it is critical to pick up the most descriptive or representative attributes together with their values. In this paper, we present three main principles to judge the importance of attributes:(1) whether the attribute is distinguishable or not; (2)whether the attribute is explicit or implicit; (3)whether the attribute describes the entity directly or indirectly. Relevance feedback[1] is a new track in INEX 2010. IR system with relevance feedback permits interactivities between the users and the system. Users provide relevance (irrelevance) information of search result to IR system, which is utilized by IR system to return more e ective results. Relevance feedback track in INEX2010 simulates a single user searching for a particular query in an IR system that supports relevance feedback. The user highlights relevant passages of text and provides this feedback to the IR system. The IR system re-ranks the remainder of the unseen result list to provide more relevant results to the user. The Relevance Feedback track mainly focuses on the improvement of search result before and after implementing relevance feedback. Consequently, our team pay more attention to acquiring more information through feedback rather than optimizing results as what we did in Ad hoc track. In section 2, we explicitly introduce the five strategies used in Ad Hoc track and reveal the corresponding evaluation results. Section 3 describes our work on Data Centric track. In section 4, the methods applied for Relevance Feedback track are presented.
2 Ad Hoc Track Figure 1 describes the framework of our search engine. The Inverted Index of the Data Collection is initially processed in the background. Afterwards, when a user submits a Query to the Interface, the search engine first retrieves the relevant documents. Based on the definition of MAXimal Lowest Common Ancestor (MAXLCA) and All Common Ancestor (ACA) query results, the relevant elements are extracted from the relevant documents. Furthermore, we use the ranking model BM25 to rank these extracted disordered element results, the output of the processing on a Ranked List. To further improve the e ect of the ranking module, the top several results in the Ranked List are re-ranked by Distribution Measurements. In Distribution Measurements, there are four criterions based on the distribution of keywords that are taken into consideration, explicitly introduced in section 4. The weights of these four criterions in the re-ranking function
Combining Strategies for XML Retrieval
321
Fig. 1. System Framework
are trained by a Learning to Optimize method. Finally, the Re-ranked Results are returned to the user as searching Results. In the search engine, five di erent strategies are used, Two-Step search, MAXLCA query results, BM25, Distribution Measurements and Learn to Optimize method. – Two-Step Retrieval: Di erent from HTML, the retrievable unit for the INEX focused task is XML elements rather than the whole document text. Therefore, the core idea of Two-Step retrieval is splitting the searching process into two steps. The first level starts from the traditional article retrieval. Then taking the top returned relevant articles as querying database, the second layer further processes extracting the relevant elements. Finally, the extracted elements are ranked and returned in a result list form. In INEX 2009, one of the most e ective search engines proposed by Queensland used Two-Step as retrieval strategy[19]. – BM25: Based on various research and comparative experiments, BM25 is confirmed to be an e ective ranking method. It takes both text and structure information into consideration. Additionally, evaluation results of Ad Hoc Track show that BM25 performs better than some other frequently cited ranking models, such as tf*idf [5] etc. Motivated by BM25’s excellent performance, we implant it into the search engine as a basic ranking method. In the remainder of the section, we apply other three technologies in the search engine. MAXLCA and ACA defines which elements are relevant and to be returned as results. Distribution measurements are re-ranking criterions used to evaluate the relevance between elements and queries according to the distribution of the keyword matches. Learn-to-optimize method is devised to tune the weights of di erent ranking methods in the final decision. 2.1 Maximal Lowest Common Ancestor (MAXLCA) Due to the fact that the returned results of XML retrieval are elements, an adaptive XML search engine should define which elements in the XML tree are relevant and to be retrieved. Several approaches have been proposed for identifying relevant results, such as XRANK [6], SLCA [7] and XSeek [8] and so on. In this paper, our definition of query results are called MAXLCA. Furthermore, we compare it with another widely used definition of query results, naming All Common Ancestor (ACA).
322
N. Gao et al.
Fig. 2. Sample XML Tree
– ACA: In an XML tree, the nodes containing all matches of keywords are returned as relevant results. For example, there is a submitted query XML retrieval, Jack and an XML tree presented as figure 2, in which the matches of keywords have been marked. Node 0 and 0.1 contains all the three matches in the tree, so that these two nodes should be returned. – MAXLCA: In an XML tree, the lowest node that contains all matches of keywords is defined as a query result. For example, within the nodes containing all matches of keywords in XML tree, 0.1 is the lowest one. Therefore, node 0.1 is the unique query result. 2.2 Distribution Measurements By observing a large amount of over 2000 query-result pairs, we discover that the distribution of the keyword matches in results plays a crucial role in picturing the theme of the passage. Moreover, four detailed statistical characteristics based on distribution are considered, which present advantage capability on distinguishing relevant and irrelevant passages. – Distance Among Keywords (DAK). Zhao etc. considers the proximity of query terms in [20]. Here in this paper the minimum distance among the keywords is calculated. The closer the matches of keywords, the more relevant an element is. – Distance Among Keyword Classes (DAKC). Xu and Croft discuss term clustering in [21]. In this paper, the matches of a certain keyword in passages are firstly clustered into several subsets. The closer the keywords subsets are, the more relevant this passage is.
Combining Strategies for XML Retrieval
323
– Degree of Integration Among Keywords (DIAK). The passage with higher degree of integration is considered as more concentrating on one certain theme and should be given a higher priority in the returned list. – Quantity Variance of Keywords (QVK). The passages whose numbers of di erent keywords vary significantly should be penalized. 2.3 Learn to Optimize the Parameters Nerual Network method in machine learning is introduced to tune the weights of the four features in distribution measurements. The Wiki English collection, queries and assessments of INEX 2009 Ad Hoc track are used as training samples. In training, there is a set of query Q q1 q2 qm extracted from the INEX 2009 Ad Hoc track. Each query qi is associated with a list of candidate elements j Ei (e1i e2i en(i) i ), where ei denotes the the j-th candidate element to query qi and n(i) is the size of E i . The candidate elements are defined as MAXLCA or ACA elements. Moreover, each candidate elements list Ei is associated with a ground-truth list Gi (g1i g2i gn(i) i ), indicating the relevance score of each elements in E i . Given that the wiki collection only contains information of whether or not the passages in a document is relevant, the F-measure [22] is applied to evaluate the ground truth score. Given a query qi , the ground-truth score of the j-th candidate element is defined as follows: precision
relevant irrelevant relevant
recall
relevant REL
(1) (2)
(1 012 ) precision recall (3) 012 precision recall In the formula, relevant is the length of relevant contents highlighted by user in e, while irrelevant stands for the length of irrelevant parts. REL indicates the total length of relevant contents in the data collection. The general bias parameter is set as 0.1, denoting that the weight of precision is ten times as much as recall. Furthermore, for each query qi , we use the distribution criterions defined in section 2.2 to get the predicted relevant scores of each candidate element, recorded in Ri (ri1 ri2 rin(i) ). In formula (4), S DAK , S DAKC , S DIAK and S QVK are the predicted scores for element j according to distance among keywords, distance among keyword classes, degree of integration among keywords and quantity variance of keywords respectively. gij
j
ri
S DAK S DAKC S DIAK ÆS QVK
(4)
Then each ground truth score list G i and predicted score list Ri form a “instance”. The loss function L is defined as the Euclidean distance between standard results lists Di and search results lists Ri . In each training epoch, the four criterions were used to compute the predicted score Ri . Then the learning module replaced the current weights with the new weights tuned according to the derivative of the loss between G i and Ri . Finally the process stops either when reaching the limit cycle index or the parameters do not change. Precise descriptions see [9].
324
N. Gao et al.
2.4 Comparison Results According to (1) two-step strategy or simple element retrieval; (2) ACA results or MAXLCA results; (3) BM25, Distribution or BM25Distribution, there should be 12 kinds of combination methods altogether. However, due to some previous experiments, we only submit 5 di erent combinations which are predicted as e ective to Ad Hoc track, illustrated in table 1. Table 1. Results Submitted to Ad Hoc Track MAXLCA
ACA
Two-Step
BM25
Distribution
AcaBM25 MaxBM25 RefMaxDis RefMaxBM25 RefMaxBM25Dis
Figure 3,4,5 illustrate the evaluation results of EÆciency task, Relevance In Context task and Restricted Relevance In Context task respectively under measure as focused retrieval. For Restricted Focused task, since we only submitted the RefMaxBM25Dis results, there is no useful and convincing comparison results that can be shown. However, as can be concluded from the other three tasks: – Two-Step search performs better than simple element search. According to tradition ranking methods, such as BM25 and tf*idf, the elements with high (tf passage length) ratio are marked as high relevance, emphasizing on small fragments even if they are extracted from irrelevant documents. Two-Step strategy eliminates such bias, since the candidate elements are all extracted from documents predicted to be relevant. – MAXLCA performs better on wiki collection than ACA. Though according to our experiments on the data collection of INEX2010 and INEX2009, the MAXLCA results perform better than ACA results, we still support the definition of ACA for its apparently high flexibility. On the other hand, the definition of MAXLCA is only suitable for short documents, such as web pages. – Rather than completely abandoning BM25, distribution measurement is suitable for improving the performance and modifying the drawbacks of it. Our approach originally aims at modifying the disadvantage parts of BM25, since it has been proved e ective by many searching systems in INEX. The distribution measurement is a re-ranking method, where each standards only focuses on one single point. Accordingly, we use a learning method to learn the optimal weights of these standards for a certain data collection and only in this way, the final re-ranking method are actually determined by the data collection. – The method using Two-Step as retrieving strategy, MAXLCA as query results, BM25 and distribution measurement as ranking functions shows the best performance.
Combining Strategies for XML Retrieval
Fig. 3. Evaluation Results of EÆciency Task
Fig. 4. Evaluation Results of Relevance in Context Task
Fig. 5. Evaluation Results of Restricted Relevance in Context Task
325
326
N. Gao et al.
Fig. 6. IMDB data segment
3 Data Centric Track In XML keyword search, there are quite numbers of results returned, only by the snippet of each result can the users judge whether it meets the search intention. Therefore, it is important to generate a good snippet for each result. [8][10] pointed that a good XML result snippet should be a self-contained information unit of a bounded size that e ectively summarizes the query result and di erentiates itself from the others. 3.1 Related Work [10] [11] pointed out that a good XML result snippet should be a self-contained information unit of a bounded size that e ectively summarizes the query result and di erentiates itself from the others. Meanwhile the authors have also accomplished a snippet generation system called eXtract [4]. The size limitation of a snippet requires the search engine to extract the most important information (information here refers to attributes of an entity) from the result document, thus the main problem is to define the importance of attributes. 3.2 Extracting the Most Representative Attributes In this paper, we use a semantic model MRepA (Most Representative Attribute)[12] to evaluate the importance of an attribute to its corresponding entity. Further, three main principles are proposed to judge the importance of attributes, presented as, (1)whether the attribute is distinguishable or not; (2)whether the attribute is explicit or implicit; (3)whether the attribute describes the entity directly or indirectly. 3.2.1 Correlation between Entities and Attributes To evaluate whether an attribute describes an entity directly, we analyze the position between the attribute and the entity in an XML document tree. In this section we define the entity-attribute path and use the number of entities on the path to measure the correlation between an attribute and its corresponding entity.
Combining Strategies for XML Retrieval
327
Definition 1. An entity-attribute path is a set of nodes which are on the path from an entity to its attribute(including the entity and the attribute). Definition 2. We define the correlation between entity e and attribute a R(e, a) as follows R(e a) klength(e a)
n
i 1
1 mi
(5)
Where n length(e a) refers to the number of entities between entity e and attribute a, and mi refers to the number of the entities of i-th category in the path. k is a parameter set less than 1. For example, in figure 4, suppose that there are 10 movie nodes, the entity-attribute path from node person to title Forrest Gump is person, filmography, act, movie, title. There are two entities person and movie on the path. The number of person is 1, while the number of the movie is 10. 3.2.2 Explicitnesses of Attributes In XML keyword search, the length of the value of an attribute is usually associated with the explicitness of the attribute. Long text tends to be more descriptive but less explicit, while short text tends to be more explicit and clear. Thus we use the length (or the number of words) of the text to judge the explicitness of an attribute roughly. Definition 3. We judge the explicitness of an attribute by the complicacy (length) of its value, and we denote the explicitness of attribute a as E(a). 3.2.3 Distinctiveness of Attributes A distinguishable attribute should match the following two conditions, (1) the attribute appears in all of the entities; (2) the values of the attribute are di erent in di erent entities. Due to the possibility that two di erent person share the same name, in this section we promote the formula to calculate how much an attribute meets the demands. Definition 4. We use distinctiveness of attributes to evaluate the distinguish ability of the attributes. Wa
exp(pa ) H(a)
H(a)
(6)
n
p(ai ) log[p(ai)]
(7)
i 1
In the above formulas, Wa is the distinctiveness of attribute a. pa refers to the percentage of the correlative entities where attribute a appears. H(a) is the entropy of attribute a, which estimates the variety intensity of attribute a.
328
N. Gao et al.
3.2.4 MRepA Model In MRepA model, we take the above three factors into consideration. Given a keyword query, we firstly return a set of entities as results, and then pick up the top-k most important attributes of each entity into the snippet. Definition 5. The importance of an attribute a to an entity e S(e,a) is defined as follows S (e a) Wa E(a)R(e a)
(8)
3.2.5 Comparison Results In Data Centric track, there are three assessments, TREC MAP metric, INEX thorough retrieval MAiP metric and INEX Relevant-in-Context MAgP T2I(300). In MAP metric, our results perform the best, using the description and narrative of the topics as extra information. However, poor responses are got under MAiP and MAgP metric. Table 2. Results Submitted to Data Centric Track MRepA
MAP 0.5046
MAgP NA
MAiP 0.0157
4 Relevance Feedback Track In relevance feedback track, we employs two techniques, a revised Rocchio algorithm and criterion weight adjustment. In section 4.1, we briefly introduce Rocchio algorithm [13]. In section 4.2 the revised algorithm is proposed. We discuss the adjustment of the criterion weights in section 4.3. 4.1 Rocchio Algorithm Rocchio algorithm operates on vector space model, in which a document is represented by a vector of n weights such as d (t1 t2 t3 t4 tn ). Where n is the number of unique terms in document collections and ti is the weight of the i-th term in document d. The keyword query is also presented as a vector. The general idea of Rocchio algorithm is that the initial query may not express the purpose of a IR system user completely and e ectively. Rocchio algorithm’s goal is to define an optimal query that maximize the di erence between the average vector of the relevant documents and the average vector of the irrelevant documents. To achieve this, Rocchio algorithm adds new query terms and re-weight query terms in the query vector, making it more discriminative in choosing relevant documents from documents collection. The following formula shows the basic Rocchio algorithm Qt
Q0
1 n1
n1
i 0
Ri
1 n2
n2
Si
(9)
i 0
where Q0 is the initial query vector and Qt is the revised query vector, n1 is the number of relevant documents and n2 is the number of irrelevant documents, R( i) is the vector of
Combining Strategies for XML Retrieval
329
a relevant document, S i is a irrelevant document vector, and , , control the influence of each part. After modification, the terms only appear in relevant document get a high positive weight and those only in irrelevant documents get a high negative weight, while the terms get relatively low weight if they appear in both relevant and irrelevant documents and have less discriminative power. Some researchers have modified and extended the formula such as assigning di erent weight to original query terms[14] and added query terms or make constraints of number of documents used in the modification[15]. There are some other feedback methods based on probabilistic model and language model. The feedback method in classical probabilistic models is to select expanded terms primarily based on Robertson and Sparck-Jones Weight[16]. In language model, Zhai et al[17] estimate a model for all relevant documents together and expand original query model with it. There are also many other methods directly estimating a relevance model with relevance document[18]. 4.2 Revised Rocchio Algorithm Due to the fact that relevant information about each part of a relevant document is accessible in INEX2010 we divided a document into several paragraphs(we use ” p ” and ” p ” to identify paragraphs) and represent each paragraph as a vector in our implementation. We treat a paragraph as a document in the searching process and give each paragraph a score. We also assign a weight to each paragraph according to its length. The score of a document is the weighted sum of its paragraphs’ scores. In our implementation, we define query expansion formula as Qt
Q0
1 n
n
Pi
(10)
i 0
Where Pi donates the vector of term weights calculated from a paragraph. For term t j in Pi , we define its weight as follows
wj
scorePi 0 scorePi 0
tj
if Pi is a relevant paragraph in relevant document if scorePi is a paragraph in irrelevant document otherwise
(11)
Where scorePi denotes the score of paragraph Pi , and scorePi t j denotes the score of paragraph after removing all t j from it. Here we use an example to illustrate why we compute Pi like this. In INEX2009, the first topic of Ad hoc track is Noble Prize. A user queries this for preparing a presentation about the Nobel Prize. Therefore, he or she wants to collect information about Nobel Prize as much as possible. Assume that there is a document with a simple paragraph as a section title, Ig Nobel Prize. Apparently, the paragraph is not relevant because the Ig Nobel Prize is an American parody of the Nobel Prizes organized by the scientific humor magazine Annals of Improbable Research. However, the score of this paragraph is relatively high because it only contains three words and two of them are keywords. Intuitively, we can figure out that the term Ig is a bad word for this topic since it turns
330
N. Gao et al.
a high-score paragraph to a irrelevant one. In addition, term Nobel or Prize has no contribution to the irrelevance of this paragraph. However, if we use the formula in section 4.1, no di erence between Ig and Nobel is reflected in the values of Ri or S i . While in the revised model, the weights of Ig and Nobel are significantly di erent. In the revised model, we focus on the contribution of a term to relevance or irrelevance of the paragraph it belongs to. 4.3 Criterion Weight Adjustment To calculate score of a paragraph, we make three criterions, the frequency entropy, the mixing entropy and the weighted term distance between paragraph vector and query vector. The frequency entropy scales di erence of terms’ appearance frequency. It assigns a high score if all keywords appear the same number of times in a paragraph. The mixing entropy scales the alternation of keywords. It assigns low score to a paragraph if it talks about one of the keyword at beginning while talks about another keyword at the end without a mixture of them. Each criterion makes contrition to the final score of a paragraph. However, it is hard decide which criterion is of greater importance to a specific topic. So we try to get this information from the feedback data. In the searching process, we keep the score history of every criterion and every keyword. When updating the criterion weights, the discriminative power of each criterion and each keyword are computed. The discriminative power is computed as follows DP
( r ir )2 dr2 dir2
(12)
Where r is the mean contribution of this criterion or keyword to relevant paragraphs and ir is the mean contribution of this criterion or keyword to irrelevant paragraphs. dr is the standard deviation of contribution of this criterion or keyword to relevant paragraphs and dir is the standard deviation of contribution of this criterion or keyword to relevant paragraphs. High DP value means strong discriminative power in current topic, so we raise its weight to let it make bigger contribution to scoring paragraph. While low DP value indicates a criterion of keyword that is not suitable for the current topic. For example, in our experiment, in the topic Nobel Prize, these two keywords are assigned the same criterion weight 0.5. However after all the document are returned, the criterion weight of Nobel is raised to 0.89 but the Prize is only 0.11. This is understandable. Prize is relatively a more widely used word because there are a lot of prizes such as Fields Medal Prize, Turing Prize. Acknowledgments. This work was supported by the National High-Tech Research and Development Plan of China under Grant No.2009AA01Z136.
References 1. 2. Carmel, D., Maarek, Y.S., Mandelbrod, M., et al.: Searching XML documents via XML fragments. In: SIGIR 2003, pp. 151–158 (2003)
Combining Strategies for XML Retrieval
331
3. Gao, N., Deng, Z.H., Jiang, J.J., Xiang, Y.Q., Yu, H.: MAXLCA A Semantic XML Search Model Using Keywords. Technical Report 4. Huang, Y., Liu, Z., Chen, Y.: eXtract: A Snippet Generation System for XML Search. In: VLDB 2008, pp. 1392–1395 (2008) 5. Theobald, M., Schenkel, R., Wiekum, G.: An EÆcient and Versatile Query Engine for TopX Search. In: VLDB 2005, pp. 625–636 (2005) 6. Guo, L., Shao, F., Botev, C., Shanmugasundaram, J.: XRANK: Ranked Keyword Search over XML Documents. In: SIGMOD 2003, pp. 16–27 (2003) 7. Xu, Y., Papakonstantinou, Y.: EÆcient Keyword Search for Smallest LCAs in XML Databases. In: SIGMOD 2005, pp. 537–538 (2005) 8. Liu, Z., Chen, Y.: Identifying Meaningful Return Information for XML Keyword Search. In: SIGMOD 2007, pp. 329–340 (2007) 9. Gao, N., Deng, Z.H., Yu, H., Jiang, J.J.: ListOPT: A Learning to Optimize Method for XML Ranking. In: PAKDD 2010 (2010) 10. Liu, Z., Chen, Y.: Identifying Meaningful Return Information for XML Keyword Search. In: SIGMOD 2007, pp. 329–340 (2007) 11. Huang, Y., Liu, Z.Y., Chen, Y.: eXtract: A Snippet Generation System for XML Search. In: VLDB 2008, pp. 1392–1395 (2008) 12. Jiang, J., Deng, Z.H., Gao, N., Lv, S.L., Yu, H.: MRepA: Extracting the Most Representative Attributes in XML Keyword Search. Technical Report 13. Ruthven, I., Lalmas, M.: A survey on the use of relevance feedback for information access systems. The Knowledge Engineering Review 18(2), 95–145 (2003) 14. Ide, E.: New experiments in relevance feedback. In: Salton, G. (ed.) The SMART Retrieval System Experiments in Automatic Document Processing, ch. 16, pp. 337–354 (1971) 15. Ide, E., Salton, G.: Interactive search strategies and dynamic file organization in information retrieval. In: Salton, G. (ed.) The SMART Retrieval System - Experiments in Automatic Document Processing, ch.18, pp. 373–393 (1971) 16. Robertson, S.E., Jones, K.S.: Relevance weighting of search terms. Journal of the American Society of Information Science 27(3), 129–146 (1976) 17. Zhai, C., Laerty, J.D.: Model-basedfeedback in the language modeling approach toinformation retrieval. In: CIKM 2001, pp. 403–410 (2001) 18. Lavrenko, V., Bruce Croft, W.: Relevance-basedlanguage models. In: SIGIR 2001, pp. 120– 127 (2001) 19. Geva, S., Kamps, J., Lethonen, M., Schenkel, R., Thom, J.A., Trotman, A.: Overview of the INEX 2009 ad hoc track. In: Geva, S., Kamps, J., Trotman, A. (eds.) INEX 2009. LNCS, vol. 6203, pp. 4–25. Springer, Heidelberg (2010) 20. Zhao, J., Yun, Y.: A proximity language model for information retrieval. In: SIGIR 2009, pp. 291–298 (2009) 21. Xu, J., Croft, W.B.: Improving the eectiveness of information retrieval with local context analysis. In: TOIS 2000, pp. 79–112 (2000) 22. van Rijsbergen, C.J.: Information Retireval. Butterworths, London (1979)
Overview of the INEX 2010 Web Service Discovery Track James A. Thom1 and Chen Wu2 1
2
RMIT University, Melbourne, Australia [email protected] University of Western Australia, Perth, Australia [email protected]
Abstract. The Web Service Discovery track aims to investigate techniques for discovery of Web services based on searching service descriptions provided in Web Services Description Language (WSDL). Participating groups contributed to topic development and to the evaluation, which allows them to compare the effectiveness of their XML retrieval techniques for the discovery of Web services. This has lead to the initial development of a test collection that will allow future comparative experiments. Keywords: Web Service discovery, WSDL.
1
Introduction
An efficient and effective Web services discovery mechanism is important in many computing paradigms including Pervasive Computing, Service-Oriented Computing, and the most recent Cloud Computing, in which Web services constitute the chief building blocks. The Web Service Discovery track aims to investigate techniques for discovery of Web services based on searching service descriptions provided in Web Services Description Language (WSDL). There were five active groups participating in the task in 2010, and they contributed to providing topics, submitting runs and assessing results.
2
WSDL Collection
The Web Service Discovery track used a collection of WSDL documents. These WSDL documents were directly crawled from real-world public Web services indexed by the Google search engine. The test collection was pre-processed so that only valid WSDL1.1-compliant descriptions are retained for XML-based retrieval. The original dataset for track contained 1987 separate documents, however after some duplicates were removed, a revised version of the dataset was released containing 1738 documents (original document numbering was retained).
This work was done while the second author was at Curtin University in Perth.
S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 332–335, 2011. c Springer-Verlag Berlin Heidelberg 2011
Overview of the INEX 2010 Web Service Discovery Track
3
333
Topics
The participating groups were asked to create a set of candidate topics, representative of a range of realistic web service discovery needs. The submitted topics were in the same format as the ad hoc track. A sample topic is shown in Fig. 1. Out of the 31 topics submitted by five groups, 25 topics were selected for the track in 2010. airline flights //*[about(., airline flights)] Given an airline flight number I would like to find details of the flight. A service for any airline that can provide the status of airline flights is relevant. Fig. 1. Sample topic
4
Submissions
The submission format was the same as the ad hoc track, submission were allowed in one of three formats: – XML elements using XPath syntax – passages in File-Offset-Length (FOL) format – ranges of elements (for backward compatibility with previous INEX formats) However all groups only submitted document level runs and had difficulty following the submission format, so some formatting corrections were required. Five groups submitted a total of 15 runs, although all runs were used to contribute documents into the pool for each topic, three runs were excluded from the final evaluation as they included more serious errors (such as duplicate answers).
5
Assessment
The pooling of documents from the submitted included approximately 100 results for each topic. As the XML structure was important for assessing whether a document (or parts of a document), the INEX evaluation tool for the ad hoc track was used but the WSDL documents’ XML markup was displayed along the content of the elements. In some cases, there was little or no text content in the XML elements, so having the XML markup was essential in assessing the relevance. Of the 25 topics, only 20 were assessed.
334
6
J.A. Thom and C. Wu
Evaluation and Results
Since only document level runs were submitted, evaluation was only performed for document retrieval using Mean Average Precision as calculated by trec_eval. The results for the 12 valid runs are shown in Table 1, with the 11 point average precision shown in Fig. 2. Table 1. Mean Average Precision for all runs 1 2 3 4 5 6 7 8 9 10 11 12
map Institute Run 0.3469 Kasetsart University Kas I138BM25ESS010 0.3239 RMIT University RMIT10WS 0.2946 Hasso-Plattner-Institut HPI2010 0.2798 Kasetsart University Kas I138ANYSS025 0.2798 Kasetsart University Kas I138ANYBM25SS015 0.2348 Queensland University of Technology QUT BM25WordNetComposition 0.2233 Queensland University of Technology QUT BM25WordNet 0.2042 Queensland University of Technology QUT BM25WordNetCompositionWordNet 0.1451 Benemrita Universidad Autnoma de Puebla BUAPFCCWSD01 0.1268 Queensland University of Technology QUT Wikipedia 0.1095 Queensland University of Technology QUT WikipediaComposition 0.0937 Queensland University of Technology QUT WikipediaCompositionWordNet
($/
($.
($-
% )+0 *-()(
)( ($,
*()(
% )+0 (*-
% )+0 *-()-
($+
% *- ! % *- % *- !
($*
() %
($)
( (
($)
($*
($+
($,
($-
($.
($/
($0
($1
)
Fig. 2. Performance of 10 best Web Service Discovery runs
7
Conclusion
Now that the basics are in place for this track, we hope to get more groups participating in 2011. Long term goal of track is given a description of a workflow (e.g. in YAWL) representing a scientific or business process, find web services that can meet the steps within the process.
Overview of the INEX 2010 Web Service Discovery Track
335
Specific plans for 2011 include the following. – Improving the collection by modifying the existing documents by extracting smaller documents from the WSDL documents to represent individual operations which will make assessments easier, and possibly adding more documents, including newer documents (e.g. more WSDL 2 documents) – Requiring topics to be a sequence/graph of queries for a service rather than isolated information needs, which will allow two tasks: (i) document/passage retrieval (similar to 2010 but hopefully explore passage retrieval) of service components, and (ii) combining operations to meet service requirements (new). Acknowledgements. This track would not been possible without the support of the INEX organisers in providing software support, as it makes extensive use of the existing INEX infrastructure.
Semantics-Based Web Service Discovery Using Information Retrieval Techniques Jun Hou, Jinglan Zhang, Richi Nayak, and Aishwarya Bose Faculty of Science and Technology, Queensland University of Technology [email protected], {jinglan.zhang,r.nayak,a.bose}@qut.edu.au
Abstract. This paper demonstrates an experimental study that examines the accuracy of various information retrieval techniques for Web service discovery. The main goal of this research is to evaluate algorithms for semantic web service discovery. The evaluation is comprehensively benchmarked using more than 1,700 real-world WSDL documents from INEX 2010 Web Service Discovery Track dataset. For automatic search, we successfully use Latent Semantic Analysis and BM25 to perform Web service discovery. Moreover, we provide linking analysis which automatically links possible atomic Web services to meet the complex requirements of users. Our fusion engine recommends a final result to users. Our experiments show that linking analysis can improve the overall performance of Web service discovery. We also find that keyword-based search can quickly return results but it has limitation of understanding users’ goals. Keywords: Web service discovery, Semantics, Latent Semantic Analysis, Linking Analysis.
1
Introduction
With the popularity of Service Oriented Architecture (SOA), many enterprises offer their distributed Web services as interfaces for their core business systems. Web services are embracing unprecedented attention from the computer world. Web services can be discovered by matchmaking requirements of service requesters with providers. Web service discovery plays a key role in finding appropriate Web services. Although a number of Web services are provided by different organizations, there are still no standards for Web service design and provision. Many Web services have the same or similar functionalities but they are described in various ways. It is a challenging task to discover accurate Web services in accordance with users’ requirements. Web Service Description Language (WSDL), the standard description language, and Universal Description Discovery and Integration (UDDI) for advertising Web services, are introduced to discover and invoke Web services. Web services requesters and providers then communicate with each other by SOAP message, a XML format communication language based on HTTP. The WSDL and UDDI search mechanism utilizes syntactic search based on keywords because it can return a large number of S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 336–346, 2011. © Springer-Verlag Berlin Heidelberg 2011
Semantics-Based Web Service Discovery Using Information Retrieval Techniques
337
Web services in a relatively short time. However, exact keyword match may miss actually relevant services and semantic search is proposed to enhance the search accuracy. In addition, since different organizations design services in enclosed circumstances, atomic Web services cannot satisfy different users’ requirements [13]. One service can invoke other services to achieve goals with complicated requirements. Therefore, a set of Web services need to be composed to fulfill given tasks. Two methods are used in this paper, namely Latent Semantic Analysis (LSA) supported by Wikipedia corpus and BM25 supported by WordNet. Wikipedia corpus is used to create Latent Semantic Kernel, while WordNet is introduced to improve the search performance of BM25. On top of that, we propose a linking analysis, which can automatically compose possible atomic Web services to conduct user-preferred tasks. In the fusion engine, a new result with atomic and composite Web services is recommended to users.
2
Related Work
This section summarizes some previous work in Semantic Web service discovery. Due to the lack of semantics in WSDL, many semantic Web service description languages such as OWL-S, WSMO and WSDL-S have emerged to explicitly annotate WSDL with semantic information. OWL-S and WSMO demonstrate Web services semantics at a distinct level [12]. OWL-S is more concentrated on the “Upper ontology” (not domain-specific ontology) for describing Web services [3]. Compared to OWL-S, WSMO is more focused on producing a reference implementation of an execution environment, the Web Service modeling execution environment and specifying mediators [13]. Mediators are not the significant consideration in OWL-S conceptual and implementation [17]. However, the discovery mechanism in WSMX is based on keyword and simple semantic description [17]. Compared to OWL-S, WSDL-S has several advantages over OWL-S. First, details of both the semantics and operations can be described in WSDL. In addition, the semantic domain models are detailed externally, which offers Web service developers an opportunity to select the preferred ontology language. On top of that, the existing tool can be updated relatively easy. The objectives of WSDL-S are to be of compatibility with OWL-S with emphasizes on a more lightweight and incremental approach [14]. Although more lightweight and flexible (supporting different ontologies) ontology languages are emerging, there is still no standard ontology and the maintenance cost is very high with low scalability. Many researchers make use of traditional Information Retrieval techniques. They parse WSDL documents into bags of words and create a term-document matrix. Then Webs services are ranked by Term Frequency–Inverse Document Frequency (TFIDF) according to the term frequency of search query in each document. A binning & merging-based Latent Semantic Kernel [2] is proposed to enhance the semantics of LSA. The experiment result shows that the LSA approach can be acceptable both in scalability and complexity [19]. A method using surface parsing of sentences to add structural relations [4] are proposed to improve the performance on single sentences
338
J. Hou et al.
in LSA. However, there still are some issues related to LSA. The pre-process including stop word removal and stemming reduces common terms and outliers, it also breaks WSDL structure at the same time. Nayak & Iryadi [15] and Hao & Zhang [7] propose Schema matching approaches in WSDL-based Web service discovery. Such approaches try to find not only text but also structure information for comparing WSDL documents. To effectively investigate semantics in text, a Wikipedia-based structural relationship-enhanced concept thesaurus [8] is introduced. This approach concentrates on improving the semantic relationships between important terms by applying text clustering. Above approaches are only keyword-based and simple keywords may not represent the preference of users very well. Users’ selection of services is highly impacted by non-functionality such as response time, price, throughout, availability, reliability etc. Researchers are devoted to dig more semantic information from current Web resources. Ding, Lei, Jia, Bin, & Lun [5] propose a discovery method based on Tag. Tags are widely used in images, bookmarks, blogs and videos to annotate the content of them. This approach suffers the same problem of above ontology languages. It is limited by the scope of comment on Web services and the variety between different comment styles. Semantic Web Services Clustering (SWSC) [16] makes use of preconditions and effects from OWL-S to improve the accuracy of Web service discovery. Using translation tools, more context information such as preconditions and effects after invocation can be collected thereby increasing the consistency of Web service discovery. In this method, hidden Web services can be discovered and be attached to similar groups before conducting search. However, scalability is still a problem.
3
Discovery Approach
We propose a novel three-phase approach for Web service discovery. Figure 1 shows an overview of this methodology. In the semantic analysis phase, there are two methods used to retrieve atomic Web services, namely Latent Semantic Analysis (LSA) supported by Wikipedia corpus and BM25 supported by WordNet. Before applying those approaches, standard text pre-processing is performed to parse WSDL documents into bags of words. During this stage, stop word removal and stemming have been executed. 3.1
Pre-processing
Stop word removal aims to reduce words which act poorly as index terms. For example, those words can be “a”, “the”, “and” etc. An external stop word list is introduced to filter out those words to perform data analysis. Stemming is a process to replace words with their root or stem forms by removing affixes (suffixes or prefixes). Words such as “computing”, “computer” and “computed” will be replaced by the word “compute”. This process reduces not only the variety of words also the computation cost. The Porter Stemming Algorithm [18] is used to strip suffix.
Semantics-Based Web Service Discovery Using Information Retrieval Techniques
339
Fig. 1. Overview of Web Service Discovery Methodology
3.2
Semantic Analysis
Latent Semantic Analysis (LSA) Figure 2 shows the overview of Latent Semantic Analysis (LSA). In LSA, the semantic kernel is used to find semantic similarity between Web services and users’ queries. The semantic kernel is constructed from a general-purpose dataset. The wikipedia dataset [2] is chosen because it is not domain-specific and covers various topics. Figure 2 shows the overview of LSA in phase I. To start, each pre-processed WSDL document is then encoded as a vector. Components of the vector are terms in the WSDL document. Each vector component reflects the importance by TF-IDF. The user query is also converted to a vector which is compared with the vector of a WSDL document. The similarity between the user query (Q) and the Web service document (W) is represented by the cosine value of two vectors. Equation 1 shows how to calculate the similarity between Q and W. Sim Q, W
Cos Q, W
(1)
However, we use semantic kernel (K) here to enhance the semantics between Q and W. The Q and W is replaced with Q K and K W respectively. Equation 2 shows the improved equation with semantic kernel. Sim Q, W
Cos Q, W
Finally, the top-k Web services are returned to users (k is set to 25).
(2)
340
J. Hou et al.
Fig. 2. Overview of LSA
BM25 BM25 is a bag-of-words retrieval algorithm that ranks documents based on the query terms appearing in each document. To increase the amount of query terms, WordNet is introduced to incorporate with BM25. WordNet is a general ontology, which can boost semantics from users’ queries. Figure 3 shows the overview of using BM25 in phase I.
Fig. 3. Overview of BM25
After pre-processing, WSDL documents (W) are computed with users’ queries (Q) for similarity. Equation 3 shows the major equation of BM25. Score Q, W
,
∑ IDF q ,
|
|
(3)
Semantics-Based Web Service Discovery Using Information Retrieval Techniques
341
f , is q s term frequency in the WSDL document W. |W| is the length of the WSDL document and avgdl is the average document length in the text collection. Equation 4 shows the details of IDF q . IDF q
log
. .
N represents the total number of WSDL documents in the collection and n the number of WSDL documents containing q . Same as LSA, the top-k Web services are returned to users (k is set to 25). 3.3
(4) is
Linking Analysis
Web services are retrieved based on the query of a user. However, one Web service may not meet the requirement of the query of a user. For example, the query from a user is “weather by postcode” and the actual Web service is “weather by location”. Obviously, the Web service needs to corporate with another Web service such as “postcode to location”. Linking analysis aims to link possible Web services to satisfy the requirements of users. Figure 4 shows the overview of linking analysis.
Fig. 4. Overview of Linking Analysis
In linking analysis, we use top 25 results from LSA or BM25 instead of directly linking Web services in the collection. In a WSDL document, the tag consists of sets of tags, which contain the description of invocable functions. We consider that Web services can be linked together if one Web service’s
342
J. Hou et al.
output parameters match another one’s input parameters in parameter name, parameter amount and data type. That information of input and output parameters is extracted for linking analysis. During the extraction, non-topic words such as “result” and “response” are filtered out. Once we get parameter names, they are decomposed into tokens. For instance, “ChangePowerUnit” is split into “Change”, “Power” and “Unit” from each capital letter. If two parameter tokens are exactly same, we consider it as best match. However, there are parameters having tokens such as “car” and “vehicle” and they semantically can be linked. Therefore, we calculate the similarity between input and output parameters to semantically link two Web services. Equation 5 shows how to compute the similarity of two parameters. Sim P , P
,
(5)
is the sum of the similarity between tokens in parameterP and Sum sim W , W P . N represents the total number of parameter tokens in P and P . n is the minimum of the number of parameter tokens in P and P . For example, if P has 2 tokens and P has 3 tokens, n will be 2 and N will be 5. If Sim P , P is greater than 0.98, we consider that the two parameters can be linked (linkable parameters). Furthermore, we use another factor, link strength, to decide if the two Web services can be linked. Link strength demonstrates the compatibility of two Web services by the number of linkable parameters. Equation 6 shows how to calculate the link strength. Link Strength
(6)
N is the total number of linkable parameters and N is the number of input parameters of one Web service. Once we have the link strength, functions of Web services are converted to a graph where nodes representing functions are connected with each other by link strength. Afterwards, we use Floyd Warshall algorithm [6] to calculate the shortest path from each method to all other methods. We define composition strength as the average of link strength of a composition. All compositions are ordered by composition strength. Each composition is treated as a new Web service and compared with users’ queries for similarity by LSA. 3.4
System Integration
The main purpose of integration is to integrate the results from composition of Web services with the atomic ones from LSA or BM25. The most important task is to decide which result appears in the final list. Generally, composition result has a higher accuracy than an atomic one. In addition, if a Web service is the component of a composition, it will not appear in the final result. Therefore, we select all compositions to the final result and then add atomic results to form top 20 recommendations. Figure 5 shows the overview of System Integration.
Semantics-Based Web Service Discovery Using Information Retrieval Techniques
343
Fig. 5. Overview of System Integration
4
Data Set
The document collection is provided by the INEX 2010 organizing committee. The dataset [11] contains over 1,700 documents in the format of WSDL 1.1, which are directly crawled from real-world public Web services indexed by the Google search engine.
5
Evaluation
There are 25 topics from different domains. User queries are created by competition participants to ensure the variety. Figure 6 demonstrates the precision & recall curve under the query term “map”. In Figure 6, we have four runs with the initial QUT and the best of them is QUT_BM25WordNetCompositon. In the BM25WordNetComposition run, we use BM25 supported by WordNet and the linking analysis. This submission outperforms BM25WordNet, which only applies BM25 supported by WordNet. It suggests that the linking analysis improves the accuracy of Web service discovery. The submission QUT_Wikipedia is created using LSA and it does not perform very well. One reason may be that the query term, “map”, is simple and LSA cannot find semantic services. Another reason might be that the semantic services found by LSA are not closely relevant to the query term.
344
J. Hou et al.
Fig. 6. INEX Web Service Discovery Results [10]
As we can see from Figure 6, the run Kas_I138BM25ESS010 (from Kasetsart University) has the highest precision when recall is less than 0.6. According to the result released from INEX, Kas_I138BM25ESS010 has the highest score, which is 0.3469 [10] with the query term “map”. Kasetsart University makes use of preprocessing techniques to boost search accuracy [9]. In our method, we only use simple pre-processing techniques due to time-constraint. Our approach can be improved by applying sophisticated pre-processing techniques. Our linking analysis is proposed under the situation of multiple query items. For example, if a user types “weather by postcode”, a combination of services “weather by city” and “city to postcode” will be retrieved. As a result, it almost has no effect with the simple query “map”. We believe our algorithm will have better performance for more complicated multiple term queries, especially for queries that can only be satisfied with the linking of multiple atomic services. Figure 6 has also shown that even by utilizing sophisticated pre-processing techniques, the score is only 0.3469, which means there are still a lot of irrelevant retrievals. That is because textual information occupies only a small part of the overall size of WSDL documents so it is not sufficient enough to answer service queries on the single basis of query terms [1]. Moreover, WSDL documents describe the interfaces of Web services in an abstract way. The small size of textual information makes it relatively difficult to understand what the service offers. For example, sometimes, the service having a service name “map” in the service name tag may provide the map of a hospital or the map of a city. Keyword-based search with simple query term may not be suitable enough to answer users’ queries.
Semantics-Based Web Service Discovery Using Information Retrieval Techniques
6
345
Conclusions and Future Work
The experiment result shows that both the Latent Semantic Analysis and BM25 boosted by WordNet approaches work for web service discovery. BM25 outperforms the Latent Semantic Analysis approach. Link analysis automatically composes Web services to fulfill complex tasks. Unfortunately this cannot be demonstrated by simple queries that can be satisfied by atomic services. In this paper, Web services are converted to bags of words and then compared with users’ queries for similarity. However, we find that WSDL documents are not like normal documents having high richness of terms. More decomposition rules are needed to deal with abbreviation and artificial names when parsing WSDL documents. Furthermore, WSDL describes services in an abstract way sometimes just a single term in one tag. The single term cannot describe the function very well and investigating semantics by single words may cause more false negatives by misunderstanding the service functionality. In addition, discovering web services by considering only query terms is overly simple because Web services are involved in more complex business scenarios. Service choreography and service orchestration are considered when deploying and invoking Web services. Web services contain more business relationships than normal documents, especially during invocation. Non-functional parameters such as response time, price, throughout, availability, reliability etc have become significant factors on selecting services. As a result, more practical situations need to be investigated to effectively retrieve and select Web services.
References 1. Al-Masri, E., Mahmoud, Q.H.: Identifying Client Goals for Web Service Discovery. In: 2009 IEEE International Conference on Services Computing, Bangalore, India, pp. 202– 209 (2009) 2. Bose, A., Nayak, R., Bruza, P.: Improving Web Service Discovery by Using Semantic Models. In: 9th International Conference on Web Information Systems Engineering, Auckland, New Zealand, pp. 366–380 (2008) 3. Burstein, M.H., McDermott, D.V.: Ontology Translation for Interoperability among Semantic Web Services. AI Magazine 26, 71–82 (2005) 4. Dennis, S.: Handbook of Latent Semantic Analysis, Mahwah, N.J (2007) 5. Ding, Z., Lei, D., Jia, Y., Bin, Z., Lun, A.: A Web Service Discovery Method Based on Tag. In: 2010 International Conference on Complex, Intelligent and Software Intensive Systems, Krakow, Poland, pp. 404–408 (2010) 6. Floyd, R.W.: Algorithm 97: Shortest Path. Communications of the ACM 5, 345 (1962) 7. Hao, Y., Zhang, Y.: Web Services Discovery Based on Schema Matching. In: 13th Australasian Conference on Computer Science, Ballarat, Australia, pp. 107–113 (2007) 8. Hu, J., Fang, L., Cao, Y., Zeng, H.-J., Li, H., Yang, Q., et al.: Enhancing Text Clustering by Leveraging Wikipedia Semantics. In: 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Singapore, pp. 179– 186 (2008) 9. INEX Workshop Pre-proceedings (2010), http://inex.otago.ac.nz/data/publications.asp
346
J. Hou et al.
10. INEX Web Service Discovery Results, http://goanna.cs.rmit.edu.au/~jat/INEX2010_WS_results.html 11. INEX Web Service Track (2010), http://www.inex.otago.ac.nz/tracks/ webservices/webservices.asp 12. Lara, R., Roman, D., Polleres, A., Fensel, D.: A Conceptual Comparison of WSMO and OWL-S. In: Zhang, L.-J., Jeckle, M. (eds.) Web Services, pp. 254–269. Springer, Heidelberg (2004) 13. Li, Q., Liu, A., Liu, H., Lin, B., Huang, L., Gu, N.: Web Services Provision: Solutions, Challenges and Opportunities (invited paper). In: 3rd International Conference on Ubiquitous Information Management and Communication, Suwon, Korea, pp. 80–87 (2009) 14. Martin, D., Burstein, M., McDermott, D., McIlraith, S., Paolucci, M., Sycara, K., et al.: Bringing Semantics to Web Services with OWL-S. World Wide Web 10, 243–277 (2007) 15. Nayak, R., Iryadi, W.: XML Schema Clustering with Semantic and Hierarchical Similarity Measures. Knowledge-Based Systems 20, 336–349 (2007) 16. Nayak, R., Lee, B.: Web Service Discovery with Additional Semantics and Clustering. In: 9th IEEE/WIC/ACM International Conference on Web Intelligence, Fremont, USA, pp. 555–558 (2007) 17. Shafiq, O., Moran, M., Cimpian, E., Mocan, A., Zaremba, M., Fensel, D.: Investigating Semantic Web Service Execution Environments: A Comparison between WSMX and OWL-S Tools. In: 2nd International Conference on Internet and Web Applications and Services, Morne, Mauritius, pp. 31–36 (2007) 18. Van Rijsbergen, C.J., Robertson, S.E., Porter, M.F.: New Models in Probabilistic Information Retrieval. British Library, London (1980) 19. Wu, C., Potdar, V., Chang, E.: Latent Semantic Analysis - The Dynamics of Semantics Web Services Discovery. In: Advances in Web semantics I: Ontologies, Web Services and Applied Semantic Web, pp. 346–373. Springer, Heidelberg (2009)
The BUAP Participation at the Web Service Discovery Track of INEX 2010 Mar´ıa Josefa Somodevilla, Beatriz Beltr´an, David Pinto, Darnes Vilari˜ no, and Jos´e Cruz Aaron Faculty of Computer Science Benem´erita Universidad Aut´ onoma de Puebla, M´exico {mariasg,bbeltran,dpinto,darnes}@cs.buap.mx
Abstract. A first approach for web services discovering based on techniques from Information Retrieval (IR), Natural Language Processing (NLP) and XML Retrieval was developed in order to use texts contained in WSDL files. It calculates the degree of similarity between words and their relative importance to support the task of web services discovering. The first algorithm uses the information contained in the WSDL (Web Service Description Language) specifications and clusters web services based on their similarity. A second approach based on a information retrieval system that index terms by using an inverted index structure was also used. Both algorithms are applied in order to evaluate 25 topics in a set of 1947 real web services (all of them provided by INEX).
1
Introduction
The Service Oriented Architecture (SOA)1 was developed based on the concept of a wide mesh of collaborating services, published and available for invocation. Web services are the set of protocols by which services are published, discovered, and used in a technology independent, standard form. As the number of web services repositories grows and the number of available services expands, finding the web service that one needs has become a key task within the invocation process. Web service discovery is concerned with locating web services that match a set of functional and non-functional criteria [1]. The Web Services Description Language (WSDL) 2 is the most basic mechanism used to describe web services. This leads many current discovery approaches to focus on locating web services based on their functional description. An efficient and effective Web services discovery mechanism is important in many computing paradigms including Pervasive Computing, Service-Oriented
1 2
This work has been partially supported by projects: CONACYT #106625, VIEP #SOGJ-ING11-I, #BEMB-ING11-I, as well as by the PROMEP/103.5/09/4213 grant. http://opengroup.org/projects/soa/doc.tpl?gdid=10632 http://www.w3.org/TR/wsdl.html
S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 347–350, 2011. c Springer-Verlag Berlin Heidelberg 2011
348
M.J. Somodevilla et al.
Computing, and the most recent Cloud Computing, in which Web services constitute the chief building blocks. The Web Service Discovery track aims to investigate techniques for discovery of Web services based on searching service descriptions provided in Web Services Description Language (WSDL) . Participating groups will contribute to topic development and evaluation, which will then allow them to compare the effectiveness of their XML retrieval techniques for the discovery of Web services. This will lead to the development of a test collection that will allow participating groups to undertake future comparative experiments. The rest of this paper is devoted to explain the two different approaches submitted to the competition, as well as the dataset used in the experiments.
2
Description of the Presented Approaches
Two algorithms based on Clustering and Information Retrieval in order to find the most appropiate web service (WSDL file) to a given topic were developed. The description of the approach that uses a clustering method follows. 1. Tag removal: A corpus was built with the content of the XML tags for each document, in addition to the attribute values of the labels. 2. Parsing WSDL: Stopwords and punctuation symbols were removed from the corpus. 3. Tokenization: The Maximum Matching Algorithm (MMA) algorithm was applied [2], using a list of 53,000 English words which split them into tokens (i.e. GetAllitems as Get All Items). 4. Re-parsing WSDL: Stopwords and punctuation symbols were removed from the corpus again due to the MMA decomposition. 5. Word stemming: The Porter stemming algorithm was applied to the corpus. 6. K-means algorithm: K=2 was used; the distance criterion NGD is presented in Eq. (1), and the convergence criterion is that the centroid words are at least twice in different iterations. 7. Content word recognition: Thereafter, we removed the words of the cluster with minimal elements (i.e. service, SOA, array and data). 8. Services corpus creation: A second corpus was constructed with the services of each XML file; again we used the MMA algorithm, we eliminate stopwords, and finally we applied the Porter algorithm. 9. Query answering: Using the two corpus constructed, a query can be answered by applying Eq. (2), and then sorting the results from lowest to highest. NGD(x, y) =
max {logf (x), logf (y)} − logf (x, y) logM − min {logf (x), logf (y)}
O(Si , Sj ) = 0.5 ∗ S (Si , Sj ) + 0.5 ∗ S (Si , Sj )
where:
S (Si , Sj ) =
a∈Si
b∈Sj
Sim(a, b)
|Si ||Sj |
(1) (2)
(3)
The BUAP Participation at the Web Service Discovery Track of INEX 2010
S (Si , Sj ) = 1 − NGD(Si , Sj )
349
(4)
Following we describe the approach that uses information retrieval for finding the corresponding web services files that satisfies the user needs. The implementation based on NLP uses an inverted index for storing all the terms detected in the WSDL files. For the case of function names, we have also used the MMA algorithm. Each term is used as the dictionary entry in the data structure, and one posting list is attached to each dictionary entry. Finally, given a query, we may calculate the intersection between pairs of posting lists (p1 and p2 ) as shown in the Algorithm 1 (taken from [3]).
Algorithm 1. Intersection of two posting lists
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
3
Input: Posting lists p1 and p2 Output: Relevant documents D1 , D2 , · · · answer = while p1 ! = NIL and p2 ! =NIL do if docID(p1 ) = docID(p2 ) then ADD(answer, docID(p1 )); p1 = next(p1 ); p2 = next(p2 ); else if docID(p1 ) < docID(p2 ) then p1 = next(p1 ) else p2 = next(p2 ) end end end return answer
Experimental Results
In Figure 1 we may see the interpolated precision, whereas in Table 1 we show the mean average precision obtained by the teams at the competition. The clusteringbased approach obtained a low performance, and, therefore, we will discard the use of this technique in future experiments. With respect to the other approach, we did not obtained an official evaluation due to a format problem with our output file. However, a preliminar evaluation (using our own evaluation tools) show a very good performance. In summary, we consider that the use of techniques of information retrieval obtains the best performance in this kind of task, when the correct features are extracted from wsdl files.
350
M.J. Somodevilla et al.
Fig. 1. Interpolated precision at standard recall levels Table 1. Results at the INEX Web Service Discovery Track Num 1 2 3 4 5 6 7 8 9 10 11 12
4
map 0.3469 0.3239 0.2946 0.2798 0.2798 0.2348 0.2233 0.2042 0.1451 0.1268 0.1095 0.0937
Institute Kasetsart University RMIT University Hasso-Plattner-Institut Kasetsart University Kasetsart University Queensland Univ. of Technology Queensland Univ. of Technology Queensland Univ. of Technology B. Univ. Automa de Puebla Queensland Univ. of Technology Queensland Univ. of Technology Queensland Univ. of Technology
Run Kas I138BM25ESS010 RMIT10WS HPI2010 Kas I138ANYSS025 Kas I138ANYBM25SS015 QUT BM25WordNetComposition QUT BM25WordNet QUT BM25WordNetCompWordNet BUAPFCCWSD01 QUT Wikipedia QUT WikipediaComposition QUT WikipediaCompositionWordNet
Conclusions
We have presented details about the implemented approaches for tackling the problem of webservice discovery. Two different approaches were implemented, one based on clustering and the second on information retrieval techniques. In general we may see that the second approach behaves better than the one based on clustering.
References 1. Dan, A., Davis, D., Kearney, R., Keller, A., King, R., Kuebler, D., Ludwig, H., Polan, M., Spreitzer, M., Youssef, A.: Web services on demand: Wsla-driven automated management. IBM Systems Journal 43(1), 136–158 (2004) 2. Guodong, Z.: A chunking strategy towards unknown word detection in chinese word segmentation. In: Dale, R., Wong, K.-F., Su, J., Kwong, O.Y. (eds.) IJCNLP 2005. LNCS (LNAI), vol. 3651, pp. 530–541. Springer, Heidelberg (2005) 3. Manning, C.D., Raghavan, P., Sch¨ utze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2009)
XML Retrieval More Efficient Using Double Scoring Scheme Tanakorn Wichaiwong and Chuleerat Jaruskulchai Department of Computer Science, Faculty of Science, Kasetsart University, Bangkok Thailand {g5184041,fscichj}@ku.ac.th
Abstract. This is the first year of Kasetsart University's participation in INEX. We participated in three tracks: the Ad Hoc, the Data Centric, and the Web Service Discovery tracks. In the Ad hoc and Data Centric tracks, the BM25F function has performed well in the past evaluations. However, there is an issue that needs more attention; what is the weight for each field? Previously, the weights are given manually to the fields. In general, many experts feel that uncontrolled selected fields. We proposed an unsupervised implementation of the BM25F scoring function, which we call the Double Scoring function. This scoring function assigns the weight for each field by an extended indexing scheme that handles the tuned weight parameter for each selected field. In the Web Service Discovery track, the standard tokens cannot be utilised directly by IR models, and need to be converted to natural language before indexing. We applied the Capitalization function to solve the tokenisation issue. Keywords: XML Retrieval, Information Retrieval, Indexing Units, Ranking Schemes.
1 Introduction The widespread use of Extensible Markup Language (XML) [1] documents in digital libraries has led to the development of information retrieval (IR) methods specifically designed for XML collections. Most traditional IR systems are limited to whole document retrieval; however, since XML documents separate content and structure, XML-IR systems are able to retrieve only the relevant portions of the documents. Therefore, users who utilise an XML-IR system could potentially receive highly relevant and precise material. In the Initiative for the Evaluation of XML Retrieval (INEX) [2] reports, the BM25F function has performed well in past evaluations. However, there is an issue that needs more attention: what is the weight for each field? Previously, the weights were obtained manually, tuning to the fields that affect the cost, the time preprocessing complexity and the heterogeneity of collections. Furthermore, these steps require training data and an evaluation set for parameter tuning, and difficulty often arises when evaluating new collections. Therefore, we have developed automatic methods to assign the weight for each field with an extended indices scheme. S. Geva et al. (Eds.): INEX 2010, LNCS 6932, pp. 351–362, 2011. © Springer-Verlag Berlin Heidelberg 2011
352
T. Wichaiwong and C. Jaruskulchai
This paper is organised as follows: Section 2 reviews related works, and Section 3 explains the implementation of our system. Section 4 presents the experiment. Conclusions and recommendations for further work are provided in Section 5.
2 Related Works Prior work found in [3, 4, 5] presents the BM25F as the fielded version of the BM25 scoring function. Compared to the classical BM25, improvements have been discovered of up to 65% as measured by nxCG on the INEX-IEEE collection, with a different task where overlap is allowed. Authors of [7] applied the BM25F to INEXWikipedia’s collection; on each element, they constructed two fields, one for “title” and another for “body”. The title field consists of a concatenation of an article title and any section titles. The body field contains another context. In the INEX-2009 reports [8], the results showed the best ranking, with an iP [0.01] of 0.6333. 2.1 BM25F Function Overview Robertson et al. [3, 4, 5, 6] presented the BM25F as the fielded version of the BM25 scoring function, which is considered to be composed of several document fields with possibly different degrees of importance, called selected fields. Using the BM25F scheme, an element's score is computed as follows: ,
25
,
(1)
Note that; BM25F(e) measures the relevance of element e to a query q. q is a set of query terms. Xe,t is a weighted normalised term frequency. K is a common tuning parameter for BM25. Wt is the inverse document frequency weight of a term t. The weighted normalised term frequency is obtained by first performing length normalisation, on the term frequency We,f,t of a term t of a field f in an element e, , ,
, ,
1
,
1
(2)
Note that: Bf is a parameter to tune. le,f is a length of a field f in an element e. lf is the average length of elements in the whole collection multiplied the normalised term frequency We,f,t by a field weight Wf , ,
, ,
(3)
XML Retrieval More Efficient Using Double Scoring Scheme
353
3 XML Retrieval Model 3.1 Inverted File Definition The Zettair search engine has good performance in [9], but this engine only supports the document level. We have to convert the element level to the document level by the absolute XPath [10], with context replacement to the tag in the Zettair TREC format; our system is then able to use the best features of Zettair [11]. When we replace all tags to by the absolute XPath, then leaf-only indexing is closest to traditional IR because each XML node is a bag of words itself and can be scored as an ordinary plain text document. Then we calculate the leaf element score for its context using BM25. For instance, take a document named x1. article
xml xml
title
body
id
xml
xml
1
section
xml
information
retrieval
