Principled Multi-Aspect Evaluation Measures of Rankings
Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt
Dokumenter
- Fulltext
Forlagets udgivne version, 1,4 MB, PDF-dokument
Information Retrieval evaluation has traditionally focused on defining principled ways of assessing the relevance of a ranked list of documents with respect to a query. Several methods extend this type of evaluation beyond relevance, making it possible to evaluate different aspects of a document ranking (e.g., relevance, usefulness, or credibility) using a single measure (multi-aspect evaluation). However, these methods either are (i) tailor-made for specific aspects and do not extend to other types or numbers of aspects, or (ii) have theoretical anomalies, e.g. assign maximum score to a ranking where all documents are labelled with the lowest grade with respect to all aspects (e.g., not relevant, not credible, etc.). We present a theoretically principled multi-aspect evaluation method that can be used for any number, and any type, of aspects. A thorough empirical evaluation using up to 5 aspects and a total of 425 runs officially submitted to 10 TREC tracks shows that our method is more discriminative than the state-of-the-art and overcomes theoretical limitations of the state-of-the-art.
Originalsprog | Engelsk |
---|---|
Titel | CIKM 2021 - Proceedings of the 30th ACM International Conference on Information and Knowledge Management |
Forlag | Association for Computing Machinery, Inc |
Publikationsdato | 2021 |
Sider | 1232-1242 |
ISBN (Elektronisk) | 9781450384469 |
DOI | |
Status | Udgivet - 2021 |
Begivenhed | 30th ACM International Conference on Information and Knowledge Management, CIKM 2021 - Virtual, Online, Australien Varighed: 1 nov. 2021 → 5 nov. 2021 |
Konference
Konference | 30th ACM International Conference on Information and Knowledge Management, CIKM 2021 |
---|---|
Land | Australien |
By | Virtual, Online |
Periode | 01/11/2021 → 05/11/2021 |
Sponsor | ACM SIGIR, ACM SIGWEB |
Bibliografisk note
Funding Information:
Acknowledgments. This paper is partially supported by the EU Horizon 2020 research and innovation programme under the MSCA grant No. 893667.
Publisher Copyright:
© 2021 Owner/Author.
Antal downloads er baseret på statistik fra Google Scholar og www.ku.dk
ID: 300918675