Keynote Talks

• “Would you trust your IR system to choose your date?” Re-thinking IR Evaluation in the 21st Century
  Elaine G. Toms, University of Sheffield, UK
  
  Abstract:
  In the quest for almost any concept or object, human nature is surprisingly and paradoxically predictable: people will either know it when they see it, or they know what it should be before they find it. Yet, the concept of the “right” document, photo or indeed any information object is in the mind of the seeker regardless of whether the seeker is looking for dinner, a new car, a date, a novel to read, or a solution to a work problem. To complicate the matter, that solution is also a moving target, and not necessarily a single reality out there to be uncovered. Once an object is found, whether it is pertinent, relevant, accurate or correct is a human judgment made at a particular moment in time for a particular purpose. An information retrieval (IR) system can only provide suggestions; its role is to support and facilitate, and not to replace the human decision-making process. In the evaluation of interactive IR systems, this requires a directional shift from the typical TREC, INEX and even CLEF evaluation processes. At present, evaluation has moved from an emphasis on topical relevance, to an emphasis on measuring almost anything that can be quantified. This is more likely to be data extracted from transaction logs in an attempt to develop a more predictable personalized search with likely the same accuracy as forecasting the future by reading tealeaves in a cup! We have failed to step back and assess the broader picture. What exactly are we evaluating and for what purpose? It could be said that we have focused far too long on the tool and not on what the tool will be used for. For example, if that IR system was choosing ones date, a restaurant or a medical treatment, would we use the same evaluation techniques in use today? This talk will consider evaluation from that holistic and contextual perspective. It will examine some of the pitfalls in existing approaches, and discuss the issues involved in designing more effective evaluation approaches for assessing interactive IR systems.
  
  
  


• Crowdsourcing for Information Retrieval Experimentation and Evaluation
  Omar Alonso, Microsoft Corporation, California, USA
  
  Abstract:
  Very recently, crowdsourcing has emerged as a viable alter- native for conducting different types of experiments in a wide range of areas. Generally speaking and in the context of IR, crowdsourcing in- volves outsourcing tasks to a large group of people instead of assign- ing such tasks to an employee or editor. The availability of commercial crowdsourcing platforms offers vast access to an on-demand workforce. This new approach makes possible to conduct experiments extremely fast, with good results at a low cost. However, like in any experiment, there are several implementation details that would make an experiment work or fail. For large scale evaluation, deployment in practice is not that simple. Tasks have to be designed carefully with special emphasis on the user interface, instructions, content, and quality control. In this invited talk, I will explore some directions that may influence the outcome of a task and I will present a framework for conducting crowdsourcing experiments making some emphasis on a number of as- pects that should be of importance for all sorts of IR-like tasks. Finally, I will outline research trends around human computation that promise to make this emerging field even more interesting in the near future.