the ultimate federated search test collection

Test topics

The collection folder meta_data/topics contains the following files:

  • FWxx-topics-official.txt : official topics (released to track participants)
  • FWxx-topics-evaluation-TREC.txt : subset of official topics, fully judged for evaluation.
  • FW14-topics-evaluation-online.txt : list of annotated topics for the FW14 online evaluation system.
  • FW-topics.xml : detailed information on all 506 FW13 and FW14 official and extra topics.

The topics in FW-topics.xml have the following format:

<topic evaluation="TREC" id="7320" official="FedWeb14" origin="trec09+millionquery">
  <query>cayenne pepper plants</query>
  <description>You are looking for information on growing cayenne pepper plants.</description>
  <narrative>You want to grow your own cayenne pepper plants, starting from a cayenne pepper, and you're looking for advice (written guidelines / videos / ...) on how to do so.</narrative>
</topic>

Whereby

  • id= is the topic ID
  • official= indicates to what subcollection the considered topic was an official topic ("FedWeb13", "FedWeb14", or "None")
  • origin= indicates the origin of the test topic ("new", "trec09+web", "trec09+millionquery", or "trec10+web")
  • <query> gives the query terms
  • <description> gives a short description of the information need
  • <narrative> provides additional information for the assessors, typically outlining the reason behind the query (by sketching a particular situation), and sometimes hinting at verticals that are more/less relevant. Narratives are available for most official evaluation topics (unless the description is clear and extensive enough), and not for most others.