Evaluating FTFs and other query systems

Criteria

When we started work on FTFs, we proposed that we should use the following criteria to evaluate grammatical query systems (in order of importance, from most to least).

  1. Linguistic adequacy.

    Queries are sufficiently expressive to capture linguistically meaningful research questions.

  2. Transparency.

    Queries are clear and meaningful to end users, to whom any new system will be unfamiliar.

  3. General expressivity.

    One can express a broad range of queries that extends beyond current linguistic concerns.

    NB. With the release of ICECUP 3.1, the expressivity of FTFs has been extended in a number of ways - in particular, to permit the expression of logic in nodes and the use of lexical wild cards. It was not found necessary to extend the set of links and edges.

  4. Computational efficiency.

    Naturally, we all prefer a rapid response to a poor one. In practice, however, it is not the proof algorithm that determines the speed but the disk access time.

How good are FTFs against these criteria?

  • Do you think that FTFs are adequate for ‘linguistically meaningful' directed queries?
  • Are FTFs sufficiently transparent, and is there anything we could do about this? Our research indicates that some users have difficulty with the ‘ancestor’ option for parent-child relations because the FTF is more ‘elastic’ and doesn’t match so intuitively against the tree.

Note that this kind of evaluation cannot be totally separated from the structure of the ICE grammar. Apart from the fact that FTFs in ICECUP are ‘abstract ICE trees’, many users find that they do not know the grammar sufficiently to form ‘definitive’ queries easily. This is also what prompted us to design ICECUP around what we call an ‘exploratory cycle’ (see Corpus Query report).

FTF home pages by Sean Wallis and Gerry Nelson.
Comments/questions to s.wallis@ucl.ac.uk.

This page last modified 12 June, 2013 by Survey Web Administrator.