public class CounterCollectionVisitor extends AbstractDocumentIteratorVisitorA visitor collecting the counts of terms in a
Several scoring schemes, such as BM25 or cosine-based measures, require the counts (number of occurrences in a given document) of terms in the query. Since we do not want restrict the ability of the user to specify sophisticated constraints such as term proximity, order, consecutivity, etc., we prefer not to use a bag-of-words query model, in which the user simply inputs a number of terms (in that case, of course, the definition of the count of each term is trivial). Rather, we provide a group of visitors that make it possible to retrieve counts for each term appearing in the query.
Since MG4J provides multi-index queries, each count is actually associated with a pair index/term (e.g., the count of class in the main text might be different from the count of class in the title). Moreover, we must be careful to define a sensible semantics, as when logical operators alternate there might be occurrences of a term in a query whose count might give misleading information (in particular if the same term appear several times).
Thus, we define a true path on the query tree (which parallels the composite tree of the associated
DocumentIterator) as a path from the root that passes only through nodes whose associated subquery evaluates to true (in the Boolean sense). A counter-collection visitor records in the counter arrays only the counts of index/term pairs appearing at the end of a true path.
For instance, in a query like a OR (title:b AND c) in a document that contains a and c in the main text, but does not contain b in the title, only the count of a will be taken into consideration. In the same way, for a query whose outmost operation is a negation no counter will ever be written.
Preparing and using counters
- after creating the
DocumentIterator, prepare a
TermCollectionVisitorand perform a visit to gather term information and possibly cache some data about the terms appearing in the iterator;
CounterSetupVisitorbased on the previous
TermCollectionVisitor, and perform a visit to read frequencies and prepare counters;
- start iterating: after each call to
nextDocument(), clear the counters, perform a visit along true paths using an instance of this class and inspect the data gathered in the CounterSetupVisitor (see, for example, the source code of
Note that all visitors are reusable: just
prepare()them before usage, but be careful as a
CounterSetupVisitormust be prepared and visited after the associated
TermCollectionVisitorhas been prepared and visited. The
prepare()method of this class is a no-op, so it is not necessary to call it.
- after creating the
public CounterCollectionVisitor(CounterSetupVisitor counterSetupVisitor)Creates a new counter-collection visitor based on a given counter-setup visitor.
counterSetupVisitor- a counter-setup visitor.
public Boolean visit(IndexIterator indexIterator) throws IOExceptionDescription copied from interface:
indexIterator- the leaf to be visited.
- an appropriate return value if the visit should continue, or