it.unimi.di.mg4j.index
Class MultiTermIndexIterator

java.lang.Object
  extended by it.unimi.di.mg4j.search.AbstractDocumentIterator
      extended by it.unimi.di.mg4j.search.AbstractIntervalDocumentIterator
          extended by it.unimi.di.mg4j.search.AbstractCompositeDocumentIterator
              extended by it.unimi.di.mg4j.search.AbstractUnionDocumentIterator
                  extended by it.unimi.di.mg4j.index.MultiTermIndexIterator
All Implemented Interfaces:
IndexIterator, DocumentIterator

public class MultiTermIndexIterator
extends AbstractUnionDocumentIterator
implements IndexIterator

A virtual index iterator that merges several component index iterators.

This class adds to AbstractUnionDocumentIterator an interval iterator generating the OR of the intervals returned for each of the documents involved. The main difference with an OrDocumentIterator built on the same array of component iterators is that this class implements IndexIterator and hence provides a count() (the sum of counts of those component iterators positioned on the current document), frequency() and nextPosition(). The frequency is by default the maximum frequency of a component iterator, but it can be set at construction time.

The main raison d'être of this class is support for query expansion: a blind application of OrDocumentIterator to an array of index iterators would mislead scorers such as BM25Scorer because low-frequency terms (e.g., hapax legomena) would be responsible for most of the score.

Note that DocumentIteratorVisitor has a visit method for generic index iterator and a visit method for instances of this class. This approach provides additional flexibility—a scorer, for instance, might treat an instance of this class as a standard IndexIterator, or it might choose to query which terms actually appear and do something more sophisticated (for instance, using weights).


Nested Class Summary
 
Nested classes/interfaces inherited from class it.unimi.di.mg4j.search.AbstractUnionDocumentIterator
AbstractUnionDocumentIterator.IntHeapSemiIndirectPriorityQueue
 
Nested classes/interfaces inherited from class it.unimi.di.mg4j.search.AbstractCompositeDocumentIterator
AbstractCompositeDocumentIterator.AbstractCompositeIndexIntervalIterator, AbstractCompositeDocumentIterator.AbstractCompositeIntervalIterator
 
Field Summary
protected  int id
          The id of this iterator.
protected  String term
          The term of this iterator.
 
Fields inherited from class it.unimi.di.mg4j.search.AbstractUnionDocumentIterator
front, frontSize, queue, refArray
 
Fields inherited from class it.unimi.di.mg4j.search.AbstractCompositeDocumentIterator
documentIterator, indexIterator, indexIteratorsWithoutPositions, n
 
Fields inherited from class it.unimi.di.mg4j.search.AbstractIntervalDocumentIterator
currentIterators, indices, intervalIterators, soleIndex, soleIntervalIterator, unmodifiableCurrentIterators
 
Fields inherited from class it.unimi.di.mg4j.search.AbstractDocumentIterator
curr, weight
 
Fields inherited from interface it.unimi.di.mg4j.index.IndexIterator
END_OF_POSITIONS
 
Fields inherited from interface it.unimi.di.mg4j.search.DocumentIterator
END_OF_LIST
 
Constructor Summary
protected MultiTermIndexIterator(int defaultFrequency, IndexIterator... indexIterator)
          Creates a new document iterator that merges the given array of iterators.
 
Method Summary
<T> T
accept(DocumentIteratorVisitor<T> visitor)
          Accepts a visitor.
<T> T
acceptDeep(DocumentIteratorVisitor<T> visitor)
           
<T> T
acceptDeepOnTruePaths(DocumentIteratorVisitor<T> visitor)
           
<T> T
acceptOnTruePaths(DocumentIteratorVisitor<T> visitor)
          Invokes AbstractUnionDocumentIterator.acceptOnTruePaths(DocumentIteratorVisitor) only on component iterators positioned on the current document.
 int count()
          The count is the sum of counts of those component iterators positioned on the current document.
 int frequency()
          The frequency is either the default frequency set at construction time, or the maximum frequency of the component iterators.
 int front(IndexIterator[] indexIterator)
          Fills the given array with the index iterators composing the current front.
static IndexIterator getInstance(Index index, IndexIterator... indexIterator)
          Returns an index iterator that merges the given array of iterators.
static IndexIterator getInstance(IndexIterator... indexIterator)
          Returns an index iterator that merges the given array of iterators.
static IndexIterator getInstance(int defaultFrequency, Index index, IndexIterator... indexIterator)
          Returns an index iterator that merges the given array of iterators.
static IndexIterator getInstance(int defaultFrequency, IndexIterator... indexIterator)
          Returns an index iterator that merges the given array of iterators.
protected  IntervalIterator getIntervalIterator(Index index, int n, boolean allIndexIterators, Object unused)
          Creates an interval iterator suitable for this AbstractIntervalDocumentIterator.
 int id()
          Returns the id of this index iterator.
 IndexIterator id(int id)
          Sets the id of this index iterator.
 Index index()
          Returns the index over which this iterator is built.
 IntervalIterator intervalIterator()
          Returns the interval iterator of this document iterator for single-index queries.
 IntervalIterator intervalIterator(Index index)
          Returns the interval iterator of this document iterator for the given index.
 int nextDocument()
          Returns the next document provided by this document iterator, or DocumentIterator.END_OF_LIST if no more documents are available.
 int nextPosition()
          Returns the next position at which the term appears in the current document.
 Payload payload()
          Returns the payload, if any, associated with the current document.
 int skipTo(int n)
          Skips all documents smaller than n.
 String term()
          Returns the term whose inverted list is returned by this index iterator.
 IndexIterator term(CharSequence term)
          Sets the term whose inverted list is returned by this index iterator.
 int termNumber()
          Returns the number of the term whose inverted list is returned by this index iterator.
 IndexIterator weight(double weight)
          Sets the weight of this index iterator.
 
Methods inherited from class it.unimi.di.mg4j.search.AbstractUnionDocumentIterator
computeFront, intervalIterators
 
Methods inherited from class it.unimi.di.mg4j.search.AbstractCompositeDocumentIterator
dispose, toString
 
Methods inherited from class it.unimi.di.mg4j.search.AbstractIntervalDocumentIterator
allIndexIterators, indices, indices
 
Methods inherited from class it.unimi.di.mg4j.search.AbstractDocumentIterator
document, ensureOnADocument, mayHaveNext, weight
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 
Methods inherited from interface it.unimi.di.mg4j.search.DocumentIterator
dispose, document, indices, intervalIterators, mayHaveNext, weight
 

Field Detail

term

protected String term
The term of this iterator.


id

protected int id
The id of this iterator.

Constructor Detail

MultiTermIndexIterator

protected MultiTermIndexIterator(int defaultFrequency,
                                 IndexIterator... indexIterator)
                          throws IOException
Creates a new document iterator that merges the given array of iterators.

Parameters:
defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).
indexIterator - the iterators to be joined.
Throws:
IOException
Method Detail

getInstance

public static IndexIterator getInstance(IndexIterator... indexIterator)
                                 throws IOException
Returns an index iterator that merges the given array of iterators. This method requires that at least one iterator is provided. The frequency is computed as a max, and index() will return the result of the same method on the first iterator.

Parameters:
indexIterator - the iterators to be joined (at least one).
Returns:
a merged index iterator.
Throws:
IllegalArgumentException - if no iterators were provided.
IOException

getInstance

public static IndexIterator getInstance(Index index,
                                        IndexIterator... indexIterator)
                                 throws IOException
Returns an index iterator that merges the given array of iterators.

Note that the special case of the empty and of the singleton arrays are handled efficiently. The frequency is computed as a max, and index() will return index.

Parameters:
index - the index that wil be returned by index().
indexIterator - the iterators to be joined.
Returns:
a merged index iterator.
Throws:
IOException

getInstance

public static IndexIterator getInstance(int defaultFrequency,
                                        IndexIterator... indexIterator)
                                 throws IOException
Returns an index iterator that merges the given array of iterators. This method requires that at least one iterator is provided.

Parameters:
defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).
indexIterator - the iterators to be joined (at least one).
Returns:
a merged index iterator.
Throws:
IllegalArgumentException - if no iterators were provided, or they run on different indices.
IOException

getInstance

public static IndexIterator getInstance(int defaultFrequency,
                                        Index index,
                                        IndexIterator... indexIterator)
                                 throws IOException
Returns an index iterator that merges the given array of iterators.

Note that the special case of the empty and of the singleton arrays are handled efficiently.

Parameters:
defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).
index - the index that wil be returned by index().
indexIterator - the iterators to be joined.
Returns:
a merged index iterator.
Throws:
IllegalArgumentException - if there is some iterator on an index different from index.
IOException

getIntervalIterator

protected IntervalIterator getIntervalIterator(Index index,
                                               int n,
                                               boolean allIndexIterators,
                                               Object unused)
Description copied from class: AbstractIntervalDocumentIterator
Creates an interval iterator suitable for this AbstractIntervalDocumentIterator.

Specified by:
getIntervalIterator in class AbstractIntervalDocumentIterator
Parameters:
index - the reference index for the iterator, or null.
n - the number of underlying or component iterators.
allIndexIterators - whether all underlying or component iterators are index iterators.
unused - an optional argument.
Returns:
an interval iterator suitable for this AbstractIntervalDocumentIterator.

skipTo

public int skipTo(int n)
           throws IOException
Description copied from interface: DocumentIterator
Skips all documents smaller than n.

Define the current document k associated with this document iterator as follows:

If k is larger than or equal to n, then this method does nothing and returns k. Otherwise, a call to this method is equivalent to

 while( ( k = nextDocument() ) < n );
 return k;
 

Thus, when a result kDocumentIterator.END_OF_LIST is returned, the state of this iterator will be exactly the same as after a call to DocumentIterator.nextDocument() that returned k. In particular, the first document larger than or equal to n (when returned by this method) will not be returned by the next call to DocumentIterator.nextDocument().

Specified by:
skipTo in interface DocumentIterator
Overrides:
skipTo in class AbstractUnionDocumentIterator
Parameters:
n - a document pointer.
Returns:
a document pointer larger than or equal to n if available, DocumentIterator.END_OF_LIST otherwise.
Throws:
IOException

nextDocument

public int nextDocument()
                 throws IOException
Description copied from interface: DocumentIterator
Returns the next document provided by this document iterator, or DocumentIterator.END_OF_LIST if no more documents are available.

Specified by:
nextDocument in interface DocumentIterator
Overrides:
nextDocument in class AbstractUnionDocumentIterator
Returns:
the next document, or DocumentIterator.END_OF_LIST if no more documents are available.
Throws:
IOException

count

public int count()
          throws IOException
The count is the sum of counts of those component iterators positioned on the current document.

Specified by:
count in interface IndexIterator
Returns:
the sum of counts.
Throws:
IOException

front

public int front(IndexIterator[] indexIterator)
Fills the given array with the index iterators composing the current front.

This method is essentially a safe exposure of the front of the queue merging the component index iterators. After a call to nextDocument(), you can use this method to know which terms actually appear in the current document. You can use the public field AbstractCompositeDocumentIterator.n to size the argument array appropriately.

Parameters:
indexIterator - an array, at least large as the number of component index iterators, that will be partially filled with the index iterators corresponding to terms appearing in the current document.
Returns:
the number of iterators written into indexIterator.

frequency

public int frequency()
              throws IOException
The frequency is either the default frequency set at construction time, or the maximum frequency of the component iterators.

Specified by:
frequency in interface IndexIterator
Returns:
the frequency.
Throws:
IOException

term

public IndexIterator term(CharSequence term)
Description copied from interface: IndexIterator
Sets the term whose inverted list is returned by this index iterator.

Usually, the term is automatically set by Index.documents(CharSequence) or by IndexReader.documents(CharSequence), but you can use this method to ensure that IndexIterator.term() doesn't throw an exception.

Specified by:
term in interface IndexIterator
Parameters:
term - a character sequence (that will be defensively copied) that will be assumed to be the term whose inverted list is returned by this index iterator.
Returns:
this index iterator.

term

public String term()
Description copied from interface: IndexIterator
Returns the term whose inverted list is returned by this index iterator.

Usually, the term is automatically set by IndexReader.documents(CharSequence) or IndexReader.documents(int), but you can supply your own term with IndexIterator.term(CharSequence).

Specified by:
term in interface IndexIterator
Returns:
the term over which this iterator is built, as a compact mutable string.
See Also:
IndexIterator.termNumber()

termNumber

public int termNumber()
Description copied from interface: IndexIterator
Returns the number of the term whose inverted list is returned by this index iterator.

Usually, the term number is automatically set by IndexReader.documents(CharSequence) or IndexReader.documents(int).

Specified by:
termNumber in interface IndexIterator
Returns:
the number of the term over which this iterator is built.
See Also:
IndexIterator.term()

id

public IndexIterator id(int id)
Description copied from interface: IndexIterator
Sets the id of this index iterator.

The id is an integer associated with each index iterator. It has no specific semantics, and can be used differently in different contexts. A typical usage pattern, for instance, is using it to assign a unique number to the index iterators contained in a composite document iterator (say, numbering consecutively the leaves of the composite).

Specified by:
id in interface IndexIterator
Parameters:
id - the new id for this index iterator.
Returns:
this index iterator.

id

public int id()
Description copied from interface: IndexIterator
Returns the id of this index iterator.

Specified by:
id in interface IndexIterator
Returns:
the id of this index iterator.
See Also:
IndexIterator.id(int)

index

public Index index()
Description copied from interface: IndexIterator
Returns the index over which this iterator is built.

Specified by:
index in interface IndexIterator
Returns:
the index over which this iterator is built.

payload

public Payload payload()
Description copied from interface: IndexIterator
Returns the payload, if any, associated with the current document.

Specified by:
payload in interface IndexIterator
Returns:
the payload associated with the current document.

weight

public IndexIterator weight(double weight)
Description copied from interface: DocumentIterator
Sets the weight of this index iterator.

Specified by:
weight in interface IndexIterator
Specified by:
weight in interface DocumentIterator
Overrides:
weight in class AbstractDocumentIterator
Parameters:
weight - the weight of this index iterator.
Returns:
this document iterator.
See Also:
DocumentIterator.weight(double)

accept

public <T> T accept(DocumentIteratorVisitor<T> visitor)
         throws IOException
Description copied from interface: DocumentIterator
Accepts a visitor.

A document iterator is usually structured as composite, with operators as internal nodes and IndexIterators as leaves. This method implements the visitor pattern.

Specified by:
accept in interface DocumentIterator
Overrides:
accept in class AbstractCompositeDocumentIterator
Parameters:
visitor - the visitor.
Returns:
an object resulting from the visit, or null if the visit was interrupted.
Throws:
IOException

acceptOnTruePaths

public <T> T acceptOnTruePaths(DocumentIteratorVisitor<T> visitor)
                    throws IOException
Description copied from class: AbstractUnionDocumentIterator
Invokes AbstractUnionDocumentIterator.acceptOnTruePaths(DocumentIteratorVisitor) only on component iterators positioned on the current document.

Specified by:
acceptOnTruePaths in interface DocumentIterator
Overrides:
acceptOnTruePaths in class AbstractUnionDocumentIterator
Parameters:
visitor - a visitor.
Returns:
true if the visit should continue.
Throws:
IOException
See Also:
DocumentIterator.accept(DocumentIteratorVisitor), CounterCollectionVisitor

acceptDeep

public <T> T acceptDeep(DocumentIteratorVisitor<T> visitor)
             throws IOException
Throws:
IOException

acceptDeepOnTruePaths

public <T> T acceptDeepOnTruePaths(DocumentIteratorVisitor<T> visitor)
                        throws IOException
Throws:
IOException

nextPosition

public int nextPosition()
                 throws IOException
Description copied from interface: IndexIterator
Returns the next position at which the term appears in the current document.

Specified by:
nextPosition in interface IndexIterator
Returns:
the next position of the current document in which the current term appears, or IndexIterator.END_OF_POSITIONS if there are no more positions.
Throws:
IOException

intervalIterator

public IntervalIterator intervalIterator()
Description copied from interface: DocumentIterator
Returns the interval iterator of this document iterator for single-index queries.

This is a commodity method that can be used only for queries built over a single index.

Specified by:
intervalIterator in interface DocumentIterator
Returns:
an interval iterator.
See Also:
DocumentIterator.intervalIterator(Index)

intervalIterator

public IntervalIterator intervalIterator(Index index)
Description copied from interface: DocumentIterator
Returns the interval iterator of this document iterator for the given index.

After a call to DocumentIterator.nextDocument(), this iterator can be used to retrieve the intervals in the current document (the one returned by DocumentIterator.nextDocument()) for the index index.

Note that if all indices have positions, it is guaranteed that at least one index will return an interval. However, for disjunctive queries it cannot be guaranteed that all indices will return an interval.

Indices without positions always return IntervalIterators.TRUE. Thus, in presence of indices without positions it is possible that no intervals at all are available.

Specified by:
intervalIterator in interface DocumentIterator
Parameters:
index - an index (must be one over which the query was built).
Returns:
an interval iterator over the current document in index.