Package it.unimi.di.big.mg4j.tool

Class PartitionLexically

  • public class PartitionLexically
extends Object
    Partitions an index lexically.

    A global index is partitioned lexically by providing a LexicalPartitioningStrategy that specifies a destination local index for each term, and a local term number. The global index is read directly at the bit level, and the posting lists are divided among the local indices using the provided strategy. For instance, an ContiguousLexicalStrategy divides an index into contiguous blocks (of terms) specified by the given strategy.

    By choice, document pointers are not remapped. Thus, it may happen that one of the local indices contains no posting with a certain document. However, computing the subset of documents contained in each local index to remap them in a contiguous interval is not a good idea, as usually the subset of documents appearing in the postings of each local index is large.

    To speed up the search of the right local index of a not-so-frequent term (in particular with a chained strategy), after partitioning an index you can create Bloom filters that will be used to try to avoid inquiring indices that do not contain a term. The filters will be automatically loaded by Index.getInstance(CharSequence, boolean, boolean).

    Note that the size file is the same for each local index and is not copied. Please use standard operating system features such as symbolic links to provide size files to local indices.

    If you plan to cluster the partitioned indices and you need document sizes (e.g., for BM25 scoring), you can use the index property Index.UriKeys.SIZES to load the original size file. If you plan on partitioning an index requiring document sizes, you should consider a custom index loading scheme that shares the size list among all local indices. Important: this class just partitions the index. No auxiliary files (most notably, term maps or prefix maps) will be generated. Please refer to a StringMap implementation (e.g., ShiftAddXorSignedStringMap or ImmutableExternalPrefixMap).

    Write-once output and distributed index partitioning

    The partitioning process writes each index file sequentially exactly once, so index partitioning can output its results to pipes, which in turn can spill their content, for instance, through the network. In other words, albeit this class theoretically creates a number of local indices on disk, those indices can be substituted with suitable pipes creating remote local indices without affecting the partitioning process. For instance, the following bash code creates three sets of pipes: 

     for i in 0 1 2; do
   for e in frequencies occurrencies index offsets properties sizes terms; do 
     mkfifo pipe-$i.$e
   done
 done

    Each pipe must be emptied elsewhere, for instance (assuming you want local indices index0, index1 and index2 on example.com): 

     for i in 0 1 2; do 
   for e in frequencies occurrencies index offsets properties sizes terms; do 
     (cat pipe-$i.$e | ssh -x example.com "cat >index-$i.$e" &)
   done
 done

    If we now start a partitioning process generating three local indices named pipe-0, pipe-1 and pipe-2 all pipes will be written to by the process, and the data will create remotely indices index-0, index-1 and index-2.

    Since:
    1.0.1
    Author:
    Sebastiano Vigna