public class PartitionLexically extends Object
A global index is partitioned lexically by providing a
that specifies a destination local index for each term, and a local term number. The global index
is read directly at the bit level, and the posting lists are divided among the
local indices using the provided strategy. For instance,
ContiguousLexicalStrategy divides an index into
contiguous blocks (of terms) specified by the given strategy.
By choice, document pointers are not remapped. Thus, it may happen that one of the local indices contains no posting with a certain document. However, computing the subset of documents contained in each local index to remap them in a contiguous interval is not a good idea, as usually the subset of documents appearing in the postings of each local index is large.
To speed up the search of the right local index of a not-so-frequent term (in
particular with a chained strategy),
after partitioning an index you can create Bloom filters that will be used to try to avoid
inquiring indices that do not contain a term. The filters will be automatically loaded
Index.getInstance(CharSequence, boolean, boolean).
Note that the size file is the same for each local index and is not copied. Please use standard operating system features such as symbolic links to provide size files to local indices.
If you plan to cluster the partitioned indices and you need document sizes
(e.g., for BM25 scoring), you can use the index property
Index.UriKeys.SIZES to load the original size file.
If you plan on partitioning an index requiring
document sizes, you should consider a custom index loading scheme
that shares the size list
among all local indices.
Important: this class just partitions the index. No auxiliary files (most notably, term maps
or prefix maps) will be generated. Please refer to a
StringMap implementation (e.g.,
The partitioning process writes each index file sequentially exactly once, so index partitioning can output its results to pipes, which in turn can spill their content, for instance, through the network. In other words, albeit this class theoretically creates a number of local indices on disk, those indices can be substituted with suitable pipes creating remote local indices without affecting the partitioning process. For instance, the following bash code creates three sets of pipes:
for i in 0 1 2; do for e in frequencies occurrencies index offsets properties sizes terms; do mkfifo pipe-$i.$e done done
Each pipe must be emptied elsewhere, for instance (assuming you want local indices index0, index1 and index2 on example.com):
for i in 0 1 2; do for e in frequencies occurrencies index offsets properties sizes terms; do (cat pipe-$i.$e | ssh -x example.com "cat >index-$i.$e" &) done done
If we now start a partitioning process generating three local indices named pipe-0, pipe-1 and pipe-2 all pipes will be written to by the process, and the data will create remotely indices index-0, index-1 and index-2.
|Modifier and Type||Class and Description|
|Modifier and Type||Field and Description|
The default buffer size for all involved indices.
|Constructor and Description|
|Modifier and Type||Method and Description|
public static final int DEFAULT_BUFFER_SIZE
public void run() throws org.apache.commons.configuration.ConfigurationException, IOException, ClassNotFoundException
public static void main(String arg) throws com.martiansoftware.jsap.JSAPException, org.apache.commons.configuration.ConfigurationException, IOException, ClassNotFoundException, SecurityException, InstantiationException, IllegalAccessException