Setup Time

Once the index has been created, there are many ways in which you can improve query resolution time. First of all, an index can be read from disk, memory-mapped, or directly loaded into main memory. These three solutions work with increasing speed and increased main memory usage. The default for quasi-succinct indices is to map the index in memory. Otherwise, the default is to read the index from disk. You can add suitable options to the index URI (e.g., mapped=1 or inmemory=1―see the Index.UriKeys documentation) to force your preferences. Analogously, offsets are necessary to locate, inside the index file, the posting list of a certain term. By default they are read from disk using a SemiExternalOffsetList, but you can load them in memory if you prefer so using the offsetstep parameter. If you load sizes (e.g., because you want to run a scorer that needs them) there is a suitable URI option (e.g., succinctsizes=1) that will load sizes in a highly compact format. This is particularly useful when pasting large indices.

To get more options, you can partition your index (see below). Once you have a cluster formed by several sub-indices, you can decide which sub-indices go to memory, which will be mapped, and so on.

An important source of delay in loading the index is the expansion of the dump file of an ImmutableExternalPrefixMap, which is the default term map generated by IndexBuilder. The dump file must be copied from the serialized representation to a temporary directory, and for large collections the process can be very slow. The solution is either to use a different term map (e.g., some kind of signed hash—see the minimal perfect hash classes of Sux4J) to generate (either programmatically or using the main method of ImmutableExternalPrefixMap) a non-self-contained, synchronized instance of ImmutableExternalPrefixMap and save it using the standard suffix for term maps. Such an instance is based on a separate dump file that must be attached to the deserialized instance before usage (see the documentation for details). You can attach the dump stream by invoking

((ImmutableExternalPrefixMap)index.termMap).setDumpStream( filename );

with the appropriate argument.