public class NRTCachingDirectory extends FilterDirectory implements Accountable
RAMDirectory
around any provided delegate directory, to
be used during NRT search.
This class is likely only useful in a near-real-time context, where indexing rate is lowish but reopen rate is highish, resulting in many tiny files being written. This directory keeps such segments (as well as the segments produced by merging them, as long as they are small enough), in RAM.
This is safe to use: when your app calls {IndexWriter#commit}, all cached files will be flushed from the cached and sync'd.
Here's a simple example usage:
Directory fsDir = FSDirectory.open(new File("/path/to/index").toPath()); NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(fsDir, 5.0, 60.0); IndexWriterConfig conf = new IndexWriterConfig(analyzer); IndexWriter writer = new IndexWriter(cachedFSDir, conf);
This will cache all newly flushed segments, all merges
whose expected segment size is <= 5 MB
, unless the net
cached bytes exceeds 60 MB at which point all writes will
not be cached (until the net bytes falls below 60 MB).
Modifier and Type | Field and Description |
---|---|
private RAMDirectory |
cache |
private long |
maxCachedBytes |
private long |
maxMergeSizeBytes |
private java.lang.Object |
uncacheLock |
private static boolean |
VERBOSE |
in
Constructor and Description |
---|
NRTCachingDirectory(Directory delegate,
double maxMergeSizeMB,
double maxCachedMB)
We will cache a newly created output if 1) it's a
flush or a merge and the estimated size of the merged segment is
<= maxMergeSizeMB , and 2) the total cached bytes is
<= maxCachedMB |
Modifier and Type | Method and Description |
---|---|
void |
close()
Close this directory, which flushes any cached files
to the delegate and then closes the delegate.
|
IndexOutput |
createOutput(java.lang.String name,
IOContext context)
Creates a new, empty file in the directory with the given name.
|
IndexOutput |
createTempOutput(java.lang.String prefix,
java.lang.String suffix,
IOContext context)
Creates a new, empty file for writing in the directory, with a
temporary file name including prefix and suffix, ending with the
reserved extension
.tmp . |
void |
deleteFile(java.lang.String name)
Removes an existing file in the directory.
|
protected boolean |
doCacheWrite(java.lang.String name,
IOContext context)
Subclass can override this to customize logic; return
true if this file should be written to the RAMDirectory.
|
long |
fileLength(java.lang.String name)
Returns the length of a file in the directory.
|
java.util.Collection<Accountable> |
getChildResources()
Returns nested resources of this class.
|
java.lang.String[] |
listAll()
Returns an array of strings, one for each entry in the directory, in sorted (UTF16, java's String.compare) order.
|
java.lang.String[] |
listCachedFiles() |
IndexInput |
openInput(java.lang.String name,
IOContext context)
Returns a stream reading an existing file.
|
long |
ramBytesUsed()
Return the memory usage of this object in bytes.
|
void |
rename(java.lang.String source,
java.lang.String dest)
Renames
source to dest as an atomic operation,
where dest does not yet exist in the directory. |
(package private) static boolean |
slowFileExists(Directory dir,
java.lang.String fileName)
Returns true if the file exists
(can be opened), false if it cannot be opened, and
(unlike Java's File.exists) throws IOException if
there's some unexpected error.
|
void |
sync(java.util.Collection<java.lang.String> fileNames)
Ensure that any writes to these files are moved to
stable storage.
|
java.lang.String |
toString() |
private void |
unCache(java.lang.String fileName) |
getDelegate, obtainLock, syncMetaData, unwrap
copyFrom, ensureOpen, openChecksumInput
private final RAMDirectory cache
private final long maxMergeSizeBytes
private final long maxCachedBytes
private static final boolean VERBOSE
private final java.lang.Object uncacheLock
public NRTCachingDirectory(Directory delegate, double maxMergeSizeMB, double maxCachedMB)
<= maxMergeSizeMB
, and 2) the total cached bytes is
<= maxCachedMB
public java.lang.String toString()
toString
in class FilterDirectory
public java.lang.String[] listAll() throws java.io.IOException
Directory
listAll
in class FilterDirectory
java.io.IOException
- in case of IO errorpublic void deleteFile(java.lang.String name) throws java.io.IOException
Directory
deleteFile
in class FilterDirectory
java.io.IOException
public long fileLength(java.lang.String name) throws java.io.IOException
Directory
FileNotFoundException
or NoSuchFileException
if the file does not exist.
fileLength
in class FilterDirectory
name
- the name of the file for which to return the length.java.io.IOException
- if there was an IO error while retrieving the file's
length.public java.lang.String[] listCachedFiles()
public IndexOutput createOutput(java.lang.String name, IOContext context) throws java.io.IOException
Directory
createOutput
in class FilterDirectory
java.io.IOException
public void sync(java.util.Collection<java.lang.String> fileNames) throws java.io.IOException
Directory
sync
in class FilterDirectory
java.io.IOException
public void rename(java.lang.String source, java.lang.String dest) throws java.io.IOException
Directory
source
to dest
as an atomic operation,
where dest
does not yet exist in the directory.
Notes: This method is used by IndexWriter to publish commits.
It is ok if this operation is not truly atomic, for example
both source
and dest
can be visible temporarily.
It is just important that the contents of dest
appear
atomically, or an exception is thrown.
rename
in class FilterDirectory
java.io.IOException
public IndexInput openInput(java.lang.String name, IOContext context) throws java.io.IOException
Directory
Throws FileNotFoundException
or NoSuchFileException
if the file does not exist.
openInput
in class FilterDirectory
java.io.IOException
public void close() throws java.io.IOException
close
in interface java.io.Closeable
close
in interface java.lang.AutoCloseable
close
in class FilterDirectory
java.io.IOException
protected boolean doCacheWrite(java.lang.String name, IOContext context)
public IndexOutput createTempOutput(java.lang.String prefix, java.lang.String suffix, IOContext context) throws java.io.IOException
Directory
.tmp
. Use
IndexOutput.getName()
to see what name was used.createTempOutput
in class FilterDirectory
java.io.IOException
static boolean slowFileExists(Directory dir, java.lang.String fileName) throws java.io.IOException
java.io.IOException
private void unCache(java.lang.String fileName) throws java.io.IOException
java.io.IOException
public long ramBytesUsed()
Accountable
ramBytesUsed
in interface Accountable
public java.util.Collection<Accountable> getChildResources()
Accountable
getChildResources
in interface Accountable
Accountables