Package jadex.collection
Class SharedPersistentMap<K,V>
java.lang.Object
jadex.collection.SharedPersistentMap<K,V>
- All Implemented Interfaces:
Map<K,
V>
A file-based map that can be shared across the operating system
using file locks.
KVNode Layout:
[Prev. Node offset 8 byte]
[Next Node offset 8 byte]
[Key Object Node offset 8 byte]
[Value Object Node offset 8 byte]
Object Node Layout:
[Object size 8 byte]
[Object data variable]
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic interface
Lambda interface throwing IOExceptions.static interface
Lambda interface throwing IOExceptions without any input or output.static interface
Lambda interface throwing IOExceptions without map being passed.protected class
View on a KVNode. -
Field Summary
FieldsModifier and TypeFieldDescriptionprotected Function
<ByteBuffer, Object> The decoder functionality.protected static final Function
<ByteBuffer, Object> protected static final Function
<Object, ByteBuffer> Default Encoder using Java serialization.protected Map
<Long, MappedByteBuffer> Objects directly mapped with their own mappingprotected Function
<Object, ByteBuffer> The serialization/encoder functionality.protected File
The file backing the mapprotected static final int
Position of the data pointer, pointing to the next empty byte in the file.protected static final int
Position of the garbage size in the header.protected static final int
Position of the map size in the header.protected static final int
Position of the map size in the header.protected static final int
Position of the magic word in the header.protected static final int
Position of the map structure version in the header.protected static final int
Size of the header: [4 bytes Magic Word] [4 bytes Map Structure Version] [8 byte Index Size] [8 bytes Map Size/Elements] [8 bytes garbage size] [104 bytes reserved]protected MappedByteBuffer
Memory mapping of the header.protected static final long[]
Table of index sizes.protected MappedByteBuffer
Memory mapping of the index.protected long
protected static final int
Magic word starting the file.protected static final byte
Type of map - normal shared map.protected List
<MappedByteBuffer> Mappings for the rest of the file.protected jadex.common.RwAutoLock
Lock for the mappings list.protected int
Structure version of the map to detect changes in the indexing/map structure performed by other processes / instances operating on the same file.protected static final double
Default maximum percentage of garbage in the map file that needs to be cleaned up.protected static final int
Maximum index exponent, limit: hashCode() returns int.protected static final double
Default maximum percentage for the load factor of the hash table before re-indexing.protected static final int
Maximum size per mapping = 256MiB.protected static final int
Bit shift defining size per mapping = 256MiB.protected double
Factor (not percentage) for maximum garbage tolerated.protected double
Maximum load factor (not percentage) of the index.protected static final long
Minimum file size (header size + initial index size).protected static final double
Default minimum percentage for the load factor of the hash table before re-indexing.protected static final int
Maximum size per mapping = 64KiB.protected double
Minimum load factor (not percentage) of the index.protected String
Current file access mode.protected static final int
Size of references in the map (8 bytes / long) and size values.protected static final int
Bit shift to convert a counter to a reference offset.protected static final int
Reserved heap sizeprotected RandomAccessFile
The memory-mapped file when opened.protected static final ThreadLocal
<jadex.collection.SharedPersistentMap.ReentrantLock> Thread local for tracking the write lock for reentrant acquisition. -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprotected long
absoluteToHeap
(long absolute) Converts an absolute position to a heap positionprotected void
addGarbage
(long garbage) Adds an amount to the garbage counter.protected void
addSize
(long size) Adds an amount to the map item size.protected SharedPersistentMap<K,
V>.KVNode appendKVPair
(K key) Appends a new KV node.void
clear()
Clears the map, deleting all elements.close()
Closes the backing file, after this operation the map must be opened again before use.boolean
containsKey
(Object key) Checks if a key is contained in the map.boolean
containsValue
(Object value) Checks if a value is contained in the map.entrySet()
Returns the entry set for the Map.protected ByteBuffer
fill
(ByteBuffer b, byte fillval) Fills a ByteBuffer with a byte value.protected SharedPersistentMap<K,
V>.KVNode findKVPair
(Object key) Finds the key-value pair that contains the given key.Gets the value for a given key.protected int
getHeaderInt
(int headerpos) Reads an int-value from the header.protected long
getHeaderLong
(int headerpos) Reads a long-value from the header.protected MappedByteBuffer
long
getIndexPosition
(int hash) protected long
getIndexReference
(long indexposition) Reads the positional reference at a given position in the hashtable index.protected long
getIndexSizeForMapSize
(long mapsize) Returns the appropriate index size for the size of the map.protected MappedByteBuffer
getMappedBuffer
(long absposition, long lsize, boolean append) protected long
Returns the next 8byte-aligned starting position after the current end of file.protected long
heapToAbsolute
(long heap) Converts an absolute position to a heap positionprotected void
Initializes the map file as an empty map.protected void
initializeMap
(long isize) Initializes the map file as an empty map with a specified index size.boolean
isEmpty()
Returns true, if this map contains no key-value mappings.keySet()
Returns the key set for the Map.static void
final int
nextPowerOfTwo
(int x) Calculates the next power of two that's equal or larger than x.open()
Opens the backing file, after this operation the map is ready for use.protected void
Performs maintenance tasks if required.Adds a new key-value pair to the map, if the key already exists, the associated value is overwritten.void
Adds all entries of another map to this one.protected void
putHeaderLong
(int headerpos, long value) Writes a long-value from the header.protected long
Reads the current index size in long words.jadex.common.IAutoLock
readLock()
Engages the read lock over the whole file.<R> R
readTransaction
(SharedPersistentMap.IOFunction<SharedPersistentMap<K, V>, R> transaction) Performs a read transaction on the map.void
readTransaction
(SharedPersistentMap.IORunnable transaction) Performs a read transaction on the map.<R> R
readTransaction
(SharedPersistentMap.IOSupplier<R> transaction) Performs a read transaction on the map.Removes key from the map.setDecoder
(Function<ByteBuffer, Object> decoder) Sets the encoder for decoding objects.setEncoder
(Function<Object, ByteBuffer> encoder) Sets the encoder for encoding/serializing objects.Sets the file backing the map.setLoadPercentage
(double percentage) Sets a percentage for the maximum load of the index (load factor), default: 70%setMaxGarbage
(double percentage) Sets the maximum percentage of the file that can be garbage before garbage collection is triggered (default: 20%).setSynchronized
(boolean sync) Configures the map for synchronized/non-synchronized writes.int
size()
Returns the number of key-values in this map.values()
Returns the values of the Map.protected void
Verifies the map file and memory mapping state.jadex.common.IAutoLock
Engages the write lock over the whole file.<R> R
writeTransaction
(SharedPersistentMap.IOFunction<SharedPersistentMap<K, V>, R> transaction) Performs a write transaction on the map.void
writeTransaction
(SharedPersistentMap.IORunnable transaction) Performs a write transaction on the map.<R> R
writeTransaction
(SharedPersistentMap.IOSupplier<R> transaction) Performs a write transaction on the map.Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface java.util.Map
compute, computeIfAbsent, computeIfPresent, equals, forEach, getOrDefault, hashCode, merge, putIfAbsent, remove, replace, replace, replaceAll
-
Field Details
-
MAX_GARBAGE_PERCENTAGE
protected static final double MAX_GARBAGE_PERCENTAGEDefault maximum percentage of garbage in the map file that needs to be cleaned up.- See Also:
-
MIN_LOAD_FACTOR
protected static final double MIN_LOAD_FACTORDefault minimum percentage for the load factor of the hash table before re-indexing.- See Also:
-
MAX_LOAD_FACTOR
protected static final double MAX_LOAD_FACTORDefault maximum percentage for the load factor of the hash table before re-indexing.- See Also:
-
MAGIC_WORD
protected static final int MAGIC_WORDMagic word starting the file.- See Also:
-
MAP_TYPE
protected static final byte MAP_TYPEType of map - normal shared map.- See Also:
-
WRITE_LOCK
Thread local for tracking the write lock for reentrant acquisition. -
REF_SIZE
protected static final int REF_SIZESize of references in the map (8 bytes / long) and size values.- See Also:
-
INDEX_SIZES
protected static final long[] INDEX_SIZESTable of index sizes. -
HEADER_SIZE
protected static final int HEADER_SIZESize of the header: [4 bytes Magic Word] [4 bytes Map Structure Version] [8 byte Index Size] [8 bytes Map Size/Elements] [8 bytes garbage size] [104 bytes reserved]- See Also:
-
HDR_POS_MGC_WRD
protected static final int HDR_POS_MGC_WRDPosition of the magic word in the header.- See Also:
-
HDR_POS_STR_VER
protected static final int HDR_POS_STR_VERPosition of the map structure version in the header.- See Also:
-
HDR_POS_IDX_SIZE
protected static final int HDR_POS_IDX_SIZEPosition of the map size in the header.- See Also:
-
HDR_POS_MAP_SIZE
protected static final int HDR_POS_MAP_SIZEPosition of the map size in the header.- See Also:
-
HDR_POS_GARBAGE_SIZE
protected static final int HDR_POS_GARBAGE_SIZEPosition of the garbage size in the header.- See Also:
-
HDR_POS_DATA_POINTER
protected static final int HDR_POS_DATA_POINTERPosition of the data pointer, pointing to the next empty byte in the file.- See Also:
-
REF_SIZE_SHIFT
protected static final int REF_SIZE_SHIFTBit shift to convert a counter to a reference offset. -
RESERVED_HEAP_SIZE
protected static final int RESERVED_HEAP_SIZEReserved heap size- See Also:
-
MAX_INDEX_EXP
protected static final int MAX_INDEX_EXPMaximum index exponent, limit: hashCode() returns int.- See Also:
-
MAX_MAP_SHIFT
protected static final int MAX_MAP_SHIFTBit shift defining size per mapping = 256MiB.- See Also:
-
MAX_MAP
protected static final int MAX_MAPMaximum size per mapping = 256MiB.- See Also:
-
MIN_MAP
protected static final int MIN_MAPMaximum size per mapping = 64KiB.- See Also:
-
MIN_FILE_SIZE
protected static final long MIN_FILE_SIZEMinimum file size (header size + initial index size). -
DEF_ENCODER
Default Encoder using Java serialization. -
DEF_DECODER
-
file
The file backing the map -
mode
Current file access mode. -
rfile
The memory-mapped file when opened. -
encoder
The serialization/encoder functionality. -
decoder
The decoder functionality. -
maxgarbagefactor
protected double maxgarbagefactorFactor (not percentage) for maximum garbage tolerated. -
minloadfactor
protected double minloadfactorMinimum load factor (not percentage) of the index. -
maxloadfactor
protected double maxloadfactorMaximum load factor (not percentage) of the index. -
headermap
Memory mapping of the header. -
indexmap
Memory mapping of the index. -
mappings
Mappings for the rest of the file. -
mappingslock
protected jadex.common.RwAutoLock mappingslockLock for the mappings list. -
directmappings
Objects directly mapped with their own mapping -
mapstructversion
protected int mapstructversionStructure version of the map to detect changes in the indexing/map structure performed by other processes / instances operating on the same file. -
indexsizecache
protected volatile long indexsizecache
-
-
Constructor Details
-
SharedPersistentMap
public SharedPersistentMap()Creates a new map, configure with builder pattern.
-
-
Method Details
-
setFile
Sets the file backing the map.- Parameters:
path
- P- Returns:
-
setFile
-
setLoadPercentage
Sets a percentage for the maximum load of the index (load factor), default: 70%- Parameters:
percentage
- Desired load percentage.- Returns:
-
setMaxGarbage
Sets the maximum percentage of the file that can be garbage before garbage collection is triggered (default: 20%).- Parameters:
percentage
- Maximum allowed percentage of garbage.- Returns:
- This map.
-
setEncoder
Sets the encoder for encoding/serializing objects.- Parameters:
encoder
- Encoder for encoding/serializing objects.- Returns:
- This map.
-
setDecoder
Sets the encoder for decoding objects.- Parameters:
decoder
- Encoder for decoding objects.- Returns:
- This map.
-
setSynchronized
Configures the map for synchronized/non-synchronized writes. If true, writes to the map are immediately written to persistent storage. This will reduce the data loss in case of a crash.- Parameters:
sync
- True, if writes should be written to storage immediately.- Returns:
- This map.
-
open
Opens the backing file, after this operation the map is ready for use.- Returns:
- This map.
-
close
Closes the backing file, after this operation the map must be opened again before use.- Returns:
- This map.
-
get
Gets the value for a given key. -
containsKey
Checks if a key is contained in the map.- Specified by:
containsKey
in interfaceMap<K,
V> - Parameters:
key
- The key.- Returns:
- True, if the key is contained, even if the associated value is null, false otherwise.
-
containsValue
Checks if a value is contained in the map.- Specified by:
containsValue
in interfaceMap<K,
V> - Parameters:
value
- The value.- Returns:
- True, if the map contains the value.
-
put
Adds a new key-value pair to the map, if the key already exists, the associated value is overwritten. -
putAll
Adds all entries of another map to this one. -
remove
Removes key from the map. -
entrySet
Returns the entry set for the Map. WARNING: Do not use without holding at least a read lock while getting and using the returned set. -
keySet
Returns the key set for the Map. WARNING: This will load all keys into memory, please consider using entrySet() instead. WARNING: Do not use without holding at least a read lock while getting and using the returned set. -
values
Returns the values of the Map. WARNING: This will load all values into memory, please consider using entrySet() instead. WARNING: Do not use without holding at least a read lock while getting and using the returned collection. -
size
public int size()Returns the number of key-values in this map. -
isEmpty
public boolean isEmpty()Returns true, if this map contains no key-value mappings. -
clear
public void clear()Clears the map, deleting all elements. -
readLock
Engages the read lock over the whole file.- Returns:
- The lock, close to release.
- Throws:
IOException
- Thrown on IO issues.
-
writeLock
Engages the write lock over the whole file.- Returns:
- The lock, close to release.
- Throws:
IOException
- Thrown on IO issues.
-
verifyMapState
protected void verifyMapState()Verifies the map file and memory mapping state. -
performMaintenanceIfRequired
Performs maintenance tasks if required. Note: Write lock must be held before method is called.- Throws:
IOException
- Thrown on IO issues.
-
getIndexSizeForMapSize
protected long getIndexSizeForMapSize(long mapsize) Returns the appropriate index size for the size of the map.- Parameters:
mapsize
- Map size.indexsize
- Current index size.- Returns:
- Correct index size.
-
readIndexSize
protected long readIndexSize()Reads the current index size in long words. NOTE: Called MUST lock the file with, at minimum, a read lock.- Returns:
- Index size or -1, if the file is invalid or empty.
-
initializeMap
Initializes the map file as an empty map.- Throws:
IOException
- Thrown on IO issues.
-
initializeMap
Initializes the map file as an empty map with a specified index size.- Parameters:
isize
- Index size.- Throws:
IOException
- Thrown on IO issues.
-
findKVPair
Finds the key-value pair that contains the given key.- Parameters:
key
- The key to look up.- Returns:
- The KVNode containing the key or null if not found.
- Throws:
IOException
- Thrown on IO issues.
-
appendKVPair
Appends a new KV node. Note: This only adds the node, not the key. There is also no check if the key is already in the map.- Parameters:
key
- The key that needs the KVNode.- Returns:
- KVNode object for the newly created node.
- Throws:
IOException
- Thrown on IO issues.
-
absoluteToHeap
protected long absoluteToHeap(long absolute) Converts an absolute position to a heap position- Parameters:
absolute
- Absolute position in the file.- Returns:
- Position in the heap.
-
heapToAbsolute
protected long heapToAbsolute(long heap) Converts an absolute position to a heap position- Parameters:
heap
- Position in the heap.- Returns:
- Absolute position in the file.
-
getIndexPosition
public long getIndexPosition(int hash) -
getIndexReference
Reads the positional reference at a given position in the hashtable index.- Parameters:
indexposition
- Position in the index.- Returns:
- KVNode reference at the given index position.
- Throws:
IOException
-
getHeaderInt
protected int getHeaderInt(int headerpos) Reads an int-value from the header.- Parameters:
headerpos
- Position in the header.- Returns:
- Value at the position.
-
getHeaderLong
protected long getHeaderLong(int headerpos) Reads a long-value from the header.- Parameters:
headerpos
- Position in the header.- Returns:
- Value at the position.
-
putHeaderLong
protected void putHeaderLong(int headerpos, long value) Writes a long-value from the header.- Parameters:
headerpos
- Position in the header.value
- Value to write at the position.
-
addGarbage
Adds an amount to the garbage counter.- Parameters:
garbage
- Amount of garbage to add to the garbage counter.- Throws:
IOException
- Thrown on IO issues.
-
addSize
Adds an amount to the map item size.- Parameters:
size
- Amount of garbage to added to the map size.- Throws:
IOException
- Thrown on IO issues.
-
getNextAlignedPosition
Returns the next 8byte-aligned starting position after the current end of file.- Returns:
- The next aligned starting position after the current end of file.
- Throws:
IOException
-
fill
Fills a ByteBuffer with a byte value.- Parameters:
b
- ByteBuffer object.fillval
- Fill value.
-
getMappedBuffer
protected MappedByteBuffer getMappedBuffer(long absposition, long lsize, boolean append) throws IOException - Throws:
IOException
-
getIndexMap
- Throws:
IOException
-
nextPowerOfTwo
public final int nextPowerOfTwo(int x) Calculates the next power of two that's equal or larger than x. Note: Does not handle zero or negative values.- Parameters:
x
- Any positive int value- Returns:
- Next power of two.
-
main
-