Class SharedPersistentMap<K,V>

java.lang.Object
jadex.collection.SharedPersistentMap<K,V>
All Implemented Interfaces:
Map<K,V>

public class SharedPersistentMap<K,V> extends Object implements Map<K,V>
A file-based map that can be shared across the operating system using file locks. KVNode Layout: [Prev. Node offset 8 byte] [Next Node offset 8 byte] [Key Object Node offset 8 byte] [Value Object Node offset 8 byte] Object Node Layout: [Object size 8 byte] [Object data variable]
  • Field Details

    • MAX_GARBAGE_PERCENTAGE

      protected static final double MAX_GARBAGE_PERCENTAGE
      Default maximum percentage of garbage in the map file that needs to be cleaned up.
      See Also:
    • MIN_LOAD_FACTOR

      protected static final double MIN_LOAD_FACTOR
      Default minimum percentage for the load factor of the hash table before re-indexing.
      See Also:
    • MAX_LOAD_FACTOR

      protected static final double MAX_LOAD_FACTOR
      Default maximum percentage for the load factor of the hash table before re-indexing.
      See Also:
    • MAGIC_WORD

      protected static final int MAGIC_WORD
      Magic word starting the file.
      See Also:
    • MAP_TYPE

      protected static final byte MAP_TYPE
      Type of map - normal shared map.
      See Also:
    • WRITE_LOCK

      protected static final ThreadLocal<jadex.collection.SharedPersistentMap.ReentrantLock> WRITE_LOCK
      Thread local for tracking the write lock for reentrant acquisition.
    • REF_SIZE

      protected static final int REF_SIZE
      Size of references in the map (8 bytes / long) and size values.
      See Also:
    • INDEX_SIZES

      protected static final long[] INDEX_SIZES
      Table of index sizes.
    • HEADER_SIZE

      protected static final int HEADER_SIZE
      Size of the header: [4 bytes Magic Word] [4 bytes Map Structure Version] [8 byte Index Size] [8 bytes Map Size/Elements] [8 bytes garbage size] [104 bytes reserved]
      See Also:
    • HDR_POS_MGC_WRD

      protected static final int HDR_POS_MGC_WRD
      Position of the magic word in the header.
      See Also:
    • HDR_POS_STR_VER

      protected static final int HDR_POS_STR_VER
      Position of the map structure version in the header.
      See Also:
    • HDR_POS_IDX_SIZE

      protected static final int HDR_POS_IDX_SIZE
      Position of the map size in the header.
      See Also:
    • HDR_POS_MAP_SIZE

      protected static final int HDR_POS_MAP_SIZE
      Position of the map size in the header.
      See Also:
    • HDR_POS_GARBAGE_SIZE

      protected static final int HDR_POS_GARBAGE_SIZE
      Position of the garbage size in the header.
      See Also:
    • HDR_POS_DATA_POINTER

      protected static final int HDR_POS_DATA_POINTER
      Position of the data pointer, pointing to the next empty byte in the file.
      See Also:
    • REF_SIZE_SHIFT

      protected static final int REF_SIZE_SHIFT
      Bit shift to convert a counter to a reference offset.
    • RESERVED_HEAP_SIZE

      protected static final int RESERVED_HEAP_SIZE
      Reserved heap size
      See Also:
    • MAX_INDEX_EXP

      protected static final int MAX_INDEX_EXP
      Maximum index exponent, limit: hashCode() returns int.
      See Also:
    • MAX_MAP_SHIFT

      protected static final int MAX_MAP_SHIFT
      Bit shift defining size per mapping = 256MiB.
      See Also:
    • MAX_MAP

      protected static final int MAX_MAP
      Maximum size per mapping = 256MiB.
      See Also:
    • MIN_MAP

      protected static final int MIN_MAP
      Maximum size per mapping = 64KiB.
      See Also:
    • MIN_FILE_SIZE

      protected static final long MIN_FILE_SIZE
      Minimum file size (header size + initial index size).
    • DEF_ENCODER

      protected static final Function<Object,ByteBuffer> DEF_ENCODER
      Default Encoder using Java serialization.
    • DEF_DECODER

      protected static final Function<ByteBuffer,Object> DEF_DECODER
    • file

      protected File file
      The file backing the map
    • mode

      protected String mode
      Current file access mode.
    • rfile

      protected RandomAccessFile rfile
      The memory-mapped file when opened.
    • encoder

      protected Function<Object,ByteBuffer> encoder
      The serialization/encoder functionality.
    • decoder

      protected Function<ByteBuffer,Object> decoder
      The decoder functionality.
    • maxgarbagefactor

      protected double maxgarbagefactor
      Factor (not percentage) for maximum garbage tolerated.
    • minloadfactor

      protected double minloadfactor
      Minimum load factor (not percentage) of the index.
    • maxloadfactor

      protected double maxloadfactor
      Maximum load factor (not percentage) of the index.
    • headermap

      protected MappedByteBuffer headermap
      Memory mapping of the header.
    • indexmap

      protected volatile MappedByteBuffer indexmap
      Memory mapping of the index.
    • mappings

      protected volatile List<MappedByteBuffer> mappings
      Mappings for the rest of the file.
    • mappingslock

      protected jadex.common.RwAutoLock mappingslock
      Lock for the mappings list.
    • directmappings

      protected Map<Long,MappedByteBuffer> directmappings
      Objects directly mapped with their own mapping
    • mapstructversion

      protected int mapstructversion
      Structure version of the map to detect changes in the indexing/map structure performed by other processes / instances operating on the same file.
    • indexsizecache

      protected volatile long indexsizecache
  • Constructor Details

    • SharedPersistentMap

      public SharedPersistentMap()
      Creates a new map, configure with builder pattern.
  • Method Details

    • setFile

      public SharedPersistentMap<K,V> setFile(String path)
      Sets the file backing the map.
      Parameters:
      path - P
      Returns:
    • setFile

      public SharedPersistentMap<K,V> setFile(File file)
    • setLoadPercentage

      public SharedPersistentMap<K,V> setLoadPercentage(double percentage)
      Sets a percentage for the maximum load of the index (load factor), default: 70%
      Parameters:
      percentage - Desired load percentage.
      Returns:
    • setMaxGarbage

      public SharedPersistentMap<K,V> setMaxGarbage(double percentage)
      Sets the maximum percentage of the file that can be garbage before garbage collection is triggered (default: 20%).
      Parameters:
      percentage - Maximum allowed percentage of garbage.
      Returns:
      This map.
    • setEncoder

      public SharedPersistentMap<K,V> setEncoder(Function<Object,ByteBuffer> encoder)
      Sets the encoder for encoding/serializing objects.
      Parameters:
      encoder - Encoder for encoding/serializing objects.
      Returns:
      This map.
    • setDecoder

      public SharedPersistentMap<K,V> setDecoder(Function<ByteBuffer,Object> decoder)
      Sets the encoder for decoding objects.
      Parameters:
      decoder - Encoder for decoding objects.
      Returns:
      This map.
    • setSynchronized

      public SharedPersistentMap<K,V> setSynchronized(boolean sync)
      Configures the map for synchronized/non-synchronized writes. If true, writes to the map are immediately written to persistent storage. This will reduce the data loss in case of a crash.
      Parameters:
      sync - True, if writes should be written to storage immediately.
      Returns:
      This map.
    • open

      public SharedPersistentMap<K,V> open()
      Opens the backing file, after this operation the map is ready for use.
      Returns:
      This map.
    • close

      public SharedPersistentMap<K,V> close()
      Closes the backing file, after this operation the map must be opened again before use.
      Returns:
      This map.
    • get

      public V get(Object key)
      Gets the value for a given key.
      Specified by:
      get in interface Map<K,V>
      Parameters:
      key - The key to be found.
      Returns:
      The associated value or null if not found.
    • containsKey

      public boolean containsKey(Object key)
      Checks if a key is contained in the map.
      Specified by:
      containsKey in interface Map<K,V>
      Parameters:
      key - The key.
      Returns:
      True, if the key is contained, even if the associated value is null, false otherwise.
    • containsValue

      public boolean containsValue(Object value)
      Checks if a value is contained in the map.
      Specified by:
      containsValue in interface Map<K,V>
      Parameters:
      value - The value.
      Returns:
      True, if the map contains the value.
    • put

      public V put(K key, V value)
      Adds a new key-value pair to the map, if the key already exists, the associated value is overwritten.
      Specified by:
      put in interface Map<K,V>
      Parameters:
      key - The key.
      value - The value.
      Returns:
      The old value, if it exists, null otherwise.
    • putAll

      public void putAll(Map<? extends K,? extends V> m)
      Adds all entries of another map to this one.
      Specified by:
      putAll in interface Map<K,V>
      Parameters:
      m - The other map.
    • remove

      public V remove(Object key)
      Removes key from the map.
      Specified by:
      remove in interface Map<K,V>
      Parameters:
      key - The key.
      Returns:
      The old value associates with the key or null if there was no mapping..
    • entrySet

      public Set<Map.Entry<K,V>> entrySet()
      Returns the entry set for the Map. WARNING: Do not use without holding at least a read lock while getting and using the returned set.
      Specified by:
      entrySet in interface Map<K,V>
      Returns:
      Returns the entry set of the map.
    • keySet

      public Set<K> keySet()
      Returns the key set for the Map. WARNING: This will load all keys into memory, please consider using entrySet() instead. WARNING: Do not use without holding at least a read lock while getting and using the returned set.
      Specified by:
      keySet in interface Map<K,V>
      Returns:
      Returns the key set of the map.
    • values

      public Collection<V> values()
      Returns the values of the Map. WARNING: This will load all values into memory, please consider using entrySet() instead. WARNING: Do not use without holding at least a read lock while getting and using the returned collection.
      Specified by:
      values in interface Map<K,V>
      Returns:
      Returns the key set of the map.
    • size

      public int size()
      Returns the number of key-values in this map.
      Specified by:
      size in interface Map<K,V>
      Returns:
      the number of key-values in this map
    • isEmpty

      public boolean isEmpty()
      Returns true, if this map contains no key-value mappings.
      Specified by:
      isEmpty in interface Map<K,V>
      Returns:
      True, if this map contains no key-value mappings
    • clear

      public void clear()
      Clears the map, deleting all elements.
      Specified by:
      clear in interface Map<K,V>
    • readTransaction

      public void readTransaction(SharedPersistentMap.IORunnable transaction)
      Performs a read transaction on the map. Engages read lock and automatically releases on success or exception.
      Parameters:
      transaction - Transaction to perform.
    • readTransaction

      public <R> R readTransaction(SharedPersistentMap.IOSupplier<R> transaction)
      Performs a read transaction on the map. Engages read lock and automatically releases on success or exception.
      Parameters:
      transaction - Transaction to perform.
      Returns:
      Return value of lambda expression.
    • readTransaction

      public <R> R readTransaction(SharedPersistentMap.IOFunction<SharedPersistentMap<K,V>,R> transaction)
      Performs a read transaction on the map. Engages read lock and automatically releases on success or exception.
      Parameters:
      transaction - Transaction to perform.
      Returns:
      Return value of lambda expression.
    • writeTransaction

      public void writeTransaction(SharedPersistentMap.IORunnable transaction)
      Performs a write transaction on the map. Engages write lock and automatically releases on success or exception.
      Parameters:
      transaction - Transaction to perform.
    • writeTransaction

      public <R> R writeTransaction(SharedPersistentMap.IOSupplier<R> transaction)
      Performs a write transaction on the map. Engages write lock and automatically releases on success or exception.
      Parameters:
      transaction - Transaction to perform.
      Returns:
      Return value of lambda expression.
    • writeTransaction

      public <R> R writeTransaction(SharedPersistentMap.IOFunction<SharedPersistentMap<K,V>,R> transaction)
      Performs a write transaction on the map. Engages write lock and automatically releases on success or exception.
      Parameters:
      transaction - Transaction to perform.
      Returns:
      Return value of lambda expression.
    • readLock

      public jadex.common.IAutoLock readLock() throws IOException
      Engages the read lock over the whole file.
      Returns:
      The lock, close to release.
      Throws:
      IOException - Thrown on IO issues.
    • writeLock

      public jadex.common.IAutoLock writeLock() throws IOException
      Engages the write lock over the whole file.
      Returns:
      The lock, close to release.
      Throws:
      IOException - Thrown on IO issues.
    • verifyMapState

      protected void verifyMapState()
      Verifies the map file and memory mapping state.
    • performMaintenanceIfRequired

      protected void performMaintenanceIfRequired() throws IOException
      Performs maintenance tasks if required. Note: Write lock must be held before method is called.
      Throws:
      IOException - Thrown on IO issues.
    • getIndexSizeForMapSize

      protected long getIndexSizeForMapSize(long mapsize)
      Returns the appropriate index size for the size of the map.
      Parameters:
      mapsize - Map size.
      indexsize - Current index size.
      Returns:
      Correct index size.
    • readIndexSize

      protected long readIndexSize()
      Reads the current index size in long words. NOTE: Called MUST lock the file with, at minimum, a read lock.
      Returns:
      Index size or -1, if the file is invalid or empty.
    • initializeMap

      protected void initializeMap() throws IOException
      Initializes the map file as an empty map.
      Throws:
      IOException - Thrown on IO issues.
    • initializeMap

      protected void initializeMap(long isize) throws IOException
      Initializes the map file as an empty map with a specified index size.
      Parameters:
      isize - Index size.
      Throws:
      IOException - Thrown on IO issues.
    • findKVPair

      protected SharedPersistentMap<K,V>.KVNode findKVPair(Object key) throws IOException
      Finds the key-value pair that contains the given key.
      Parameters:
      key - The key to look up.
      Returns:
      The KVNode containing the key or null if not found.
      Throws:
      IOException - Thrown on IO issues.
    • appendKVPair

      protected SharedPersistentMap<K,V>.KVNode appendKVPair(K key) throws IOException
      Appends a new KV node. Note: This only adds the node, not the key. There is also no check if the key is already in the map.
      Parameters:
      key - The key that needs the KVNode.
      Returns:
      KVNode object for the newly created node.
      Throws:
      IOException - Thrown on IO issues.
    • absoluteToHeap

      protected long absoluteToHeap(long absolute)
      Converts an absolute position to a heap position
      Parameters:
      absolute - Absolute position in the file.
      Returns:
      Position in the heap.
    • heapToAbsolute

      protected long heapToAbsolute(long heap)
      Converts an absolute position to a heap position
      Parameters:
      heap - Position in the heap.
      Returns:
      Absolute position in the file.
    • getIndexPosition

      public long getIndexPosition(int hash)
    • getIndexReference

      protected long getIndexReference(long indexposition) throws IOException
      Reads the positional reference at a given position in the hashtable index.
      Parameters:
      indexposition - Position in the index.
      Returns:
      KVNode reference at the given index position.
      Throws:
      IOException
    • getHeaderInt

      protected int getHeaderInt(int headerpos)
      Reads an int-value from the header.
      Parameters:
      headerpos - Position in the header.
      Returns:
      Value at the position.
    • getHeaderLong

      protected long getHeaderLong(int headerpos)
      Reads a long-value from the header.
      Parameters:
      headerpos - Position in the header.
      Returns:
      Value at the position.
    • putHeaderLong

      protected void putHeaderLong(int headerpos, long value)
      Writes a long-value from the header.
      Parameters:
      headerpos - Position in the header.
      value - Value to write at the position.
    • addGarbage

      protected void addGarbage(long garbage) throws IOException
      Adds an amount to the garbage counter.
      Parameters:
      garbage - Amount of garbage to add to the garbage counter.
      Throws:
      IOException - Thrown on IO issues.
    • addSize

      protected void addSize(long size) throws IOException
      Adds an amount to the map item size.
      Parameters:
      size - Amount of garbage to added to the map size.
      Throws:
      IOException - Thrown on IO issues.
    • getNextAlignedPosition

      protected long getNextAlignedPosition() throws IOException
      Returns the next 8byte-aligned starting position after the current end of file.
      Returns:
      The next aligned starting position after the current end of file.
      Throws:
      IOException
    • fill

      protected ByteBuffer fill(ByteBuffer b, byte fillval)
      Fills a ByteBuffer with a byte value.
      Parameters:
      b - ByteBuffer object.
      fillval - Fill value.
    • getMappedBuffer

      protected MappedByteBuffer getMappedBuffer(long absposition, long lsize, boolean append) throws IOException
      Throws:
      IOException
    • getIndexMap

      protected MappedByteBuffer getIndexMap() throws IOException
      Throws:
      IOException
    • nextPowerOfTwo

      public final int nextPowerOfTwo(int x)
      Calculates the next power of two that's equal or larger than x. Note: Does not handle zero or negative values.
      Parameters:
      x - Any positive int value
      Returns:
      Next power of two.
    • main

      public static void main(String[] args)