Index64
| File:Index64-logo.png | |
| Developer(s) | Index64 development team |
|---|---|
| Stable release | 3.0
/ May 27, 2014 |
| Written in | C++ |
| Engine | |
| Operating system | Windows, Linux |
| Available in | English |
| Type | Library |
| License | "multithread license" (PDF). "single thread license" (PDF). |
| Website | index64 |
Search Index64 on Amazon.
Index64 is a concurrent key-value store for Intel architectures. It indexes key-value pairs on single and multi core environments using lock free algorithms and allowing simultaneous reads-writes. By 2014, Index64 is the only concurrent key-value store available for download.[citation needed] The underlying technology is based on specific Intel 64 CPU instructions.
History
In 2011, a first version was developed in order to respond to specific needs of a data-intensive program running localization features on a large scale. This localization software developed under iOS was finally never launched.
In 2012, version 2.0 was released and distributed for free under the name "Data-Parallel". This version was enhanced with on-the-fly data-partitioning capabilities. Although well suited for massive batch operations, this technology was not versatile enough to meet most of the indexing needs.
In 2013, a new multithread approach has been implemented. The data structure of the index was altered to accept concurrent reads-writes. The key-value store has been renamed "Index64 version 3.0".
Features
Some features of the 3.0 Index64 key-value store are:[1]
- Concurrent accesses. An application can update a directory while reading this directory to serve requests.
- Scalability. Any index size is possible.
- Any-core dynamic resizing. The number of cores used to perform bulk operations can be changed from one call to another; this is a way to adapt programs to varying loads.
- Memory management. Since allocation and deallocation are managed from within the key-value store, the application doesn't need to perform regular memory cleanup that could alter performance.
- Scan methods. Index64 allows forward and backward scanning among the sorted data.
- Non unique keys and composite keys. Index64 handles keys with multiple values and keys made up of several concatenated data.
Examples
Here is a trivial example demonstrating basic string indexing:
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <index64mt.h>
class I64MtString index;
int main(int, char**, char**) {
// Set maximum strings length in char including trailing 0
index.setBytes(6);
// Index some null-terminated strings
const char* data[3] = { "zero", "one", "two", };
index.insert(data[0], &data[0]);
index.insert(data[1], &data[1]);
index.insert(data[2], &data[2]);
// Retrieve a value
tValue value;
if (index.select("one", value) == eReturnCodeOk)
printf("Key one has value %p", value);
else
printf("Key one is not indexed.");
}
In the following example the integrated thread pool is used to make concurrent writes:
#include <index64mt.h>
class I64MtSigned32 index;
struct sData { int K; int Col1; int Col2; } Table[1000000];
// Build the index assuming that the table is loaded
void BuildIndex() {
// Start the integrated thread pool
index.bulkAllocateThreads(6);
// Engage the 6 threads in indexing the table
index.bulkInsert(0, 6, 1000000, &Table[0].K, sizeof(struct sData));
}
About concurrent indexing
Increasing the number of working threads increases the number of operations executed at the same time; however, some factors may affect the speed resulting from this parallelization.
Memory bandwidth
With a memory bandwidth of 10 GB/s for writings, the performance peak of Index64 is reached with 8 threads.[2] With more threads the performance stabilizes since the memory bandwidth is already fully used.
Tasks interdependency
Parts of program that access shared data have to be done sequentially to ensure the integrity of the shared data. These sequential parts define the critical path of performance that corresponds to the optimum execution time that could result from parallelization. This performance cap is a direct application of Amdahl's law.
See also
References
- ↑ "Index64 3.0 reference manual", Index64, May 2014
- ↑ "Index64 3.0 benchmark", Index64, May 2014
This article "Index64" is from Wikipedia. The list of its authors can be seen in its historical. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.
