VeloxDFS is a decentralized distributed file system based on ChordDHT and HDFS.
This distributed file system serves as the foundation and essential component of the Velox Big Data Framework (VBDF), however, it can be used independently of the other components such as VeloxMR (MapReduce Framework).
Key features of current VeloxDFS implementation includes:
Related projects
| Project | Description | URL |
|---|---|---|
| VeloxMR | experimental MapReduce engine based on VeloxDFS | https://github.com/DICL/VeloxMR |
| eclipsed | deployment/debugging helper script writen in RUBY | https://github.com/DICL/eclipsed |
| velox-hadoop | VeloxDFS JAVA library for Hadoop | https://github.com/DICL/velox-hadoop |
| velox-deploy-ansible | ansible playbook to automatize velox/hadoop installation | https://github.com/DICL/velox-deploy-ansible |
| hadoop-etc | Hadoop configuration files to use VeloxDFS | https://github.com/vicentebolea/hadoop-etc |
| velox-report | Project to benchmark and log VeloxDFS performance | https://github.com/vicentebolea/velox-report |
To deploy the system please refer to velox command and its following options:
$ veloxd up|down|restart|status
The command line API has the following options:
$ veloxdfs put|get|cat|ls|rm|format|show
The C++ API can be found at vdfs.hh and DFS.hh files, as for the JAVA API, it can be found at src/java directory.
$ mkdir -p local_eclipse/{tmp,sandbox} # Create a sandbox directories
$ cd local_eclipse # enter in the directory
$ git clone git@github.com:DICL/VeloxDFS.git # Clone the project from github
$ cd VeloxDFS
$ sh autogen.sh # Generate configure script
$ cd ../tmp # Go to building folder
$ sh ../VeloxDFS/configure --prefix=`pwd`/../sandbox # Check requirements and generate the Makefile
# If you get a boost error go the FAQ section of the README
### This last command will be needed whenever you want to recompile the source
$ make [-j#] install # Compile & install add -j flag to speed up
Now edit in your ~/.bashrc or ~/.profile:
export PATH="/home/*..PATH/To/eclipse/..sandbox/bin":$PATH export LIBRARY_PATH="/home/*..PATH/To/eclipse/..sandbox/lib" export C_INCLUDE_PATH="/home/*..PATH/To/eclipse/..sandbox/include"
"log" : {
"type" : "LOG_LOCAL6"
"name" : "ECLIPSE"
"mask" : "DEBUG"
},
"cache" : {
"numbin" : 100,
"size" : 200000,
"concurrency" : 1
},
"filesystem" : {
"block" : 137438953,
"buffer" : 512,
"replica" : 1
}
configure stops with errors related to boost library.Answer : It probably means that you do not have boost library installed in the default location, in such case you should specify the boost library location.
sh ../VeloxDFS/configure --prefix ~/sandbox --with-boost=/usr/local --with-boost-libdir=/usr/local/lib
In this example we assume that the boost headers are in /usr/local/include while the library files are inside /usr/local/lib.
1.8.6