welcome
=====
This is the code package for CORE's MSST13 version. This package is ONLY
tested with Ubuntu 12.04. Have fun.
Install
=====
The CORE in this package relies on hadoop-0.22.0 and its HDFS-RAID, so
in order to install CORE, we need to embed CORE's source code into
hadoop-0.22.0 and recompile it. Then we can set up a CORE cluster following a
similar way as we set up a hadoop cluster.
You should plan where to locate the namenode and where to locate the datanode.
Then do the following in one of the node.
1. install g++, ant, java, set up $HADOOP_HOME(where you will place
your hadoop), $JAVA_HOME(where is the sdk).
2. download
[hadoop-0.22.0](http://archive.apache.org/dist/hadoop/core/hadoop-0.22.0/hadoop-0.22.0.tar.gz)
3. Untar hadoop-0.22.0.tar.gz, mv to $HADOOP_HOME
4. Run install.sh. This script will integrate CORE into hadoop-0.22.0 with some default configurations.
bash install.sh
5. In $HADOOP_HOME/conf/masters, enter the hostname or host ip of your namenode.
(don't forget to remove the default one if you don't need it)
6. In $HADOOP_HOME/conf/slaves, enter the hostname or host ip of your datanode.
(don't forget to remove the default one if you don't need it)
7. In $HADOOP_HOME/conf/core-site.xml, config the hadoop tmp directory
(where you want to put the hadoop file in your machine, better to
use absolute path) and
fs.default.name.
hadoop.tmp.dir
*put your path here(absolute path)*
fs.default.name
hdfs://*your namenode hostname or ip*:54310
8. In $HADOOP_HOME/conf/hdfs-site.xml, config the stripeLength(k) and
parityLength(m). (Where the property name of parity length dependent
on your chosen code)
hdfs.raid.stripeLength
*your k*
hdfs.raidjrs.paritylength
*your m for jrs code*
hdfs.raidia.paritylength
*your m for ia code*
9. In $HADOOP_HOME/conf/hdfs-site.xml, config the *block size* and *packet
size*(strip size). *packet size* should be a multiple of *parity length*(m) and *block
size* should be a multiple of *packet size*.
dfs.block.size
*your block size*
hdfs.raid.packet.size
*your packet size*
10. In $HADOOP_HOME/conf/hdfs-site.xml, config the path of
raid.xml (Normally it should be in $HADOOP_HOME/conf/raid.xml,
better to use absolute path).
raid.config.file
*your raid.xml path(Absolute Path)*
This is needed by the RaidNode
11. In $HADOOP_HOME/conf/raid.xml, config the type of code you want to
use. (ia/jrs/iao/jrso, o stands for original architecture)
*your code type*
12. In $HADOOP_HOME/conf/raid.xml, config the source file path.
13. In $HADOOP_HOME/conf/hadoop_env.sh, set the JAVA_HOME
export JAVA_HOME=*your java home(Absolute Path)*
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
After finish the previous node installation and configuration, copy the
$HADOOP_HOME folder and spread the folder into the *namenode* and *ALL*
*datanodes*. Make sure each nodes can be public key accessible by each
other. Also make sure that c++, java, ant and $HADOOP_HOME are well set
for ALL nodes. If you have serveral different type of nodes (for
example, different OS), you may need to repeat the previous steps in
different types of nodes.
Add $HADOOP_HOME/bin to PATH
export PATH=$PATH:$HADOOP_HOME/bin
Format hdfs
hadoop namenode -format
Run hdfs
start-hdfs.sh
Run Raidnode
start-raidnode.sh
Configuration
=====
Configurable options in hadoop-0.22.0 (include hdfs-raid in 0.22.0) are
inherited. Please refer to
[hdfs](http://hadoop.apache.org/docs/stable/cluster_setup.html) and
[hdfs-raid](http://wiki.apache.org/hadoop/HDFS-RAID).
Additional configurable options for CORE only
For hdfs-site.xml
Config the number of encoder working thread
hdfs.raid.encoder.threadnum
2
Config the number of decoder working thread
hdfs.raid.decoder.threadnum
1
Config the parity length of jerasure's rs code
hdfs.raidjrs.paritylength
8
Config the parity length of IA code
hdfs.raidia.paritylength
8
Config the strip size(make sure that the strip size must be a multiple
of parity length(m))
hdfs.raid.packet.size
4194304
A new block placement policy named Controlable Policy. It is a ad hoc
policy for experment only and by no means to be used in practical. It
can help to make sure that blocks from the same stripe will be located
in different nodes. If you choose to use this policy, you need to make
sure that *hdfs.raidia.paritylength* is set and there is *NO REBOOT*
between data import and migration.
dfs.block.replicator.classname
org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyControlable
For raid.xml
Choose code type(options: jrs, ia)
ia
Running CORE
=====
Similar to HDFS-RAID.
To run corresponding experiment, you need to determine some
configurations.
1. Code type.
2. Parameter *k* and *m*.
3. Block size and strip size.
4. Config your cluster(namenode and datanode).
5. Customize raid source file path.
6. Choose block placement policy.
For all experiments, you can check the log files in *$HADOOP_HOME/logs/*
for the updated status.
Migration
1. Import data into raid source file folder.
2. $HADOOP_HOME/bin/start-raidnode.sh
Recovery
1. After migration, you can artificially remove few datanodes. (you can
either disconnect the node, kill datanode process or remove
corresponding data block.)
2. Wait for recovery to finish. (Since block fixer has a checking
window, if you want to make the recovery start earlier, you can
reboot hdfs after you remove datanodes.)
Degraded Read.
1. After migration, you can artificially remove few datanodes. (you can
either disconnect the node, kill datanode process or remove
corresponding data block.)
2. Try to read data. (either by mapreduce or shell command)
3. You can stop raidnode to avoid recovery during degraded read.
Developers' Guide
=====
*./native* hosts the native coding relative code including the Jerasure
library.
*./java* hosts some modified hadoop java code.
To add a new coding scheme
Suppose you want to add a new coding scheme, called newCoding.
Here are the steps you need to take:
1. Implement C++ version of the coding scheme.
You need to implement a class newCoding which extends the
base class Coding. You need to implement the following virtual
functions:
generate_encoding_matrix();
Generate the encoding equations.
encode2();
Encode the original data to encoded data.
test_validity();
Test whether a failure pattern is good or not.
set_f2();
Generate decoding equations for data reconstruction.
encode_offline_recovery2();
Encoding function called by the surviving nodes.
reconstruct_lost_date2();
Decoding function called by the relayer
2. Update the native code.
3. Modify the java source code.