| m u s h i n |mu-shin.ca

 
June 5, 2013

HBase + Node.js: Doing it the Thrift Way

Category: Code — Tags: , — Marc Trudel-Bélisle @ 8:34 AM

Note: I am by no mean an expert with using Thrift. Or HBase, for that matter. But so far, there seems to be next to no support for getting started with HBase in Nodeland.

There is a great module, https://github.com/wdavidw/node-hbase, which I have tried to revive (pull request standing: https://github.com/wdavidw/node-hbase/pull/13). But as I have discovered anyway, the HBase REST API seems to be very, very slow.

There is also the moribund https://npmjs.org/package/hbase-thrift; not supported, GitHub repo is dead (in the end, their code was very useful to me, although the generated Thrift files were of course outdated).

Hmmm.

I ended up spending quite a bit of time figuring it out. This will assume you have already HBase installed somewhere (I personally tested from my machine).

From here, here is what you need to get done.

0. Download HBase

If you are not running HBase locally, make sure to download a copy of your HBase version.

wget http://ftp.riken.jp/net/apache/hbase/hbase-0.94.7/hbase-0.94.7.tar.gz && tar -zxvf hbase-0.94.7.tar.gz && rm -rf hbase-0.94.7.tar.gz
ln -s hbase-0.94.7 hbase

1. Dowload Thrift

Thrift, by Facebook. Unfortunately, it seems like Avro is the future, but right now, it is not on HBase stable yet.

Anyway, download Thrift, compile:

wget http://ftp.riken.jp/net/apache/thrift/0.9.0/thrift-0.9.0.tar.gz && tar -zxvf thrift-0.9.0.tar.gz && rm thrift-0.9.0.tar.gz
cd thrift-0.9.0/
./configure && make
cd ..
ln -s thrift-0.9.0 thrift

2. Create your Node.js project

To get ourselves started, let’s create a new project, hbase, and get the Node.js Thrift module.

mkdir hbase && cd hbase
npm init # Or skip if you are just testing
npm install --save node-thrift

3. Generate your Thrift files

../thrift/compiler/cpp/thrift --gen js:node ../hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift
ls

This will take the HBase Thrift definition and convert it to Node.js compatible JavaScript under gen-nodejs. Feel free to rename this or move the produced file somewhere else; for the sake of this example, I won’t.

4. Write this example code

Put this in your index.js:

var thrift = require('thrift');

var HBase = require('./gen-nodejs/Hbase');
var HBaseTypes = require('./gen-nodejs/Hbase_types');

var connection = thrift.createConnection('localhost', 9090, { transport: thrift.TBufferedTransport });
var client = thrift.createClient(HBase, connection);

connection.on('connect', function () {
console.log('Connected');
client.getTableNames(function (err, data) {
console.log('Tables:', data);
connection.end();
});
});

If you already have tables in your HBase setup, you should see the following:

┌─[mt@Marcs-MacBook-Pro]─[~/Sources/hbase]
└──╼ node .
Connected
Tables: [ 't1', 't2' ]

Unfortunately, from there, the documentation is, well… thin. But the cool thing is, since it is Thrift-based, it is easy to go and look at a PHP or Python example, and then translate this into JavaScript (basically, finding the right data type and adding a callback(err, data) as a last argument).

Update: Well, must have been blind. Here is the Thrift documentation (just add a callback at the end of each call and transform Lists<> to arrays, and you should be pretty much good to go) http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/thrift/doc-files/Hbase.html#Fn_Hbase_atomicIncrement

1 Comment »

  1. Thanks, really cool !
    Node hbase thrift examples were really missing.
    What do you think to make a github repo ? if you dont, i thnik will :-)

    Comment by max — October 17, 2013 @ 1:04 PM

RSS feed for comments on this post. TrackBack URL

Leave your comment

Powered by WordPress