lang="en-US"> | m u s h i n | » HBase + Node.js: Doing it the Thrift Way
  • June 2013
  • March 2011
  • January 2011
  • December 2010
  • November 2010
  • July 2010
  • April 2010
  • February 2010
  • December 2009
  • November 2009
  • October 2009
  • July 2009
  • June 2009
  • May 2009
  • April 2009
  • March 2009
  • February 2009
  •  

    | m u s h i n |mu-shin.ca

     
    June 5, 2013

    HBase + Node.js: Doing it the Thrift Way

    Category: Code — Tags: , — Marc Trudel-Bélisle @ 8:34 AM

    Note: I am by no mean an expert with using Thrift. Or HBase, for that matter. But so far, there seems to be next to no support for getting started with HBase in Nodeland.

    There is a great module, https://github.com/wdavidw/node-hbase, which I have tried to revive (pull request standing: https://github.com/wdavidw/node-hbase/pull/13). But as I have discovered anyway, the HBase REST API seems to be very, very slow.

    There is also the moribund https://npmjs.org/package/hbase-thrift; not supported, GitHub repo is dead (in the end, their code was very useful to me, although the generated Thrift files were of course outdated).

    Hmmm.

    I ended up spending quite a bit of time figuring it out. This will assume you have already HBase installed somewhere (I personally tested from my machine).

    From here, here is what you need to get done.

    0. Download HBase

    If you are not running HBase locally, make sure to download a copy of your HBase version.

    wget http://ftp.riken.jp/net/apache/hbase/hbase-0.94.7/hbase-0.94.7.tar.gz && tar -zxvf hbase-0.94.7.tar.gz && rm -rf hbase-0.94.7.tar.gz
    ln -s hbase-0.94.7 hbase
    

    1. Dowload Thrift

    Thrift, by Facebook. Unfortunately, it seems like Avro is the future, but right now, it is not on HBase stable yet.

    Anyway, download Thrift, compile:

    wget http://ftp.riken.jp/net/apache/thrift/0.9.0/thrift-0.9.0.tar.gz && tar -zxvf thrift-0.9.0.tar.gz && rm thrift-0.9.0.tar.gz
    cd thrift-0.9.0/
    ./configure && make
    cd ..
    ln -s thrift-0.9.0 thrift
    

    2. Create your Node.js project

    To get ourselves started, let’s create a new project, hbase, and get the Node.js Thrift module.

    mkdir hbase && cd hbase
    npm init # Or skip if you are just testing
    npm install --save node-thrift
    

    3. Generate your Thrift files

    ../thrift/compiler/cpp/thrift --gen js:node ../hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift
    ls
    

    This will take the HBase Thrift definition and convert it to Node.js compatible JavaScript under gen-nodejs. Feel free to rename this or move the produced file somewhere else; for the sake of this example, I won’t.

    4. Write this example code

    Put this in your index.js:

    var thrift = require('thrift');
    
    var HBase = require('./gen-nodejs/Hbase');
    var HBaseTypes = require('./gen-nodejs/Hbase_types');
    
    var connection = thrift.createConnection('localhost', 9090, { transport: thrift.TBufferedTransport });
    var client = thrift.createClient(HBase, connection);
    
    connection.on('connect', function () {
    console.log('Connected');
    client.getTableNames(function (err, data) {
    console.log('Tables:', data);
    connection.end();
    });
    });
    

    If you already have tables in your HBase setup, you should see the following:

    ┌─[mt@Marcs-MacBook-Pro]─[~/Sources/hbase]
    └──╼ node .
    Connected
    Tables: [ 't1', 't2' ]
    

    Unfortunately, from there, the documentation is, well… thin. But the cool thing is, since it is Thrift-based, it is easy to go and look at a PHP or Python example, and then translate this into JavaScript (basically, finding the right data type and adding a callback(err, data) as a last argument).

    Update: Well, must have been blind. Here is the Thrift documentation (just add a callback at the end of each call and transform Lists<> to arrays, and you should be pretty much good to go) http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/thrift/doc-files/Hbase.html#Fn_Hbase_atomicIncrement

    1 Comment »

    1. Thanks, really cool !
      Node hbase thrift examples were really missing.
      What do you think to make a github repo ? if you dont, i thnik will :-)

      Comment by max — October 17, 2013 @ 1:04 PM

    RSS feed for comments on this post. TrackBack URL

    Leave your comment

    Powered by WordPress