HPC vs Big Data - Part 3

 In several prior posts comparing Big Data to HPC, I asked a question about how HPC and Big Data were different and why it was difficult for HPC people to "get" Big Data and vice versa.

I had another idea of how to present it.

Earlier I stated that: 

HPC: Computation Time >> IO Time

Big Data: IO Time >> Computation Time

However, this by itself can sometimes be confusing.  Some algorithms are programmed in both the HPC world (e.g. MPI) and the Big Data world (e.g. Hadoop/Spark).  So how can this dichotomy be represented?

The reason the above definition can be confusing is because it suggests things are black and white, which is of course not true.  In reality, the trade off between I/O time and Computation time is along a sliding scale.  To represent this, lets look at this simple (terribly drawn) diagram.




In it, we have increasing computation on the X axis going right and increasing I/O on the Y axis going up.  Based on my prior definition, we can consider HPC jobs to be those along the bottom right, where computation is much greater than the I/O.  Big Data is the jobs in the upper left of this diagram, where I/O time is much greater than computation time.

Naturally, if your users and/or code is significantly in one area vs the other, you'll have different prioritization on hardware, software, etc. than those in the other area.  Thus leading to the very different universes of cluster computing.

What (hopefully) clarifies some of the confusion on HPC vs Big Data is the fact that at some point in the middle these two worlds sort of meet.  There are Data + Computation combinations where it's nebulous which direction is superior.  A user might in fact be indifferent towards which type of hardware/software situation is better for them.  For users in the middle, it is far more likely they will simply gravitate towards the system they are more familiar with (e.g. if you know Java, use Hadoop/Spark, if you know C, use MPI) or simply have access to (if you are familiar with AWS, just use it, if you have access to an MPI cluster, just use it).

I hope this clarifies the confusion between implementation that may be done in both worlds.

Good vs. Bad Movie Fight Scenes

I recently came upon the following video:




It talks about "rhythm" and sequencing in fight scenes.  There's a great quote around 4:25 that speaks of "the audience doesn't know the rhythm's there until it's not there."

I started to pay attention to this in some martial arts films and you can't help but notice it more once you are looking for it.  Instead of "rhythm" what I like to think of it is "sequences of action" in a single film take.  When a performer isn't good or the director is trying to save money on takes, very rarely will "sequences of action" be done in a single take.  In other words, will only a single punch or kick happen in a video take before they cut away to a different angle?  Or will multiple kicks/punches occur within a single take in a sequence of a choreographed fight?

Lets start by looking at a bad fight scene.  This scene in The Medallion, a pretty terrible Jackie Chan film, with Claire Forlani in a pretty awful fight.




Very rarely is more than a single punch or kick ever done in a take.  One punch or kick is done, then they director cuts to a different angle.  Only at 41 seconds into the video do they even bother to sequence about 3 kicks together in a take and at about 46 seconds there are two kicks in the take.

In contrast, lets take a look at a fight that is perhaps a gold standard of excellence, the Michelle Yeoh vs Zhang Ziyi fight in Crouching Tiger Hidden Dragon.




This fight is wonderful.  Throughout the fight you see sequences of multiple thrusts & blocks of a sword in a single take.  The overhead sequences at 1:02 in the video is particularly wonderful.  I counted about 15 actions (attacks/parrys/blocks) sequenced together in a single take as the actresses move across the floor.

It's interesting to look at Jackie Chan fights that were directed in Hong Kong vs. America.  In the first fight scene in Rush Hour, we get this very meh fight sequence.



Again, you can see that most of the takes contain a single punch or action.  Only at about 1:05 do they bother to sequence about 2 attacks in a single shot and a few multiple actions in a shot around 1:20.

In contrast, I think of this incredible fight from First Strike



There are many sequences of 3-4 actions in a single take which give the fight a much better rhythm.  The chair sequence at 1:06 is particularly wonderful.  Does Jackie Chan really need to jump over a chair, duck a chair thrown, and catch one in a single take?  No.  But it adds something special to the scene.

Likewise with the ladder sequence.  You could probably forgive Jackie Chan for only doing a single attack or block with a ladder given its hefty weight.  However, at 3:38 he actually launches an attack with the ladder opening up and bothers to block two further attacks in a single take.  That's the kind of thing that makes these fights far more special.

While looking at martial arts fights on YouTube, I thought I'd bring up one particularly awesome scene.  It's the elevator fight scene from Ip Man 3.




Once they exit the elevator, there are sequences of 5+ punches/blocks/kicks in a single take.  The overhead shot at 1:54 is particularly amazing.  I count 15 actions that take place as the actors move down the stairs, around the hallway, and end with one actor getting kicked down the next batch of stairs.  All in a single take.

Finally, just to show that it doesn't take trained martial artists in Hong Kong to do good fights, lets take a look at this Neo vs. Agent Smith fight in the Matrix.



To my knowledge, neither actor is actually trained in martial arts.  But with some good choreography and good editing and willingness from the director, you can sequence many punches/blocks into a single take and get a much better fight scene out of it.

Beginners Guide to Hbase with Python & Thrift

I recently wanted to play around with Hbase and Python, which subsequently lead me to use Thrift.

I know there are tons of guides on the web, but a number I found were outdated or based on downloaded versions of things instead of packaged distro versions.   I eventually had to piece together information from several sources.  So I thought I'd put it altogether on this page for anyone looking for simple "cut and paste" instructions to begin with.  I'm not going to go through the basics of Hbase and Thrift, as there are many guides out there, but I'll give updated instructions based on the following versions I used.

Hortonworks 2.2.6.0-2800
Hbase 0.98.4
Redhat 6.7

As an aside, there may be newer interfaces (such as happybase) that are now more popular.  I may look into those later, but these are just my notes on this particular subject.

As another aside, my Hbase has already been populated with data, so there's no need to create/insert data, so I'm skipping that.

The two primary Hbase + Python + Thrift sources I used for this were:

Using Facebook’s Thrift with Python and HBase (posted July 2008)

and

How-to: Use the HBase Thrift Interface, Part 1 and Part 2 and Part 3 (posted September 2013)

So lets start.

First up, I downloaded thrift 0.9.3 and did the normal configure and make, but this didn't compile for me.

src/thrift/qt/moc_TQTcpServer.cpp:14:2: error: #error "This file was generated using the moc from 4.8.1. It"
src/thrift/qt/moc_TQTcpServer.cpp:15:2: error: #error "cannot be used with the include files from this version of Qt."
src/thrift/qt/moc_TQTcpServer.cpp:16:2: error: #error "(The moc has changed too much.)"

this was also the case with thrift 0.9.2, 0.9.1, and 0.9.0.

I went to thrift 0.8.0 and hit other build errors.  These were maybe solvable, but being lazy I just downloaded 0.7.0 to try it, and it compiled fine.  So I ended up using thrift 0.7.0.

Since this compiled, I need to install it somewhere.  I'm going to install into a non-privileged directory, so set all of these prefixes appropriately when you configure and make install.  If you're root and you can install anywhere, you can probably ignore all of this.  Adjust appropriately if you use bash instead of tcsh.

setenv PY_PREFIX /yourprefix/thriftpy/
setenv JAVA_PREFIX /yourprefix/thriftpy/
setenv RUBY_PREFIX /yourprefix/thriftpy/
setenv PHP_PREFIX /yourprefix/thriftpy/
setenv PHP_CONFIG_PREFIX /yourprefix/thriftpy/
setenv PERL_PREFIX /yourprefix/thriftpy/
./configure --prefix=/yourprefix/thriftpy --exec-prefix=/yourprefix/thriftpy
make install

After that, you should hopefully have thrift installed into your appropriate path and /yourprefix/thriftpy/bin/thrift should be available to run.

Now run thrift to generate Python files for Hbase

> /yourprefix/thriftpy/bin/thrift --gen py /usr/hdp/current/hbase-client/include/thrift/hbase1.thrift

Now if you're wondering why hbase1.thrift instead of hbase2.thrift, it's because there is now a newer thrift interface available compared to the original.  I use hbase1.thrift just to begin with.


Now there should be a "gen-py" sub-directory where you ran this.

Now we need to start the thrift server.  With HDP you can do:

/usr/bin/hbase thrift start

I started this in another window so I can control+C is later on.

Ok, onto code.  I started with code sort of from both the sites above and put together:

#!/usr/bin/env python                                                                                                                                 
import sys

sys.path.append('../gen-py/')
sys.path.append('../thriftpy/lib64/python2.6/site-packages/')

from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol

from hbase import Hbase

# Make socket                                                                                                                                         
transport = TSocket.TSocket('localhost', 9090)

# Buffering is critical. Raw sockets are very slow                                                                                                    
transport = TTransport.TBufferedTransport(transport)

# Wrap in a protocol                                                                                                                                  
protocol = TBinaryProtocol.TBinaryProtocol(transport)

client = Hbase.Client(protocol)

transport.open()

tablenames = client.getTableNames()

print "table names are: " + ",".join(tablenames) 

You'll notice a few sys.path.append calls to add paths to my local dirs with libraries in them.  You may need to adjust accordingly for your installed things.

Hopefully, you should just be able to run this and there will be no problems (unless you've never created tables in Hbase, which you should create some).

Ok, now to read some data.  Again, pasting together some code from the above resources, this will get a single row of data.

rows = client.getRow('mytable', 'myrowkey')
 
for row in rows:
    message = row.columns.get('mycolumnfamily:mycolumnname').value
    print "message = " + message
    rowKey = row.row
    print "rowkey = " + rowKey

This didn't work for me and I got an error of

TypeError: getRow() takes exactly 4 arguments (3 given)

Hmmm.  Now, if you look in the file hbase1.thrift from before, you can see what the interface for getRow looks like.

  /**
   * Get all the data for the specified table and row at the latest
   * timestamp. Returns an empty list if the row does not exist.
   *
   * @return TRowResult containing the row and map of columns to TCells
   */
  list getRow(
    /** name of table */
    1:Text tableName,

    /** row key */
    2:Text row,

    /** Get attributes */
    3:map attributes
  ) throws (1:IOError io)

Hmmm, it seems there is a new attributes argument.  I couldn't figure out what this did by searching online.  I didn't want to dig into the code too much at this point so I just passed in None to the getRow call like so:

rows = client.getRow('mytable', 'myrowkey', None)
 
for row in rows:
    message = row.columns.get('mycolumnfamily:mycolumnname').value
    print "message = " + message
    rowKey = row.row
    print "rowkey = " + rowKey

And lucky for me it worked.

Now getting one row of data is boring, we actually want to do scans.  So I started with the following.

scan = Hbase.TScan(startRow="someStartPrefix", stopRow="someStopPrefix")
scannerId = client.scannerOpenWithScan(desiredtable, scan)

rowList = client.scannerGetList(scannerId, 5)

while rowList:
    for row in rowList:
        mydata = row.columns.get("mycolumnfamily:mycolumnname").value
        rowKey = row.row
        print "rowKey = " + rowKey + ", mydata = " + mydata
    rowList = client.scannerGetList(scannerId, 5)

Again, I hit

TypeError: scannerOpenWithScan() takes exactly 4 arguments (3 given)

Just like getRow, there is a similar attributes argument I don't know what to do with. So I add a None argument to scannerOpenWithScan like so.

scannerId = client.scannerOpenWithScan(desiredtable, scan, None)

And this works and I get results.

Now, getting data with Hbase with start & stop rows is boring.  It's far more interesting to do filters.  How can we pass filters in Python?  Again, looking at the hbase1.thrift again, I can see what arguments TScan can take.

/**
 * A Scan object is used to specify scanner parameters when opening a scanner.
 */
struct TScan {
  1:optional Text startRow,
  2:optional Text stopRow,
  3:optional i64 timestamp,
  4:optional list columns,
  5:optional i32 caching,
  6:optional Text filterString,
  7:optional i32 batchSize,
  8:optional bool sortColumns
}

Hmmm, this filterString argument looks interesting.  But what to fill it with?  After I some Googling, I figure out it can be filled with functions you can find in the Hbase thrift documentation.

So here's some examples.

scan = Hbase.TScan(filterString="RowFilter(>=, 'binary:FOO')")

this is functionally identical to

scan = Hbase.TScan(startRow="FOO")

You can AND/OR things together.  So for example:

scan = Hbase.TScan(filterString="(RowFilter(>=, 'binary:STARTPREFIX') AND RowFilter(<=, 'binary:ENDPREFIX')) AND (RowFilter(=, 'substring:FOO') OR RowFilter(=, 'substring:BAR'))

Would find rows with the substring FOO or BAR within a range of STARTPREFIX and ENDPREFIX.

Well, that's as far as I've gotten.  There were a few gotcha points, so I hope that this helps somebody out there.

Dinner @ Uncle Yu's in Livermore, CA

Uncle Yu's is a nicer American style Chinese restaurant in Livermore, CA.  What do I mean by this?  You'll see white table cloths to go along with your kung pao chicken.  The restaurant advertises their attempt to purchase higher quality ingredients for all their dishes, which does come out when you taste them.  The restaurant also does wine pairings, although I've personally never partaken in it.

As far as American style Chinese restaurants go it's pretty good.  You're not going to find much here that's authentic Chinese except for maybe the peking duck.  It's a bit pricier than most, lunch combos are $12-$14 instead of the typical $7-$8 I get at other places.  Dinner entrees are priced up too.

I've been here a number of times for lunch and dinner over the years.  It's outside of the Michelin star, super hyped, or higher end fine dining type of restaurants that I normally blog about.  However, we were here for a friend's birthday and decided to try the 5 course tasting menu.  So I thought I'd write up about it for anyone who may be thinking of getting it and got here to my blog via Google trying to find out about it.

1) salmon puff w/ avocado spread

First up was these two salmon puffs.  Pretty good, although I wish there was a bit more avocado sauce.

2) "happy soup" w/ scallop, shrimp, lobster & fried bao

Next up was this "happy soup", which was a bit sour and spicy.  I didn't expect to like this when reading the description, but it was quite good.   I think tom yum soup from Thai restaurants came to mind, which I do not like.  However, this flavor was much better.  It came with some fried bao to dip into the soup.  I actually don't think I've ever had these bao fried before, only steamed before.  It being fried worked out much better for dipping.  Steamed bao probably would have fallen apart for dipping.

3) tea duck

Next up was tea duck wrapped in pancakes with (what I believe was) some hoisin sauce on the side.  The tea duck was pretty good and not too dry.  A solid classic dish.

4) beef tenderloin

Next up was this stir fried beef tenderloin with mushrooms.  Also pretty good.

5) chocolate cake

Finally some chocolate cake at the end.  At first I thought, "chocolate cake?", I thought they could have atleast given us a red-bean cake of some sort.  Oh well, this is an American style Chinese place.

Overall, a pretty good meal.  Obviously, I'm not going to try and compare this to other restaurants I've reviewed like Benu.  They are just not in the same category.  You aren't going to find any dishes here that are going to challenge your palate.

However, I probably would not recommend this for the price.  I can't recall the exact price, but I believe it was $60 a person before tax & tip.  As a comparison, the tasting at Al's Place is $60, the one at Kin Khao is $55, and the four course price fixe at Aster is $59.  The wine pairings at those restaurants are also cheaper.

So unless this is specifically what you're looking for, I would probably pass on the tasting menu.

Dinner @ Mourad in San Francisco, CA

I recently was able to hit up Mourad in San Francisco.  It's a relatively new restaurant from the folks that are behind Aziza.  It was given a debut of 3 out of 4 stars from the San Francisco Chronicle (very good for a restaurant that is just starting) and got a Michelin Star in 2016.

We were out to celebrate a friend's birthday and were intrigued by Mourad.  While it has a pricier a la carte menu (appetizers around $20, entrees around $35, and desserts at $12) and a tasting menu for $120, they also had something they call "la'acha".  It's a family style meal for the table.  They have several options such as lamb, short rib, and chicken.  The portions serve anywhere from two to six people.  Each also comes with several sides.

We decided to go for the duck family style option which they said served 2-3 people.  The family style meals take atleast 30 minutes to prepare, with the duck taking 45 minutes, so we got some appetizers first.  Since we got duck for our main entree, we got two seafood appetizers to share.

1) OCTOPUS, chickpea, artichoke, olive, merguez

First up was an octopus appetizer.  You can see the little bits of artichoke beneath the octopus, bits of orange on top, and the chickpea spread in the front.  The waiter then poured a liquid which I believe he said was a lamb based broth.  I don't recall seeing merguez (which according to Google is a sausage).  Perhaps that was part of the broth.  Overall, this dish was delicious.  Been awhile since I had octopus that good.

2) SALMON, kumquat, soubise, radish, lentil, z'hug

Admittedly, I'm not a big fan of salmon, however this salmon was delicious.  Smoked perfectly so it was still soft in the middle.  I had to look up what z'hug is, and it's apparently a spicy sauce originated from Yemen.  I assume that's what this was marinated in.  The lentils were toasted and on the bottom and gave this some texture differences.

3A) DUCK, date, smoke, mandarin

Finally, the main course.  When this first came out, my first thought was, "This is for 2-3 people?  It looks like a lot."  However, I think the portions were perfect for three people.  The duck was already de-boned and stuffed with sausage (I believe they said duck sausage) before it was roasted.  You can see some dates near the bottom in a date reduction sauce and some of the mandarin oranges on top.  It came with the following sauces.  Overall, I think the dish was really good.

I remember the one in the upper left being a harissa sauce, the upper right being tomato based, but can't remember the bottom one, although it's clearly an herb based sauce.  I though all three were good, but I think the duck and the date reduction sauce was delicious by itself.

The following were the four sides that came with the duck.  Although I took close ups of the dishes, I hope they do not come across looking larger than they really are.  They were all relatively small, in probably "cereal bowl" size dishes.  Hopefully the utensils in the picture give an idea of the size of the bowls.  While they are tiny, they were perfect for three people.

3B) KALE, olive, citrus

I'm actually not a big fan of kale, but this was really good.  The citrus I believe was lemon and I believe the chunks of yellow above were lemon rinds.

3C) HEIRLOOM BEANS, tomato, feta, za’atar crumble

The beans were underneath the feta and bread crumbs in the this picture.  I had to look up za'atar, and it's is apparently some type of herb.  I guess it was in the beans.

3D) POTATO, buttermilk, onion

I think the waiter said there was chicken skin on the potatoes, which I couldn't quite pick up.

3E) COUSCOUS, brown butter

The waiter said this was cooked with preserved lemon.  This was really good.

My friend commented on the sides with something like, "These are really plain ingredients, but they are just tastier here."  I completely concur.  For some reason they were all tastier.

With the main course out of the way, we now work our way to dessert.  We didn't really share these so I can only really comment on the first of these three.

4A) BLACK CURRANT, lavender sponge, coconut

Overall really good.  You should be able to see the black currant red sauce throughout this dish and the coconut ice cream at the bottom.  There are some pomegranate sprinkled throughout too.  I'm not a fan of coconut flavor, however the ice cream was really good.  Not too strong overall.  The lavendar cake was really good, although I think there was too much black currant sauce overall and may have overpowered things a bit.

4B) APPLE, chocolate, caramel, cardamom

This dessert looked really pretty and the bit of apple sorbet on the right I tried was really good.  Unsure of what all of the components on the left were, but I believe there was marshmallow, wafers, and compressed apples along with the chocolate branch.

4C) HONEY & ALMOND, beeswax, orange blossom

I tried a small bite of this but can't remember much, sorry.  There was some softer creamier stuff on the bottom, some crunchier stuff on top, and then what I assume was the honey-ish/beeswax-ish item on the top.

5) mignardise - pistachio cake w/ chocolate

Overall a really good meal.  I would highly recommend for a group outing as the family style meal is a great deal and makes the meal not quite that pricey per person.

Could Justice League Members be President of the United States?

Recently, I've been watching some animated films from the DC Universe.

With the 2016 Presidential election going on and the debate over Ted Cruz's candidacy given his Canadian birth, a funny thought popped in my head.  What members of the Justice League could be President of the United States?  In either their superhero or alter ego forms.

In order to be President you must meet three conditions:

• be 35 years of age
• be a resident in the US for 14 years
• be a natural born citizen

Obviously there are various continuities and origin stories as well as different people playing the roles in the below.  The below is imperfectly based on the various origin stories told through the animated DC Universe.  In addition, ages of characters and how long they've been living in the US are somewhat unknown.

Superman - Born on Krypton, definitely not a natural born citizen.  However, as Clark Kent, he probably qualifies.  He's been working for awhile, so we'll say he's old enough.

Batman - AFAICT Bruce Wayne qualifies, he's gotta be old enough after years of running Wayne Enterprises.

Wonder Woman/Princess Diana - Born in Themiscyra, definitely not a natural born citizen.

Flash - AFAICT Barry Allen would qualify, assuming he's old enough.  He's often portrayed as "younger" though.

Martian Manhunter - Born on Mars, definitely not a natural born citizen.  Alter ego of J'onn J'onzz probably couldn't prove he's been living in the US for 14 years though.

Aquaman - Born in Atlantis, definitely not a natural born citizen.

Shazam -  Billy Batson is 12 years old, so he's out.

Green Lantern - AFAICT qualifies regardless if it's John Stewart or Hal Jordan, and assuming both are old enough and they haven't wandered off planet for extended periods of time.

Hawkman or Hawkwoman - Ugh, different continuities.  I believe in the animated series Hawkpeople are born on Thanagar, so definitely not a natural born citizens.  Other continuities have them as Earth born, so they'd probably qualify in that case.

Cyborg - Unlike some of the above, Cyborg has always been portrayed as very young, running around with the Teen Titans for a long time and is a highschooler in Justice League: War.  I think he's out due to age.

Atom - Ray Palmer has always been portrayed as a Professor, he's gotta be old enough to qualify.

Black Canary - AFAICT qualifies, assuming she's old enough.

Green Arrow - Typically portrayed old, Oliver Queen should qualify.

What Is a Chinese Banquet Meal?

A few days ago I had a friend ask me about what goes on at a Chinese Banquet meal.  I thought I'd write up a short description.

A Chinese Banquet meal is almost always served family style.  The number of courses of food will vary depending on portion size and number of people seated at your table, but 10 courses is pretty common.  In my experience Chinese Banquet meals typically have the following structure

Cold Plate - The first dish that typically comes out is a cold plate of various appetizers.  The items on this dish will vary, but slices of pork, chicken, and jellyfish are the most common items I've seen.  I've also seen wakame salad (seawood salad you usually see in Japanese restaurants), octopus, and duck wings.

Soup - The soup comes next.  I think seafood soup is the most common type offered although this may differ by restaurant.

Vegetable dish - Atleast one vegetable dish is served.  A common one is bok choy with mushrooms.  If the meal is fancier, this might be billed as a seafood dish with vegetables (e.g. scallops with vegetables).
 
Fowl dish - In my experience it's duck more often than chicken.  Peking duck or some variant is quite common.  Poached chicken with ginger sauce is common too.

Crustacean dish - There will typically be a lobster or crab dish.  I think lobster is more common.  Unlike most Western restaurants, these are typically cut up and stir fried (or possibly baked) so you get tiny chunks to dig meat out of.

Fish dish - Typically steamed fresh fish.

Carb Dish - Either noodles or fried rice.  For those not too familiar with Chinese Banquet meals, you don't get rice with any of the prior courses.  I was told its because rice and carbs are cheap and a host wants you to fill up on the good stuff first.

Dessert - Of course varies by restaurant, but a sweet bean or rice based dessert is common.

If your party is a little on the large size, a few more dishes will be sprinkled in the middle to make sure there is enough food for everyone.

So as an example, here's a sample banquet menu from Mayflower Restaurant in Dublin, CA.  It follows my description nearly perfectly.

Deluxe Combination Platter
Great Wealth Crab Meat & Bird's Nest Soup
Sauteed Mince Squab with Egg White & Green
Sauteed Scallop & Calimari in XO Sauce
Whole Abalone with Sea Cucumber & Green
Crispy Chicken with Garlic
Lobster with Garlic & Butter in Supreme Broth
Smoked Sea Bass
E-Fu Noodles with Dried Scallop & Yellow Chives
Dessert