Renaming "Master/Slave" architecture to other terms

I learned of an interesting trend in software circles today. There has been a move to rename "master/slave" architecture into other terms, such as "master/worker", "leader/follower", etc. in an effort to remove the word "slaves" from code, docs, etc. I first noticed it done in Hadoop 3.0.  But it got a lot more attention when Django made the change and Mozilla donated 15K to BuildBot to fund similar work. Googling around, it seems Jenkins, Drupal, and Mesos also did the change in the last few years.  I'm sure there are more that I missed and am just not searching the right terms.

As you might imagine, this has lead to quite politicized discussions on bug/issue trackers.  Are we going "too politically correct"?  Couldn't engineer hours be used more effectively?  You can imagine all of the discussions that could follow.

But I found this comment from the Django issue tracker on this subject to be quite enlightening.

"I'm very glad for this change because as a PoC I felt very uncomfortable seeing and using this terminology in my code"

We can choose to be more inclusive and welcoming to all people that may code.  Or we can choose to stick to the past terms based on a somewhat arbitrarily chosen naming from eons ago.  As a group, I think we should try to make programming more "inclusive" than "exclusive".  I wrote about this a bit awhile back regarding the use of 0xB16B00B5 (i.e. "Big Boobs") in the Linux kernel and how such language can make programming seem like something for boys instead of girls.

I know there are those who think it shouldn't be changed due to it's large legacy meaning/usage and the fact that it's clearly not related to human slavery.  I'm sure there are others that feel that such as a change is just "being too politically correct."
 
I was trying to think of an example in software history that would serve as a good illustration of how it's a good idea to change these terms, even though they become nearly defacto terminology.

I couldn't think of one.  Some of it just might be because software is too "new" of a thing still.

I did end up thinking of one in medical history related to Down Syndrome.

According to the wikipedia page, Doctor John Langdon Down first characterized Down Syndrome in the 1860s and initially called people with it "mongoloids" because he considered the facial features to be similar to those of "Mongolians".  It wasn't until the 1960s that the term was official changed to "Down Syndrome", named after the doctor that characterized it.

I hope to most readers here that the reason for the change is obvious.  The term "mongoloid" was often used as a pejorative for people of Asian descent.  It was simply embarrassing to continue to use such an outdated/racist word to describe a genetic disorder going into the 1900s.  According to Wikipedia, it wasn't really until the 1970s that the term really disappeared.

I can't help but think, it took about 100 years for the term to be officially changed from it's original naming to the modern one.  That's a long time.  So that's a lot of medical history that had to be changed.  I believe that the change was due to the realization that while the original name was widely used in the medical community, times do change.  And as times change, it's perhaps best to move on and adapt to that change.

So I think the software community can change too.  So I decided to make the changes in Magpie.  While I can't do it everywhere, as some tools such as Spark and Hbase still rely on the term, perhaps this is the beginning of the change throughout.  Once projects like Spark and Hbase migrate, I can propagate it further.

Oddities in HBO's Silicon Valley

I recently began watching HBO's Silicon Valley.  Once in awhile I see a technical and/or conceptual oddity that I can't help but ponder over and think "uhhh no".  These are the ones that I noticed and bothered me.

I'm not going over any specific misuse of technical terms or over exaggerated elements for Hollywood humor.

Needless to say there are ***SPOILERS*** below if you haven't watched the show.

Season 1 - Episode 1

Richard is an employee at Hooli, but is also in a startup incubator launching Pied Piper.  Later in the episode Hooli CEO Gavin Belson tries to purchase Pied Piper.

Shouldn't Hooli have an intellectual property claim on Pied Piper, given that Richard is still an employee at Hooli?

Now I can't speak for the IP rights contracts that every company in the world has, and there are obvious exceptions (some companies you can sign documents to indicate you have some outside business interests that are independent of your job), but this seemed odd to me.

(They eventually delve into this in Season 2.)

Season 1 - Episode 6

In this episode, elite programmer "The Carver" is hired to help get Pied Piper towards its deadline.  In a somewhat cliche plot, he messes up some code and the team has to spend a lot of time fixing his mistake.

I couldn't help but think, "You guys don't have a code revision repository of any sort?"  The kind of thing that could be backed out within a minute or two? Later in the episode, they are distinctly comparing lines of code on two screens, so it strongly suggests they don't have a code repo.

But this isn't what gets me.  Later on, they also ran and passed regression tests to know they fixed the bugs/mistakes.  Regression tests suggest that you have different versions of software.  After all, what's the point of regression tests if you're not trying to catch old bugs/mistakes?  So they have different versions of software, but no code repo to manage the different versions of code?

Season 2 - Episode 10

During the trial, Erlich mentions that Big Head's code for "Nip Alert" was so bad that it crashed Richard's laptop and Richard had to take it to the Apple store to fix and it was "in the shop" for three days.

Now "crash" is ambiguous, but I interpreted the dialogue context to mean this caused a hardware problem.  It wouldn't have to be "in the shop" for three days otherwise.  It would be very hard, if not impossible, for a web based application (let alone a normal installed application) to destroy a laptop's hardware.  I can see potential theoretical ways, but it's hard to imagine a general application doing this. [1]  Add in the fact that Big Head is generally regarded as incompetent, it's hard to believe he could have done anything that advanced.

If "crash" caused a software based problem, such as a process foolishly running amok and deleting valuable system software (again, shouldn't really be happening for a userspace non-privileged process and shouldn't be happening with such a dumb application), this one is perhaps a bit more believable.  But a simple software reinstall and update from backup is all you need.  No need to bring it into the shop.

Season 3 - Episode 6

In this episode Richard is shown to be a lover of tab indentation in code.  He meets a girl named Winnie who codes uses spaces instead of tabs.

In one scene Richard and Winnie are sitting on the couch both coding and Winnie is programming and constantly smashing the spacebar to indent her code.  It's loud and annoying and eventually Richard can't handle it.  Hilarity ensues.

The "tabs" vs "spaces" debate is endless, but what I found interesting was the fact that Winnie repeatedly hit the spacebar to indent her code.  Today, almost every editor will automatically convert a "tab" to the appropriate number of spaces.  So there is no need to ever hit the space bar over and over again.  Even those who prefer spaces to tabs don't actually hit the spacebar over and over again, they use the tab key.

[1] - While writing this I Googled "can you program software to damage hardware" and there are some write ups with theoretical ideas that seem to confirm my suspicions.

Heroic Programming/Engineering in Hollywood

Once in awhile when I'm watching a tv show that includes some programming/engineering, I am amazed by the level of "Heroic Programming" or "Heroic Engineering" capabilities of the characters.

The one I've noticed the most is Timothy McGee on NCIS.  At multiple points in time McGee is capable of engineering solutions to help solve a crime on the order of an afternoon.  These are feats that would take people days to months.  On a few occasions McGee single handedly developed data mining algorithms and capabilities that probably bordered on the capabilities of Palantir.

Keep in mind that McGee has mastered these heroic capabilities, despite the fact that he doesn't even program as his day job.  So he has seemingly developed all these abilities as a side-task and/or hobby.

Another one I saw recently was on the show Silicon Valley.  While the individuals on the show are atleast full time engineers, they seem to have epic capabilities.  A very small team of just three engineers seems capable to engineer everything, on every platform, in every domain, without flaw, and at lightning speed.

Reflecting on a moment of personal growth a long time ago

After writing my prior blog post about remembering how I started programming, I recalled something from about 17 years ago.  Looking back, it was a great "personal growth" moment in my (then young) career.

I was at my first internship after starting college.  The company was a startup that worked predominantly on Unix/Solaris.  Very early on in the internship (I think the first week), I realized I knew alot less about Unix than the other interns and at times was really lost in what was going on during training.  I became worried about my ability to actually be able to do the job.  To be honest, it may have been the most lost I've ever felt at any point in my life.

As an example of how little I knew at the time, I recall that while I had setup my personal website using instructions I found, I did not actually know what things like "chmod 744" actually did.  At the time it was just something I knew you did to get things working.  I remember I didn't know the grep command.  I remember at one point asking another intern what a daemon was. 

So this was my knowledge of Unix and I was supposed to be doing Unix programming for this internship.

Now, I should mention that this was before the time of Google's dominance.  It was the time before Wikipedia and blogs.  There was no stackoverflow.

However, at the same time, I was much younger in my career.  I'm sure there were elements of "how to learn" that were not quite yet cemented in my mind.

Now, I could have done one of several things once realizing I didn't know anywhere near as much as I hoped (or should have).  I could have faked I did know what I was doing and continue on.  I could have shut my mouth and just see what happens.  On the extreme end I could have quit thinking I wouldn't be able to cut it there.  I've certainly known people who did those things like this in their lives.

I ended up going to my boss towards the end of the first week, telling him about my concern for this and how I felt really lost.  I don't think I'd ever been so worried over a conversation in my life.  But he was very calm and caring, telling me thanks for admitting it.  He got me some books from some of the engineers to take home and catch up on.  So I did read them starting that weekend.

The end result after that summer?  I ended up learning a lot more.  Became comfortable in Unix environments.  I got more accomplished that summer.  I like to think I proved I was a good employee, as I was invited back to work part time during the school year.

So I have looked back at this event and considered it an important moment in my personal growth when I was younger.  The formula for succeeding on projects/work isn't magical.  Ping others for help/pointers (today "others" can be Google), read stuff, put in hours, try things out, and eventually you can figure it out.  As I look back on my career, this exact formula is the method by which I had success on many projects despite feeling extremely lost/confused in the beginning.

Remembering How I Started with Programming

As I've said on this blog before, sometimes I can't help but feel dumb at times as I see so much brilliant code being written out there.

But at the same time, I have to remember that it takes years to master certain skills and knowledge.  I like to think people have looked at my code and think it's pretty nifty too :-)

Just the other day I was writing a simple python script for some personal use at home.  I wanted it to do a few different things in it and at some point it occurred to me that I've never parsed command line options in a Python script before.  I guess up to this point I'd only written Python scripts to do one simple thing or modified much larger bodies of code that others had started (i.e. I probably cut & pasted prior argument parsing code to add the option(s) I wanted).

I went online and started reading through the Python argparse documentation.

I suddenly paused and realized something.

About 17 years ago I was in my first internship in college.  I was writing tools/commands in C to execute test code.  I still remember reading the getopt manpage at one point because I had never actually done argument parsing in C before.  The first time you see a string like "hVa:b:", it can be quite confusing.

About 14 years ago, in my first month of my first fulltime job after college, I was working on Genders.  I remember reading about getopt_long because I had never dealt with long options in C before.

So I actually sat back and paused, realizing that I've learned a lot over the last 16 or so years.  And sometimes when you feel dumb that you aren't an expert in one particular subject, you have to remember that the things you are more of an expert in, you had to read that documentation for the very first time at some point too.

The Mendoza Line In Programming

For those who are unaware, there is a concept called "The Mendoza Line" in baseball.

Short description, there was a baseball player named Mario Mendoza, who (to put it frankly) wasn't a very good baseball player.  Now, making it to the major leagues is an accomplishment itself, and Mario Mendoza was able to scrape together a 9 year career mostly as a utility player.  He represented the minimum level one had to perform at to play and have a career in baseball.  Thus the term the "Mendoza Line" was coined.  If you fell below the line, you couldn't be a major league player.

I've wondered at times what is the "Mendoza Line" for a programmer?  I have a suggestion for the line.

Lets say there is some manual activity that will take you time X to complete. For this example, lets say there are similar typos in a lot config files.

If time X is really small, like a minute, we'll probably just do it manually.  This is perhaps the case when the typo is only in a small number (e.g. 1-2) of config files.

Lets consider if time X is a reasonable size and non-trivial.

If it would take you > time X to write a script to do the manual activity, you still might do it manually.  In my example above, perhaps the typo is in 5-8 config files.  Perhaps the config files and typos aren't similar enough to warrant figuring out the right regex.  So even if it's annoying, you still do it manually.

If you can develop a script faster than time X, obviously you should develop a script to do this manual activity.  Using my config file example, perhaps the typos are in over 500 config files.  Now finding the right regex is worth the time and energy so you don't have to do this task manually.

Now, we all have to do some manual activities in our day to day lives on a computer.  Some of these things are automatable (sp?), but we continue to do the manual activity b/c the effort to automate just isn't worth it (e.g. most don't script hitting the brew button on a coffee maker).

I like to think of this as the "Mendoza Line" for programming.  If you can recognize some manual activity is taking up some time X, can you recognize a script can do it faster, and write the script in less than time X.  If you can do this, you probably can have a career as a programmer.

Now I do want to say that the variable "X" does depend on a number of factors.  It can include job function, experience, and skill set.

I know very talented system administrators that can pump out very nifty sed/awk lines to do things in just a minute.  This makes sense.  There are probably many manual things that they have to do in their jobs so they have found it worthwhile to learn and commit to memory all the advanced parsing that sed/awk can do.  I personally have not b/c I don't do it enough in my day to day life.  So "X" for a task might be small for an advanced system administrator (lets say ~5 minutes) while it'll be larger for me (lets say ~15 minutes) for some setup/config activity.

Another variable to X is how often you might repeat it.  If you don't believe you'll ever have to do the manual activity again (b/c it's not your job and you're doing it as a favor to someone else) perhaps time X is viewed as small only because it's a one time activity.  But if your job function involves doing the activity many times, the potential cost is much larger.

There are perhaps other factors that affect "X", but hopefully the point is understood

I'm really good at fixing typos!

I have a joke amongst colleagues that I am "only good at fixing typos in open source projects."

Why?

Whenever I submit a code patch for some bug and/or feature in a open source project, it can often go ignored.  Why?  Because project maintainers are busy and they don't have time to look at every patch that comes in.  If the patch solves some specific problem they are looking at, then you're in luck.  But more than often I'm fixing something that isn't on their current radar.  Eventually, the patch gets lost amongst the other patches in their review pile and is forgotten.

Why do I know that patches will be forgotten or ignored for long periods of time?  It's because I do the same thing! :-)  It's normal and reality with open source projects.

But what happens when I submit a patch that just fixes a typo in comments or a typo in documentation?  This takes almost no effort to look through and review, so the patch is accepted immediately.

I do the same thing in projects I maintain.

As time goes on, there are a few projects that I suddenly feel like the only thing I can contribute is typo fixes.

Thus, I'm really good at fixing typos :-)

c'est la vie

At what point am I a "Googler" and not a programmer?

Just the other day what should have been about 15-30 minutes of fixes to a Perl script took me atleast an hour because of constant "I can't remember how to do this in Perl" Google searches:

"perl elif or elsif" - I remembered enough that "else if" wasn't the correct syntax in Perl, but couldn't remember which of the common alternates it was.

"perl conditional string compare" - I couldn't remember if it was "eq" or "-eq".

"perl string concatentation" - I actually had bugs in my script because I thought "+" was supported in Perl.  Whoops, it's "." in Perl.

"perl break continue" - Script errors informed me that I was using the wrong syntax.  Had to out right Google the answer b/c I couldn't remember it (in Perl it's "last" and "next").

There might have been a few more searches.

Needless to say, Perl is not my bread and butter language.  It's something I use once in awhile because I need to script something of some complexity just beyond a shell script.  When you jump between languages, the exact syntax of what you should be using can become a blur.

So it lead me to this question I think about once in awhile "Am I a professional Googler?"  There are times I spend way more time Googling than I would like to or think I should be.

First, is Googling an answer bad?  Admittedly, it is something that just feels bad.  A large part of this is because when you can't recall things from memory, you tend to think of yourself as stupid.  In some ways, it makes you feel lazy.  That perhaps reading a book would be wiser or finding the answer in a book would be the "smarter" thing to do.

IMO, this isn't the case.  You can't remember everything.  Many things I've done professionally in my career I haven't done from memory.  Significant portions of programming knowledge came from books.  When I needed a piece of information I developed a mental index of where to look in my mountains of books for specific answers.  In fact, I distinctly remember having to go to my Perl book to look up "What's the perl equivalent of break & continue" in the past because I couldn't remember it then.

Eventually that mental index simply turned into Google.  Various blogs, technical documentation, and StackOverflow have turned into the books on my shelf.

So I do not believe there isn't inherently anything wrong with Googling various answers and it doesn't make you any less smart.

But ...

Using Google to recall syntax nuances is one thing.  I think there is a mental line one crosses to begin to wonder, "Am I really Googling too much?"  At some point are you barely even programing?

I like to think of a recent project I worked on called Magpie.

Originally, I had more grandiose Big Data goals related to Big Data on HPC.  But as things sometimes go, a far more humble solution using just Bash scripts was found to be just as much, if not more, useful than many of the original project ideas.

These Bash scripts were initially hacked up because they were meant to be "support" for my actual work.  When we determined that the support-ish Bash scripts were what people found to be the most useful product of the work, did we turn it into a more polished product.

I never sat down and actually learned Bash scripting syntax at any point in my life.  Basics such as conditionals and loops were cut and pasted from online guides when I've had to develop small scripts before.  More nuanced things were gleened from more Googling and cut and pasting.  There were many searches for "bash" + "sed|awk" + "some pattern" because I never learned many of the advanced features of sed or awk or any of the advanced Bash-isms.

I've cut and pasted my share of code before.  Whenever I've needed a function that I deemed reasonably common and didn't want to write on my own, I was able to find one online.  But this was different.  I felt that I cut and pasted an entire solution together from stuff on StackOverflow.  To be honest, if I was given simple interview programming questions and asked to do it in Bash, I doubt I'd be able to do it.

In contrast, I think of how I originally learned to program in C.  I took a small C programming class in highschool.  In college it was in a few of my classes.  I read about half of the famous Stevens network programming book in college and the rest once I began my career.  There were other books too.  My programming expertise in C was hardened through many homework assignments, work assignments, and bugs.

Granted, C is a far different beast than Bash, but when I learned C the web was just in its infancy.  A few not-great search engines like Excite and AltaVista existed.  Google was just coming online.  There was no Stackoverflow or blogs with help on individual niche topics.  So learning it primarily meant you had to learn from scratch.  So C programming is simply backed into my brain.

I wonder if things would have been the same if StackOverflow was around when I was first coding in C?  Would I have not ever read the Stevens networking books and coded through StackOverflow?  Is this what people do in college now doing their assignments?

It's something I've pondered.  But getting back to the original question ... Am I a professional Googler?

I eventually realize the answer is no.

In actuality, it takes years to get the point that you can become proficient enough to actually understand what it is you are looking for so you can cut and paste off the internet.  When you find the piece of cut & paste that is correct, you know it and you can use it.

What I have to remember is yes, I did cut & paste off the internet for much of Magpie.  Perhaps it does make you feel lazy.  But it was individual lines of syntax, not an entire solution.  The real meat of Magpie is all of the Hadoop, Spark, etc. knowledge in Magpie within the scripts.  I just needed bits of code to piece it together.

It's no different then when you cut & paste from your personal library of code.  The first time you need to program something tricky, you really figure it out.  But everytime you need it later, you cut & paste (or if it's a library, just call it, etc.).  But eventually, you probably forget all the subtle details you put into making it robust and solid in the first place.  Do I remember all the details of how to make a portable non-blocking connect call?  Absolutely not.  But after I wrote it the first time, I used it several more times over.

While writing this blog post, I had almost forgotten something.  While I was developing Magpie, I did write many 5-10 line scripts off to the side to figure out "is this how the syntax works?" or "is this what I'm looking for?"  I had completely forgotten this reality.  So I didn't actually blindly cut and paste all of the time.  And of course I did eventually learn some things.

And ultimately, at the end of the day, one tends to invest time in those things they think worthwhile.  Do I consider Magpie to be of some complexity that it requires my attention to learn all Bash syntax in depth?  The answer is no.  So I didn't bother to.

Beginners Guide to Hbase with Python & Thrift

I recently wanted to play around with Hbase and Python, which subsequently lead me to use Thrift.

I know there are tons of guides on the web, but a number I found were outdated or based on downloaded versions of things instead of packaged distro versions.   I eventually had to piece together information from several sources.  So I thought I'd put it altogether on this page for anyone looking for simple "cut and paste" instructions to begin with.  I'm not going to go through the basics of Hbase and Thrift, as there are many guides out there, but I'll give updated instructions based on the following versions I used.

Hortonworks 2.2.6.0-2800
Hbase 0.98.4
Redhat 6.7

As an aside, there may be newer interfaces (such as happybase) that are now more popular.  I may look into those later, but these are just my notes on this particular subject.

As another aside, my Hbase has already been populated with data, so there's no need to create/insert data, so I'm skipping that.

The two primary Hbase + Python + Thrift sources I used for this were:

Using Facebook’s Thrift with Python and HBase (posted July 2008)

and

How-to: Use the HBase Thrift Interface, Part 1 and Part 2 and Part 3 (posted September 2013)

So lets start.

First up, I downloaded thrift 0.9.3 and did the normal configure and make, but this didn't compile for me.

src/thrift/qt/moc_TQTcpServer.cpp:14:2: error: #error "This file was generated using the moc from 4.8.1. It"
src/thrift/qt/moc_TQTcpServer.cpp:15:2: error: #error "cannot be used with the include files from this version of Qt."
src/thrift/qt/moc_TQTcpServer.cpp:16:2: error: #error "(The moc has changed too much.)"

this was also the case with thrift 0.9.2, 0.9.1, and 0.9.0.

I went to thrift 0.8.0 and hit other build errors.  These were maybe solvable, but being lazy I just downloaded 0.7.0 to try it, and it compiled fine.  So I ended up using thrift 0.7.0.

Since this compiled, I need to install it somewhere.  I'm going to install into a non-privileged directory, so set all of these prefixes appropriately when you configure and make install.  If you're root and you can install anywhere, you can probably ignore all of this.  Adjust appropriately if you use bash instead of tcsh.

setenv PY_PREFIX /yourprefix/thriftpy/
setenv JAVA_PREFIX /yourprefix/thriftpy/
setenv RUBY_PREFIX /yourprefix/thriftpy/
setenv PHP_PREFIX /yourprefix/thriftpy/
setenv PHP_CONFIG_PREFIX /yourprefix/thriftpy/
setenv PERL_PREFIX /yourprefix/thriftpy/
./configure --prefix=/yourprefix/thriftpy --exec-prefix=/yourprefix/thriftpy
make install

After that, you should hopefully have thrift installed into your appropriate path and /yourprefix/thriftpy/bin/thrift should be available to run.

Now run thrift to generate Python files for Hbase

> /yourprefix/thriftpy/bin/thrift --gen py /usr/hdp/current/hbase-client/include/thrift/hbase1.thrift

Now if you're wondering why hbase1.thrift instead of hbase2.thrift, it's because there is now a newer thrift interface available compared to the original.  I use hbase1.thrift just to begin with.


Now there should be a "gen-py" sub-directory where you ran this.

Now we need to start the thrift server.  With HDP you can do:

/usr/bin/hbase thrift start

I started this in another window so I can control+C is later on.

Ok, onto code.  I started with code sort of from both the sites above and put together:

#!/usr/bin/env python                                                                                                                                 
import sys

sys.path.append('../gen-py/')
sys.path.append('../thriftpy/lib64/python2.6/site-packages/')

from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol

from hbase import Hbase

# Make socket                                                                                                                                         
transport = TSocket.TSocket('localhost', 9090)

# Buffering is critical. Raw sockets are very slow                                                                                                    
transport = TTransport.TBufferedTransport(transport)

# Wrap in a protocol                                                                                                                                  
protocol = TBinaryProtocol.TBinaryProtocol(transport)

client = Hbase.Client(protocol)

transport.open()

tablenames = client.getTableNames()

print "table names are: " + ",".join(tablenames) 

You'll notice a few sys.path.append calls to add paths to my local dirs with libraries in them.  You may need to adjust accordingly for your installed things.

Hopefully, you should just be able to run this and there will be no problems (unless you've never created tables in Hbase, which you should create some).

Ok, now to read some data.  Again, pasting together some code from the above resources, this will get a single row of data.

rows = client.getRow('mytable', 'myrowkey')
 
for row in rows:
    message = row.columns.get('mycolumnfamily:mycolumnname').value
    print "message = " + message
    rowKey = row.row
    print "rowkey = " + rowKey

This didn't work for me and I got an error of

TypeError: getRow() takes exactly 4 arguments (3 given)

Hmmm.  Now, if you look in the file hbase1.thrift from before, you can see what the interface for getRow looks like.

  /**
   * Get all the data for the specified table and row at the latest
   * timestamp. Returns an empty list if the row does not exist.
   *
   * @return TRowResult containing the row and map of columns to TCells
   */
  list getRow(
    /** name of table */
    1:Text tableName,

    /** row key */
    2:Text row,

    /** Get attributes */
    3:map attributes
  ) throws (1:IOError io)

Hmmm, it seems there is a new attributes argument.  I couldn't figure out what this did by searching online.  I didn't want to dig into the code too much at this point so I just passed in None to the getRow call like so:

rows = client.getRow('mytable', 'myrowkey', None)
 
for row in rows:
    message = row.columns.get('mycolumnfamily:mycolumnname').value
    print "message = " + message
    rowKey = row.row
    print "rowkey = " + rowKey

And lucky for me it worked.

Now getting one row of data is boring, we actually want to do scans.  So I started with the following.

scan = Hbase.TScan(startRow="someStartPrefix", stopRow="someStopPrefix")
scannerId = client.scannerOpenWithScan(desiredtable, scan)

rowList = client.scannerGetList(scannerId, 5)

while rowList:
    for row in rowList:
        mydata = row.columns.get("mycolumnfamily:mycolumnname").value
        rowKey = row.row
        print "rowKey = " + rowKey + ", mydata = " + mydata
    rowList = client.scannerGetList(scannerId, 5)

Again, I hit

TypeError: scannerOpenWithScan() takes exactly 4 arguments (3 given)

Just like getRow, there is a similar attributes argument I don't know what to do with. So I add a None argument to scannerOpenWithScan like so.

scannerId = client.scannerOpenWithScan(desiredtable, scan, None)

And this works and I get results.

Now, getting data with Hbase with start & stop rows is boring.  It's far more interesting to do filters.  How can we pass filters in Python?  Again, looking at the hbase1.thrift again, I can see what arguments TScan can take.

/**
 * A Scan object is used to specify scanner parameters when opening a scanner.
 */
struct TScan {
  1:optional Text startRow,
  2:optional Text stopRow,
  3:optional i64 timestamp,
  4:optional list columns,
  5:optional i32 caching,
  6:optional Text filterString,
  7:optional i32 batchSize,
  8:optional bool sortColumns
}

Hmmm, this filterString argument looks interesting.  But what to fill it with?  After I some Googling, I figure out it can be filled with functions you can find in the Hbase thrift documentation.

So here's some examples.

scan = Hbase.TScan(filterString="RowFilter(>=, 'binary:FOO')")

this is functionally identical to

scan = Hbase.TScan(startRow="FOO")

You can AND/OR things together.  So for example:

scan = Hbase.TScan(filterString="(RowFilter(>=, 'binary:STARTPREFIX') AND RowFilter(<=, 'binary:ENDPREFIX')) AND (RowFilter(=, 'substring:FOO') OR RowFilter(=, 'substring:BAR'))

Would find rows with the substring FOO or BAR within a range of STARTPREFIX and ENDPREFIX.

Well, that's as far as I've gotten.  There were a few gotcha points, so I hope that this helps somebody out there.

Hadoop Job Submission Errors

Ugh, for the life of me I couldn't figure this out today until the "Duh ... I didn't do ..".  We all have those days.  Hopefully this will help someone out there a little quicker.

I couldn't submit a small Hadoop job today and was repeatedly getting errors like this:

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class FOOCLASS not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1961)
        at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.ClassNotFoundException: Class WordCount$WordMapper not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1867)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1959)
        ... 8 more 

The key line of error output was:

WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).

After Googling I tried all the normal expectations.

1)

Make sure the jar has all the classes in it that it's supposed to have.

2)

Make sure you're calling job.setJarByClass() in your code.

3)

Make sure the permissions on the jar file are correct.

But still couldn't get it to work.  Then a stackoverflow post suggested calling job.setJar() to manually set the jar by it's full path.

This worked.  Why?

Duh ... Hadoop couldn't find the jar submitted in question.  So step 4 to try.

4)

Make sure your jar can be found in the environment variable HADOOP_CLASSPATH.

An alternate option may be to ensure your jar is the classpath of   yarn.application.classpath.  But I didn't try this.

Moving from Spark 0.9.1 to Spark 1.0.0

I recently had to support Spark 1.0.0 in a project (Magpie).

The conversion from Spark 0.9.1 to Spark 1.0.0 was a bit annoying, as many changes had happened.

Here are a list of changes that I thought were worth mentioning.  Hope what I say here can help others.

1) Running examples differences

In Spark 0.9.1, you would run one of the Spark examples (such as SparkPi) like this:

> bin/run-example org.apache.spark.examples.SparkPi spark://SPARKMASTER:7077 
 
In Spark 1.0.0, running examples through run-example requires the Spark master to be specified through the MASTER environment variable and not on the command line.  So for Spark 1.0.0, you'll want to do something like this instead:

> export MASTER="spark://SPARKMASTER:7077"
> bin/run-example org.apache.spark.examples.SparkPi

otherwise you'll get a bad input error/exception.

If you don't set the MASTER environment variable, run-example will assume you want to run the example locally.

Note that setting the MASTER environment variable is specific to the run-example script.  It won't pick up the default value from the new spark-defaults.conf file.

2) spark-submit script

The spark-submit is a new wrapper script for submitting Spark jobs.  Although you can still use spark-class directly, this is the primary job submission script. It has the following usage:

> bin/spark-submit --class JOBCLASSTORUN [spark-submit options] APPLICATIONJAR [application args]
Note that the job jar is now passed in on the command line, unlike before with spark-class.

There's all sorts of new options in spark-submit.  Here are some options of note taken from the --help output:

  --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
  --class CLASS_NAME          Your application's main class (for Java / Scala apps).
  --jars JARS                 Comma-separated list of local jars to include on the driver
                              and executor classpaths.
  --properties-file FILE      Path to a file from which to load extra properties. If not
                              specified, this will look for conf/spark-defaults.conf.
  --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 512M).
  --driver-java-options       Extra Java options to pass to the driver.
  --driver-library-path       Extra library path entries to pass to the driver.
  --driver-class-path         Extra class path entries to pass to the driver. Note that
                              jars added with --jars are automatically included in the
                              classpath.

But you probably won't end up using these, you'll more likely use ...

3) spark-defaults.conf

Previously, options were configured through SPARK_JAVA_OPTS, but that is now deprecated. Everything should now be done through the spark-defaults.conf file. It is read and loaded from spark-submit when you submit a job. By default it is read in conf/spark-defaults.conf but that can be altered using the --properties-file option. In addition, settings through SPARK_CLASSPATH or SPARK_LIBRARY_PATH should now be set through spark-defaults.conf as well.

Here are several options for spark-defaults.conf of particular note, with the full list in the Spark documentation.

spark.master                     set Spark master, e.g. spark://SPARKMASTER:7077
spark.executor.memory            set executor memory, e.g. 1024m, see more below
spark.executor.extraClassPath    what used to be set by SPARK_CLASSPATH
spark.executor.extraLibraryPath  what used to be set by SPARK_LIBRARY_PATH 

4) deprecated SPARK_MEM environment variable

The SPARK_MEM environment variable has been deprecated and replaced by two new configurations so users can configure the Spark executors and driver memory separately..

The SPARK_DRIVER_MEMORY environment will set memory for your Spark driver.  This could also be handled via the --driver-memory option in spark-submit.

The memory for Spark executors is now handled by the spark.executor.memory option in spark-defaults.conf.  Documentation indicates the environment variable SPARK_EXECUTOR_MEMORY will also work, but I didn't try that.

5) deprecated spark.local.dir

The configuration option spark.local.dir is now apparently deprecated in favor of the SPARK_LOCAL_DIRS environment variable.

HDFS vs Lustre

There's been discussion out there about comparing the HDFS filesystem to a traditional parallel filesystem like Lustre.  The problem is it's really difficult to compare apples to apples.

As an example, I saw a white paper awhile back (sorry, I can't find it online) that compared HDFS to Lustre.  HDFS beat Lustre in this person's performance tests by a good margin.  After digging into the paper I saw why.  This fellow ran Lustre over a 1 GigE ethernet network. 

Is this a fair test?  On the one hand it isn't because Lustre is a network based filesystem.  If you simple choose to bottleneck Lustre, of course it will lose.   On the other hand, it's a fair test, because it uses the same hardware most use with HDFS.

So lets say we replaced the GigE with Infiniband.  Would it now be a fair test?  Perhaps its slightly fairer, but HDFS people can say HDFS wasn't designed for more expensive hardware and therefore doesn't take advantage of it.  In the case of Infiniband, HDFS isn't using RDMA during replication.

I don't know the right comparison.  However, HDFS vs Lustre may not be the correct comparison to think about.  At the end of the day, I could probably concoct an HDFS setup that will always beat a Lustre setup and vice versa.

I believe thinking about this as HDFS vs Lustre isn't the right approach.  It's really Hadoop Cluster vs HPC Cluster.  At the end of the day, while Hadoop is famous for handling large data, the reality is because of shuffle/sorting/scheduling/etc. in Hadoop, it also reads/writes tons of small files.  The memory for a Hadoop Cluster vs HPC cluster may also be different.  That affects spilling of data, page cache, etc.

Update: See "Big Data vs HPC" follow up.
Update: See "HPC vs Big Data" follow up.

Update 6/2/15:

Not so long ago I was talking to someone about the HDFS vs Lustre comparison.

Many people have done HDFS vs "Some Networked Filesystem" experiments.

However, I think these experiments are inherently flawed.  The experiments always look something like this.

Datanodes
8 nodes
4 SATA disks
8 core
32G RAM

Networked Storage
4 nodes
8 SATA disks
8 core each
32G RAM

with additional hardware details beyond this.

The comparison will be HDFS using the Datanodes for data & map reduce.  Then it'll be a comparison to the Networked Storage, also using the Datanodes as the computation facility.

Do you see the inherent problem in the above comparison?

.
.

It's staring you right in the eyes.

.
.
.

It's an 8 node test vs a 12 node test.

It's a 256G RAM test vs a 384G RAM test.

This isn't to say that the comparison is poorly done.  But this is part of the inherent problem of comparing HDFS vs Networked File systems.  What is a fair comparison?

Why bother to have specifications ...

Ugh ... a very annoying e-mail thread from earlier this week.  For the record, I'm not attacking the other fellow on the other end of this thread.  They're just doing their job too.

Company: We noticed in your open-source software that you output X incorrectly.  Please look at this other open-source software Y to see how it should be output.

Me: Please look at standard Z.  According to standard Z, I'm doing things correctly.  Here's the code snippets to show it.

Company: Hmm, you're right.  I guess a number of vendors are not properly sending the data in the right format.  Unfortunately, we can't get all those vendors to change the format.

Me: Well, how about I add a workaround option on the command line.  Those who are knowledgeable of this subject matter can specify it if they want to.

Company: We tried out your workaround option and it's almost correct.  A few bytes were flipped in the output.

Me: Huh, that's strange.  The first few fields are sent little-endian, but the latter few fields are sent big-endian.  This is really weird.

Company: For legacy reasons, it appears large company A has been doing this, so I guess vendors have followed suit.

The Baseball Scheduling Problem

Came upon this ESPN video at random the other day.

http://www.grantland.com/story/_/id/9930115/our-latest-30-30-documentary-short-tells-story-how-husband-wife-ended-creating-schedule-mlb

For years I had assumed that baseball (and all professional sports) had used computers to determine their season schedules, but apparently the scheduling problem was so difficult and so complex, human beings did Major League Baseball's schedule by hand until 2004.  The couple highlighted in the video did it for about 20 years.  Apparently multiple companies and research universities (MIT, Stanford, Carnegie Mellon, IBM are mentioned) failed to come up w/ schedules in the 80s and 90s.  In some cases, it was not failed to come up with "good schedules", but apparently failed to come up with a schedule at all.

The scheduling problem seems mighty complex.  The scheduling conditions below are from the video and my own knowledge of baseball.

• 30 baseball teams, each plays 162 games (2430 games total)
• The schedule is unbalanced, teams in the NL or AL play within their league more often than the other (15 teams per league).
• Teams within divisions play each other more often than other divisions in the league (5 teams per division).
• Teams cannot play any other team in the other leagues (interleague play) but only specific ones in certain divisions.
• Cities w/ two teams (Chicago, NY, etc.) must not be in the city at the same time on specific days (especially weekends and such)
• Special games are scheduled in for specific rivalries.  e.g. New York Yankees and New York Mets will play each other each year no matter what.
• For ratings/ticket sales, specific rivalries (intra division or otherwise) must be scheduled at specific times. e.g. intra-division battles almost always must be scheduled at the end of the year in September.  e.g. Boston vs NY or St. Louis vs Chicago games should almost always be on weekends.
• A specific number of rest days are mandated for each team.  They must be distributed evenly throughout the year (i.e. not back to back).
• Rest days must be scheduled in when long travel occurs (e.g. west coast teams going to east coast).
• Traveling must not be unbearable (e.g. you can't schedule team to play in NY, then LA, then Miami back to back to back).
• Each team has special requests to be home/away on certain dates b/c of special occasions or city specific events.  For example the SF Giants may not want to be in town during Fleet Week.
• Scheduling must be fair and equitable for holidays.  For example, it would be unfair to never allow a team to have a home game during the July 4th weekend.  In addition, you must handle Canadian holidays.
• Some teams may share their stadium with other professional sports teams, so that must be handled.  Although this constraint certainly existed in the past, it may not exist anymore (off the top of my head, St. Louis, San Diego, San Francisco, and Houston used to share their stadiums but no longer do).  Although I'm sure special case handling probably still exists (e.g. state championships, conventions, etc.).

I'm sure there are even more constraints that they don't mention and I can't think of.

Crazy Grad. School Stories

Not so long ago, I had a chance to meet and listen to a talk from Jorge Cham, the created of PhdComics.  He gave a hilarious talk about his comic strip, suffering through graduate school, and about how to deal with grad. school.

At the talk, I asked him what the craziest grad. student story of suffering or pain was that he had heard during his touring and talks.  He said that amongst the worst he ever heard were about students that went into their final thesis defense without a presentation or slides.

So I began to think of the best "grad. school" story I've heard.  Here's the one I could think of.  It's not about research or grad. school suffering, but it's still a funny story none the less.

My friend was TAing the CS class for business students.  It's the class where you teach Excel, Powerpoint, and some introductory programming in Visual Basic.  Since the class is all business students, and the majority of them do not intend to ever look at code in their life, the programming assignments are not liked.


So he tells me there is a website called hireacoder.com, where you can find programming contract work.  While the website is mostly for small contract work (i.e. design this aspect of a website for me) a fair amount of the contract work posted is apparently homework assignments from students.  While most of the people posting contracts are smart enough to anonymize the assignment, one was not and listed the class name and school in the contract (apparently cut & pasting it from the assignment).

A good samaritan found the class online and e-mailed the Professor and TAs about the cheater.  At first, the Professor said she was irate, but eventually got over it, realizing there was nothing she could do to catch the culprit.  After all, how were you going to catch the cheater out of a class for several hundred students.

Weeks later while grading the assignments, one of the TAs found a very special submission where "Written by Joe Smith @ Hire A Coder" was at the top of the comments.  Clearly, the student didn't even bother to look through the code they had bought and blindly turned it in.

When questioning the student, the Professor and TAs asked how much they paid for the assignment to be done.  The student said about $50 (late 1990s money).  Which the Professor replied, "There are senior CS students that would have done that for a slice of pizza.  So not only did you fail my class, you've also failed finance."

Vendor Communication Frustrations

It's common for me to find firmware bugs on vendor hardware that interacts poorly with software written in house.  My assumption is that vendors sell their software as a "value add", only test their hardware with their software, and have limited care for third party software.  Once in awhile the vendor fixes hardware issues in their software, so the issue is technically in the third party software, but it's not the third party's fault.

When I report a bug in the firmware, it's common for a vendor to respond with something akin to, "It works for our software, it must be your software."  It's a normal, yet frustrating, response.  In addition, to get the problem resolved often means getting through layers of support before reaching an engineer that understands the situation.

Over time, I've come up with a few mechanisms to handle this type of issue with vendors more quickly.  Here are some of them:

A) Add packet dumps or equivalently "advanced" debug information in the ticket.

I'll add information to the bug that is reasonably complex or difficult to understand.  The common example is something similar to a TCP packet dump.  I add something in the ticket with arrows (e.g. "See here --->") and something like "this 0x8 should be an 0x5".

Due to the complexity of the information, the ticket is typically passed up the food chain more quickly.  In addition, it sometimes does not matter what software was running to generate the TCP packet dump.  I can just say it was their software, they would never know.

Ironically, I sometimes have nicer/better debugging information (or even the flat out solution) I could put in the ticket.  However, I often elect to put the more confusing information in the ticket b/c it ultimately resolves the problem more quickly.

B) Be very specific in your requests

It's better to ask for very clear specific information.  A normal request might go like this: "I noticed an OEM piece of information on your motherboard.  Can you please describe what this means?"

While this is a perfectly reasonable request, it leads to the wrong answers.  A common answer might be, "Use our software X, it can show you all the information", which is of course not a useful response.  Or you might get "That information means FOO, you need to do BAR for your system", which might be true, but isn't what we're really looking for.

It's better to be very specific with your request, for example  "I noticed the following OEM piece of information: 0x20 0x18 0xA2.  What is the mapping of OEM hex to English.  0x20 = ???, 0x18 = ???, 0xA2 = ???."

C) Site standards/specifications that can't be challenged

I'll site standards/specifications, making it clear that if they wish to counter my claim, they will have to know what they are talking about and prove it to me.

For example, I might write, "Please see the following packet dump.  Clearly the X field violates section 1.2.A of the specification, second paragraph, third sentence."

This type of statement is so specific and advanced, it will typically skip lower support levels.

How Coding WTFs Happen

I love to laugh at the coding gaffes on TheDailyWTF, but I know that I shouldn't laugh, because I know the coding gaffes could happen to me by chance.  I saw a code snippet like this one awhile back in code I work with.

switch (val)
{
  case ENUM_A:
    do_A();
    break;
  case ENUM_B:
  case ENUM_C:
  default:
    do_default();
    break;
}

I remember this pretty clearly b/c I know exactly how it reached this point.  The code used to be like this:

switch (val)
{
  case ENUM_A:
    do_A();
    break;
  case ENUM_B:
    do_B();
    break;
  case ENUM_C:
    do_C();
    break;
  default:
    do_default();
    break;
}

However, as code was condensed and options eliminated, people removed only the portion of code that was necessary to remove.  Only after looking at the code from a distance does someone finally realize this should be condensed to something far more reasonable (and removing the unnecessary enums too):

if (NEW_BOOL_FLAG)
   do_A();
else
   do_default();

Immature Programming

The other day I came upon the following article:

"Microsoft Apologies for "Big Boobs" Coding Gag"

The short story, a Microsoft programmer had put "0xB16B00B5" (i.e. "big boobs") in some Linux kernel code.

I thought this quote in the article summarized my thoughts on the subject

"At the most basic level it's just straightforward childish humour, and the use of vaguely-English strings in magic hex constants is hardly uncommon. But it's also specifically male childish humour. Puerile sniggering at breasts contributes to the continuing impression that software development is a boys club where girls aren't welcome."

I couldn't agree more.

Then I paused for a second ...

While I like to think of myself as having always been the consummate professional, I suppose I was once younger and more immature.  I was once told the following "cool" hex string by an older programmer and put it into some code maybe 10 or so years ago:

#define _MAGIC  0xcafebabe

IMO not as bad as "big boobs", but not exactly good.

So when I was told this "cool" hex string, a younger version of me thought nothing of it.  The fact older colleagues also thought it was cool, probably established the idea this kind of thing was acceptable.

A now older version of me now would pause and think "I probably shouldn't."

So what's the lesson learned now that I'm a tad older?

A) I shouldn't encourage this by teaching it to younger engineers

B) I should never accept them in patches or let them get through code reviews.

Update (8/1/12):

Sigh, the ghost of 0xcafebabe affected me another time somewhat recently.  A colleague of mine had programmed:

#define PLUGIN_FOO_INIT_NUMBER 0xcafe

When I had to write a new plugin, I copied his code to start the new one.  What did I put at the top of the file?

#define PLUGIN_BAR_INIT_NUMBER 0xbabe

Well, the best thing I can do is admit the mistake, change the code, and be more conscious of this in the future.

Getting Everyone On The Same Page

I was reading this article on TheDailyWTF.

http://thedailywtf.com/Articles/Get-to-da-COPPA!.aspx

Like many TheDailyWTF articles, it's a nice humorous story, but I love the last part of the story.

The short story is engineer Derek could not come to a reasonable compromise with engineer Steve on a solution to a problem. After many internal frustrations, Derek bypassed Steve's engineering team entirely, went to the software install team, and changed the install process for the software. The result? Derek implements his solution without ever involving the team that writes the software. The end of the article states, "Steve's team got to keep their constraint, and the customers didn't."

I love that last sentence.

"Steve's team got to keep their constraint, and the customers didn't."

The core engineering team will never see a setup (and will never know the setup for awhile) that the customers will always have. Long term, that can't be good.

While this is an extreme example, it got me thinking. How often do internal managers/staff not come to an agreement or get on the same page? As a consequence, staff begin doing whatever it takes to get the job done, bypassing teams, roles, procedures, etc. I think tiny versions of it happen all the time. Some amount of it we accept b/c we work with a lot of people, but how much of it can be made better?

In twenty years, will I be the old timer that still programs in C?

I was speaking to someone at a party that told me COBOL programmers make a lot of money nowadays. Somewhat shocked, I asked how this was possible. His answer was simple. There's a ton of COBOL legacy software out there and very few out there who can work on it. Companies are paying up the wazoo to grab those few out there still remaining.

I recently participated in some interviewing events for recent college grads/soon to be college grad
s. It blew me away how few of the candidates had worked in C or C++. In fact, some barely even touched C/C++.

So I started wondering, will I be that rare C programmer in 20 years? I'll be the rare programmer who knows the ancient art of working with pointers?