The Cathedral & The Bazaar - over 15 years later

I recently finished The Cathedral & The Bazaar, a relatively famous collection of essays from Eric S. Raymond.

I think it's pretty interesting reading the book in 2016, about 17 years after it was first published (and probably closer to 20 years after it was first written online).  A few thoughts as I was reading through it.

Open Source vs Closed Source - Do most end customers even care?

Since the book was written, many more open source vendors now exist.  Some that come to mind are Redhat, Suse, Cloudera, and Hortonworks.  Some of these vendors distribute software that is only based on open source.  Other vendors distribute software that is a mixture of open source and closed source software.  The same can be said of several large vendors, such as IBM and Oracle.

I actually wonder if most end customers can even tell what is open source and what is closed source from a vendor without looking into it a bit.  Most may know the Linux kernel itself is open source, but do they know which kernel drivers are open vs closed source?

Ultimately, if it's not 100% clear what is open vs closed, I wonder if most customers even care.  At the end of the day, it's just a software offering from a vendor, the fact it may be "open source" may not matter to most.

Open Source - No Longer about Gift Culture and Reputation?

In the book the "gift culture" of open source and the desire for "reputation" in the open source world are discussed a lot.

However, over the last 15-20 years open source has become pervasive and the technology world has changed enough that IMO there are other reasons people now to open source.

With the emergence of more free code repository sites (most notably Github in recent years, sourceforge in the past), I believe a number of people open source just because they get a free code repository for their needs.  It's much easier than setting one up for yourself.  In addition you get some issue tracking software and documentation/wiki software along with it for free.  The huge number of "scratch" project repos on Github are probably a testament to this.

For many companies that support open source as part of their product offerings, it's simply a part of their business now.  Software engineers do open source as just part of their jobs, not necessarily for reputation or "gift" culture.

For many companies, it's a recruiting tool.  I doubt that many companies with a corporate presence actually "care" about open source more than just an ends to a mean (i.e. hiring people).

How to become a hacker - same languages?

In the Appendix, Raymond speaks of the programming languages that people should learn to be successful "hackers" in the open source world.  The languages he recommends are Python, Java, C/C++, Perl, and Lisp.

I was a little surprised to see Python in that list given this was written in 1999.  I didn't think Python was as popular back then as it apparently was.  I didn't think Python began to take off until some time in the 2000s.

But other than that, I find it interesting that the list is still quite accurate.  These are probably still the right languages to learn.

I'm really good at fixing typos!

I have a joke amongst colleagues that I am "only good at fixing typos in open source projects."

Why?

Whenever I submit a code patch for some bug and/or feature in a open source project, it can often go ignored.  Why?  Because project maintainers are busy and they don't have time to look at every patch that comes in.  If the patch solves some specific problem they are looking at, then you're in luck.  But more than often I'm fixing something that isn't on their current radar.  Eventually, the patch gets lost amongst the other patches in their review pile and is forgotten.

Why do I know that patches will be forgotten or ignored for long periods of time?  It's because I do the same thing! :-)  It's normal and reality with open source projects.

But what happens when I submit a patch that just fixes a typo in comments or a typo in documentation?  This takes almost no effort to look through and review, so the patch is accepted immediately.

I do the same thing in projects I maintain.

As time goes on, there are a few projects that I suddenly feel like the only thing I can contribute is typo fixes.

Thus, I'm really good at fixing typos :-)

c'est la vie

Moving from Spark 0.9.1 to Spark 1.0.0

I recently had to support Spark 1.0.0 in a project (Magpie).

The conversion from Spark 0.9.1 to Spark 1.0.0 was a bit annoying, as many changes had happened.

Here are a list of changes that I thought were worth mentioning.  Hope what I say here can help others.

1) Running examples differences

In Spark 0.9.1, you would run one of the Spark examples (such as SparkPi) like this:

> bin/run-example org.apache.spark.examples.SparkPi spark://SPARKMASTER:7077 
 
In Spark 1.0.0, running examples through run-example requires the Spark master to be specified through the MASTER environment variable and not on the command line.  So for Spark 1.0.0, you'll want to do something like this instead:

> export MASTER="spark://SPARKMASTER:7077"
> bin/run-example org.apache.spark.examples.SparkPi

otherwise you'll get a bad input error/exception.

If you don't set the MASTER environment variable, run-example will assume you want to run the example locally.

Note that setting the MASTER environment variable is specific to the run-example script.  It won't pick up the default value from the new spark-defaults.conf file.

2) spark-submit script

The spark-submit is a new wrapper script for submitting Spark jobs.  Although you can still use spark-class directly, this is the primary job submission script. It has the following usage:

> bin/spark-submit --class JOBCLASSTORUN [spark-submit options] APPLICATIONJAR [application args]
Note that the job jar is now passed in on the command line, unlike before with spark-class.

There's all sorts of new options in spark-submit.  Here are some options of note taken from the --help output:

  --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
  --class CLASS_NAME          Your application's main class (for Java / Scala apps).
  --jars JARS                 Comma-separated list of local jars to include on the driver
                              and executor classpaths.
  --properties-file FILE      Path to a file from which to load extra properties. If not
                              specified, this will look for conf/spark-defaults.conf.
  --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 512M).
  --driver-java-options       Extra Java options to pass to the driver.
  --driver-library-path       Extra library path entries to pass to the driver.
  --driver-class-path         Extra class path entries to pass to the driver. Note that
                              jars added with --jars are automatically included in the
                              classpath.

But you probably won't end up using these, you'll more likely use ...

3) spark-defaults.conf

Previously, options were configured through SPARK_JAVA_OPTS, but that is now deprecated. Everything should now be done through the spark-defaults.conf file. It is read and loaded from spark-submit when you submit a job. By default it is read in conf/spark-defaults.conf but that can be altered using the --properties-file option. In addition, settings through SPARK_CLASSPATH or SPARK_LIBRARY_PATH should now be set through spark-defaults.conf as well.

Here are several options for spark-defaults.conf of particular note, with the full list in the Spark documentation.

spark.master                     set Spark master, e.g. spark://SPARKMASTER:7077
spark.executor.memory            set executor memory, e.g. 1024m, see more below
spark.executor.extraClassPath    what used to be set by SPARK_CLASSPATH
spark.executor.extraLibraryPath  what used to be set by SPARK_LIBRARY_PATH 

4) deprecated SPARK_MEM environment variable

The SPARK_MEM environment variable has been deprecated and replaced by two new configurations so users can configure the Spark executors and driver memory separately..

The SPARK_DRIVER_MEMORY environment will set memory for your Spark driver.  This could also be handled via the --driver-memory option in spark-submit.

The memory for Spark executors is now handled by the spark.executor.memory option in spark-defaults.conf.  Documentation indicates the environment variable SPARK_EXECUTOR_MEMORY will also work, but I didn't try that.

5) deprecated spark.local.dir

The configuration option spark.local.dir is now apparently deprecated in favor of the SPARK_LOCAL_DIRS environment variable.

Why bother to have specifications ...

Ugh ... a very annoying e-mail thread from earlier this week.  For the record, I'm not attacking the other fellow on the other end of this thread.  They're just doing their job too.

Company: We noticed in your open-source software that you output X incorrectly.  Please look at this other open-source software Y to see how it should be output.

Me: Please look at standard Z.  According to standard Z, I'm doing things correctly.  Here's the code snippets to show it.

Company: Hmm, you're right.  I guess a number of vendors are not properly sending the data in the right format.  Unfortunately, we can't get all those vendors to change the format.

Me: Well, how about I add a workaround option on the command line.  Those who are knowledgeable of this subject matter can specify it if they want to.

Company: We tried out your workaround option and it's almost correct.  A few bytes were flipped in the output.

Me: Huh, that's strange.  The first few fields are sent little-endian, but the latter few fields are sent big-endian.  This is really weird.

Company: For legacy reasons, it appears large company A has been doing this, so I guess vendors have followed suit.

Vendor Communication Frustrations

It's common for me to find firmware bugs on vendor hardware that interacts poorly with software written in house.  My assumption is that vendors sell their software as a "value add", only test their hardware with their software, and have limited care for third party software.  Once in awhile the vendor fixes hardware issues in their software, so the issue is technically in the third party software, but it's not the third party's fault.

When I report a bug in the firmware, it's common for a vendor to respond with something akin to, "It works for our software, it must be your software."  It's a normal, yet frustrating, response.  In addition, to get the problem resolved often means getting through layers of support before reaching an engineer that understands the situation.

Over time, I've come up with a few mechanisms to handle this type of issue with vendors more quickly.  Here are some of them:

A) Add packet dumps or equivalently "advanced" debug information in the ticket.

I'll add information to the bug that is reasonably complex or difficult to understand.  The common example is something similar to a TCP packet dump.  I add something in the ticket with arrows (e.g. "See here --->") and something like "this 0x8 should be an 0x5".

Due to the complexity of the information, the ticket is typically passed up the food chain more quickly.  In addition, it sometimes does not matter what software was running to generate the TCP packet dump.  I can just say it was their software, they would never know.

Ironically, I sometimes have nicer/better debugging information (or even the flat out solution) I could put in the ticket.  However, I often elect to put the more confusing information in the ticket b/c it ultimately resolves the problem more quickly.

B) Be very specific in your requests

It's better to ask for very clear specific information.  A normal request might go like this: "I noticed an OEM piece of information on your motherboard.  Can you please describe what this means?"

While this is a perfectly reasonable request, it leads to the wrong answers.  A common answer might be, "Use our software X, it can show you all the information", which is of course not a useful response.  Or you might get "That information means FOO, you need to do BAR for your system", which might be true, but isn't what we're really looking for.

It's better to be very specific with your request, for example  "I noticed the following OEM piece of information: 0x20 0x18 0xA2.  What is the mapping of OEM hex to English.  0x20 = ???, 0x18 = ???, 0xA2 = ???."

C) Site standards/specifications that can't be challenged

I'll site standards/specifications, making it clear that if they wish to counter my claim, they will have to know what they are talking about and prove it to me.

For example, I might write, "Please see the following packet dump.  Clearly the X field violates section 1.2.A of the specification, second paragraph, third sentence."

This type of statement is so specific and advanced, it will typically skip lower support levels.

Intel acquires Qlogic's Infiniband Assets - End of Infiniband?

So today, it was announced that Intel had acquired Qlogic's Infiniband assets for $125M.  My immediate reaction was, "uh oh, is the age of Infiniband in HPC over?"

Why would this be the end of Infiniband?  Here's my analysis and thinking on the topic.

In 2005 there were five early players in the Infiniband market worth mentioning: Topspin, Voltaire, Mellanox, Pathscale, and Silverstorm.

• Topspin was acquired by Cisco in 2005 for $250M. Cisco shut down their Infiniband R&D in 2009.
• Silverstorm and Pathscale were acquired by Qlogic in 2006 for $60M and $109M respectively (total $169M). Pathscale's compilers were sold for undisclosed amounts (or atleast I can't find the number online). Given they are undisclosed numbers, its unlikely the numbers were big.  So Qlogic likely couldn't sell their Infiniband assets for even the price they paid for them.
• Mellanox acquired Voltaire in 2010 for $218M.

So that takes us to the beginning of 2011. What was once 5 major players (or 4 if you count the merged Silverstorm/Pathscale as 1) became 2. The growth of Infiniband as a data center solution to replace Ethernet did not appear to be happening.  Naturally, consolidation occurred.

So no worries yet for Infiniband, there were still 2 major players left.  Well lets look at the financials.

• Mellanox has been profitable for awhile.  Last year (2010) they profited $13M on $154M in sales.  Analysts say that Mellanox had huge sales this year at $258M.  Going off old income statements, $50-$60M of that is from Voltaire, so that's some decent growth.  Of course, a non-trivial portion of this profit is not from Infiniband, but from Mellanox's Ethernet sales.  How much?  Unfortunately I can't find breakdowns.
• I couldn't find breakdowns in revenue/profit for Qlogic, but given the sale of their Infiniband divisions was for $125M, it indicates it wasn't much (Qlogic had a market cap of $1.6B starting today).
• As far as I can tell from data sheets, Voltaire never had a single profitable year.

I'm not some fancy financial analyst, but what I derive from this is that the total profitability of Infiniband is somewhere on the order of $25-$50 million globally. It's enough for there to be businesses out there and enough for there to be a market, but it's not a lot.

So the big question is, why did Intel buy Qlogic's Infiniband assets?

1. Were they interested in the ~$5M profit that Qlogic's Infiniband assets could net them?  I doubt it.  (I derived the ballpark $5M because this article puts Mellanox as owning about 85% of the Infiniband market.)
2. Perhaps Intel thinks they can do some bundling to increase the profitability of Infiniband.  Hypothetically, put them on Intel motherboards.  It's certainly possible.  But how much gain can they really get for a market that appears to not be interested in Infiniband?  Turn the $5M into $20-$30M in a few years?  It seems hardly worth it for an Intel.
3. Intel thinks they can turn Infiniband around as a data center/HPC solution and make it far more popular.  If this were 2005, I would be willing to believe it.  I think the lack of wider adoption of Infiniband is a bit cemented.  Newer/better Ethernet solutions are now catching up too, so it's not the same market as 2005.
4. Support Infiniband as a community service.  With Mellanox having 85% of the market, there was a decent chance Qlogic's Infiniband could eventually sink.  Without a competitor and decent prices, the HPC community could buy less Intel chips.  There is a good argument for this, although I think the odds of this are low.  Intel could completely ignore the HPC community and they would still buy tons of their chips.  Perhaps less overall, but is it enough of a difference for Intel to do a $125M community service for them?
5. This is an aqhire move, designed to give Intel the talent necessary to make the HPC networking product they really want to make.  While it could be based on Infiniband, it's unlikely to be standard Infiniband or standardized as Infiniband. There are very few companies/groups out there that know how to make HPC networking equipment, and the Infiniband group at Qlogic is one of them.  They could have bought Cray, but they'd be buying a lot of software assets they probably weren't interested in

My gut tells me its #5 above.  As an additional argument for this point, in 2011  Intel hired away a ton of HPC talent including Chief Architect of Blue Gene from IBM.  Did they hire a whole bunch of these elite HPC executives and architects to help push commodity Intel chips and hardware like they have in the past?  I doubt it.

So how is this the end for Infiniband?  Well, if my guess above occurs, you'll only have Mellanox as the player in the Infiniband market.  While I have respect for Mellanox, I have a hard time believing they are going to care about standardizing their hardware or pushing changes to standards groups if they are the only ones manufacturing it.  Eventually, Infiniband would become synonymous with whatever Mellanox produces, and Infiniband itself will be gone.

Update (4/26/12):

Heh, from #5 above:

They could have bought Cray, but they'd be buying a lot of software assets they probably weren't interested in 

and what do ya know, Intel bought Cray's interconnect assets earlier this week.

With that acquisition, that's a lot of networking HPC expertise to be buying up and absorbing.  Perhaps this enhances my argument that Intel is gathering forces to create a new HPC interconnect technology?

One colleague suggested that Intel might be trying to have a "portfolio" of different products.  It's certainly possible that they are, but it doesn't seem like something they would want to do.  Having a portfolio of products is more up the alley of an HP or an IBM.  It'll be interesting to see what Intel does, but the full manifestation of this will probably not be seen for years.

Definition of Poll

So I was talking with a colleague the other day about my initial confusion over the function ibv_poll_cq() in Infiniband verbs. I initially thought/assumed that the function operated similarly to the poll() syscall and was a tad confused on where/how a timeout could be specified to ibv_poll_cq(). As I worked through things, I later realized the ibv_poll_cq() does not iterate, sleep, or wait for an event to occur. It returns immediately if any events are ready or not. It's similar in action to poll() with a timeout of 0.

My colleague and I then got into a discussion of the definition of "poll". My colleague comes from a bit more of a hardware background and considered "poll" to mean a singular check of a state (such as in a register). I on the other hand, perhaps coming from more of a generic systems background, had always considered "poll" to imply both the check of state and some waiting system with it (may it be via sleep, busy-wait, etc.). I asked my colleague how he would say he wants to check a state multiple times? He said, "That's a poll loop."

Another way to think of it, is suppose I had some generic function like:

poll_for_event()

Would the initial assumption be that this function blocks or does not block?

So I was curious about the definition and how it is used in the broader community. I started with an assumption that the term "poll" in Computer Science comes from "poll" in relation to voting. According to Websters the definition of poll is:

"a sampling or collection of opinions on a subject, taken from either a selected or a random group of persons, as for the purpose of analysis."

The question is, can "poll" be applied to a single individual or must it be applied to a group? For example, can you say, "I want to poll a person about this vote?" Naturally, I have no background in linguistics or philology, so I'm just going on a guess here. When speaking of "poll" and voting, I think it is synonymous with gathering the opinions of multiple people and the singular case makes no sense. Otherwise, a "poll" accomplishes nothing towards determing majority vote/opinion. So in my opinion, this might lend the definition of "poll" to include the waiting/loop/whatever instead of a single status check.

Now, "poll" can be twisted when moved into a different field (such as a "bug"). When I initially thought of the syscall poll(), I was thinking predominantly about how it returns to the user after a specified timeout and returns event statuses. However, perhaps it is not named "poll" for this reason. Perhaps it is named "poll" due to its management of multiple file descriptors. In some respect, poll() gathers the "opinions" of each of the file descriptors and records them. So it sort of matches the classic voting definition of poll above.

But this doesn't help our discussion. Lets go back to the case where we're talking about "polling" a state or status. Wikipedia says:

"Polling, or polled operation, in computer science, refers to actively sampling the status of an external device by a client program as a synchronous activity."

and

"Polling is sometimes used synonymously with busy-wait polling (busy waiting)."

Here, it is implied that "poll" implies active multiple-checking and not a singular check of a status. However, I notice that in the above definitions from Wikipedia it is "polling" and "polled", but not "poll". In fact, as I look around the web, virtually every definition or description of "polling" in reference to this topic is listed as "polling". With rare occasion is this topic discussed using "poll" or "to poll" or "polls" (discounting the situations where it appears people are referencing poll() specifically).

So does "poll" refer to a singular check or an active sampling? In all liklihood different niches of the technical community eventually came up with their own definitions/meanings. It's one of those funny things that just happens (Perhaps I'll write someday of the confusion I've had with colleagues over magic numbers). I bet it comes down to common programming scenarios and expectations. Lets go back to my imaginary function:

poll_for_event()

In some communities, such as network programmers or GUI programmers, the assumption might be that the function blocks. The reasoning is simple. Under most circumstances, there's not much to do until the function returns (e.g. http request arrives, GUI item is clicked). So a person in this community who says, "poll", you are likely to believe that you wait until an event occurs.

Then perhaps you have people in the kernel community, where you have many other things you would rather be doing if an event isn't ready. So the natural assumption is the function won't block.

gcc extensions ...

Despite doing C programming for so many years, once in awhile some extension or really weird code makes me go "Huh?" I learned about a ternary operator extension the other day. The ternary operator is normally:

d = a ? b : c

but gcc allows

z = x ? : y

When I saw this, my immediate reaction was, "Uhhh, will that even compile?" What this extension allows is for x to be returned when the condition is true. So it's pretty much identical to:

z = x ? x : y

However, if x is an expression, it is only evaluated once. So there are some circumstances it could be quite useful. I felt sort of dumb when I didn't know it, but I felt better when multiple other co-workers didn't know this extension :-)

Expectations When You Support Open Standards

I've spoken about this topic with many engineers, managers, sales folks, etc. So I decided it might be interesting to post about this topic and my experience with it.

Organizations/vendors often advertise support for software or protocols based on open standards. However, I've had many experiences where it does not appear vendors understand the consequences of that advertisement. The following is an example of something I've encountered in my work.

I'm the co-author and present maintainer of FreeIPMI. If you're not familiar with what IPMI is, you can read about it in the FreeIPMI FAQ. For this discussion, the most important part to understand about IPMI is that it is an open standard which you can download here.

The IPMI standard supports many things, one of which is the ability to read sensors off a motherboard. For example, it supports the ability to read CPU temperatures and report them back to the user in Celsius (e.g. 50 degrees Celsius). The types of sensors that could exist on a motherboard are very large and cannot always be handled by IPMI. So the IPMI specification supports the ability for vendors to add OEM extensions to read non-standard sensors. For example, I've encountered a motherboard where a CPU temperature sensor was not reported in Celsius, but instead reported by flags indicating "low", "medium", "high", or "overheat." This type of sensor is not defined by the IPMI specification and is specific to just one manufacturer's motherboard.

Normally, I (as potential motherboard customer) do not know how to read and interpret an OEM specific sensor. The question is, is it fair or unfair for a vendor to keep information about an OEM specific sensor closed/hidden/secret? Is it fair or unfair to only allow an OEM specific sensor to be readable by a vendor's software instead of third-party software?

In my opinion, it depends on the sensor. In the case of a OEM CPU temperature sensor above, I believe hiding this information to be unfair. A CPU temperature sensor is a standard IPMI supported mechanism. An OEM specific CPU temperature sensor is what I call a "minor permutation" from the specification. A user/customer of IPMI would expect they could read CPU temperature sensors using third-party IPMI software.

In contrast, suppose that a sensor on the motherboard supports the ability to read a sensor quite different than what is supported in the IPMI specification (i.e. it is not a "minor permutation"). In this case, I do not believe a user/customer would have reasonable expectation for third-party IPMI software to read the sensor. So in this case, I would consider it fair to keep the information closed/hidden and only readable by vendor software.

Now, you might ask, what are reasonable expectations of a customer/user? In my opinion, it comes down to how the product is advertised. Does the product brief say that you can read motherboard sensors via IPMI? Or does it say you can read them via software X? If a vendor says the former, they should be prepared to deal with questions from customers about third party software. If they say the later, I believe customers cannot have expectations of support for third-party software.

Why is this a conversation that always comes up for me? You may have guessed it by now, but many vendors do not wish to open up and share their OEM extensions with me for inclusion in FreeIPMI. As an example, the following are several IPMI standard mechanisms.

• Configure MAC Address
  
• Configure IP Address
  
• Get firmware version
  
• Get product name/serial number
  
• Set LED
  

Despite the fact that they are standard mechanisms, due to minor implementation differences, they have also been implemented as OEM extensions by some vendors (see ipmi-oem). The first two (configuring MAC and IP address) are particularly egregious OEM extensions, because the OEM extension (and possibly vendor specific software) are now required for basic setup and configuration of IPMI (let alone using all the features of IPMI). [1]

Hopefully, this illustrates the expectations a user/customer/software developer would have when a vendor supports an open standard. As an addendum, I do not wish to imply that companies with IPMI OEM extensions are being evil by keeping some of their OEM extensions hidden/closed/secret. Many companies I have worked with on IPMI have been very gracious in (eventually) opening up their OEM extensions for addition into FreeIPMI (see FAQ). I believe that initial resistance from vendors is often due to poor communication rather than evil intent. For many vendors, I am a strange corner case in their customer base. Support staff folks just aren't prepared to handle the kinds of support requests I make, so their initial reaction is naturally "No, we can't do that."

[1] - As an interesting aside, I once reported to a vendor that their standard IPMI MAC address configuration functionality was broken in their firmware. The vendor responded that it was working with their software, and the bug must of have been in FreeIPMI. After much back and forth, I was able to determine that their vendor specific software was configuring the MAC address using an OEM extension, not the standard mechanism. So in this particular case, the vendor support staff was completely unaware that their software used an OEM extension. Ugh ...