Infiniband - You Gotta Have Pretty Good Game

A long time ago I saw a t-shirt from an Infiniband company that said the following

"Infinband - Fast, Cheap, Easy"

my thought process when first reading this was:

• Fast - Yup, it's fast.  Definitely faster than ethernet.
• Cheap - Yeah, way cheaper than ethernet.
• Easy - Ummm, no, you gotta have pretty good game ... and a wingman would help.

Now, I'm sure there are some market analysts at Cisco, Intel, IDC, etc. that did some fancy market analysis to figure out why Infiniband did not grow at the rates people predicted.  My guess has always been that it was because Infiniband isn't easy.  It's just way too different than ethernet, leading many institutions to not bother with it because it wasn't worth the hassle of learning, retraining, coming up to speed, etc.

I sometimes like to think of the issue with an Infiniband expert trying to explain Infiniband to a knowledgeable ethernet user. 

Ethernet Guy: So I installed all the hardware, loaded the drivers, but nothing is working.
Infiniband Expert:  Did you run the subnet manager? The subnet manager sets up and routes the fabric.
Ethernet Guy: Is that an option on the switch?
Infiniband Expert: Maybe, it's a piece of software that may run on the switch or a server. 
Ethernet Guy: Where is it on my fabric?
Infiniband Guy: On your network, it's a daemon running on a server.
Ethernet Guy: Ugh, but server configuration is handled by a different group.

or

Ethernet Guy: What are GUIDs in Infiniband?
Infiniband Expert: The GUIDs in Infiniband are like MAC addresses.  They are NIC specific identifiers.
Ethernet Guy: Ok, then what's a LID.
Infinitude Expert: A LID is sort of like an IP address.  It's the software based identifier for a port.
Ethernet Guy: So why does my Infiniband NIC have a LID and an IP address?
Infiniband Expert: You get the IP address from IP over IB.  It's a separate driver.
Ethernet Guy: So I need to load 2 drivers for one NIC?
Infiniband Expert: Yup
Ethernet Guy: So how do I see the LID for my NIC.
Infiniband Expert: You can use one of many tools, like ibstat.
Ethernet Guy: Why isn't it in ifconfig?

I could go on and on, but the point is it's so different that the learning curve is quite steep.  There's new ways to debug problems, new ways to route, new advanced configuration, new tools to learn, etc.  For the majority of institutions, the performance gain of Infiniband must be immensely superior to justify the cost for retraining, transition, inefficiency, maintenance, etc.

How many institutions found Infiniband to be "immensely superior" for their needs?  It seems to be not many.  I can imagine this conversation happening in many companies:

Manager: Hey, Engineer can you take a look at this Infiniband thing.   The sales people say it's super fast for the price.
Engineer: Sure thing, I'll play with it.
<1 week later>
Manager: So how is Infiniband?
Engineer: I can't figure any of this out.

and that's the end of Infiniband at that company.

Intel acquires Qlogic's Infiniband Assets - End of Infiniband?

So today, it was announced that Intel had acquired Qlogic's Infiniband assets for $125M.  My immediate reaction was, "uh oh, is the age of Infiniband in HPC over?"

Why would this be the end of Infiniband?  Here's my analysis and thinking on the topic.

In 2005 there were five early players in the Infiniband market worth mentioning: Topspin, Voltaire, Mellanox, Pathscale, and Silverstorm.

• Topspin was acquired by Cisco in 2005 for $250M. Cisco shut down their Infiniband R&D in 2009.
• Silverstorm and Pathscale were acquired by Qlogic in 2006 for $60M and $109M respectively (total $169M). Pathscale's compilers were sold for undisclosed amounts (or atleast I can't find the number online). Given they are undisclosed numbers, its unlikely the numbers were big.  So Qlogic likely couldn't sell their Infiniband assets for even the price they paid for them.
• Mellanox acquired Voltaire in 2010 for $218M.

So that takes us to the beginning of 2011. What was once 5 major players (or 4 if you count the merged Silverstorm/Pathscale as 1) became 2. The growth of Infiniband as a data center solution to replace Ethernet did not appear to be happening.  Naturally, consolidation occurred.

So no worries yet for Infiniband, there were still 2 major players left.  Well lets look at the financials.

• Mellanox has been profitable for awhile.  Last year (2010) they profited $13M on $154M in sales.  Analysts say that Mellanox had huge sales this year at $258M.  Going off old income statements, $50-$60M of that is from Voltaire, so that's some decent growth.  Of course, a non-trivial portion of this profit is not from Infiniband, but from Mellanox's Ethernet sales.  How much?  Unfortunately I can't find breakdowns.
• I couldn't find breakdowns in revenue/profit for Qlogic, but given the sale of their Infiniband divisions was for $125M, it indicates it wasn't much (Qlogic had a market cap of $1.6B starting today).
• As far as I can tell from data sheets, Voltaire never had a single profitable year.

I'm not some fancy financial analyst, but what I derive from this is that the total profitability of Infiniband is somewhere on the order of $25-$50 million globally. It's enough for there to be businesses out there and enough for there to be a market, but it's not a lot.

So the big question is, why did Intel buy Qlogic's Infiniband assets?

1. Were they interested in the ~$5M profit that Qlogic's Infiniband assets could net them?  I doubt it.  (I derived the ballpark $5M because this article puts Mellanox as owning about 85% of the Infiniband market.)
2. Perhaps Intel thinks they can do some bundling to increase the profitability of Infiniband.  Hypothetically, put them on Intel motherboards.  It's certainly possible.  But how much gain can they really get for a market that appears to not be interested in Infiniband?  Turn the $5M into $20-$30M in a few years?  It seems hardly worth it for an Intel.
3. Intel thinks they can turn Infiniband around as a data center/HPC solution and make it far more popular.  If this were 2005, I would be willing to believe it.  I think the lack of wider adoption of Infiniband is a bit cemented.  Newer/better Ethernet solutions are now catching up too, so it's not the same market as 2005.
4. Support Infiniband as a community service.  With Mellanox having 85% of the market, there was a decent chance Qlogic's Infiniband could eventually sink.  Without a competitor and decent prices, the HPC community could buy less Intel chips.  There is a good argument for this, although I think the odds of this are low.  Intel could completely ignore the HPC community and they would still buy tons of their chips.  Perhaps less overall, but is it enough of a difference for Intel to do a $125M community service for them?
5. This is an aqhire move, designed to give Intel the talent necessary to make the HPC networking product they really want to make.  While it could be based on Infiniband, it's unlikely to be standard Infiniband or standardized as Infiniband. There are very few companies/groups out there that know how to make HPC networking equipment, and the Infiniband group at Qlogic is one of them.  They could have bought Cray, but they'd be buying a lot of software assets they probably weren't interested in

My gut tells me its #5 above.  As an additional argument for this point, in 2011  Intel hired away a ton of HPC talent including Chief Architect of Blue Gene from IBM.  Did they hire a whole bunch of these elite HPC executives and architects to help push commodity Intel chips and hardware like they have in the past?  I doubt it.

So how is this the end for Infiniband?  Well, if my guess above occurs, you'll only have Mellanox as the player in the Infiniband market.  While I have respect for Mellanox, I have a hard time believing they are going to care about standardizing their hardware or pushing changes to standards groups if they are the only ones manufacturing it.  Eventually, Infiniband would become synonymous with whatever Mellanox produces, and Infiniband itself will be gone.

Update (4/26/12):

Heh, from #5 above:

They could have bought Cray, but they'd be buying a lot of software assets they probably weren't interested in 

and what do ya know, Intel bought Cray's interconnect assets earlier this week.

With that acquisition, that's a lot of networking HPC expertise to be buying up and absorbing.  Perhaps this enhances my argument that Intel is gathering forces to create a new HPC interconnect technology?

One colleague suggested that Intel might be trying to have a "portfolio" of different products.  It's certainly possible that they are, but it doesn't seem like something they would want to do.  Having a portfolio of products is more up the alley of an HP or an IBM.  It'll be interesting to see what Intel does, but the full manifestation of this will probably not be seen for years.

Definition of Poll

So I was talking with a colleague the other day about my initial confusion over the function ibv_poll_cq() in Infiniband verbs. I initially thought/assumed that the function operated similarly to the poll() syscall and was a tad confused on where/how a timeout could be specified to ibv_poll_cq(). As I worked through things, I later realized the ibv_poll_cq() does not iterate, sleep, or wait for an event to occur. It returns immediately if any events are ready or not. It's similar in action to poll() with a timeout of 0.

My colleague and I then got into a discussion of the definition of "poll". My colleague comes from a bit more of a hardware background and considered "poll" to mean a singular check of a state (such as in a register). I on the other hand, perhaps coming from more of a generic systems background, had always considered "poll" to imply both the check of state and some waiting system with it (may it be via sleep, busy-wait, etc.). I asked my colleague how he would say he wants to check a state multiple times? He said, "That's a poll loop."

Another way to think of it, is suppose I had some generic function like:

poll_for_event()

Would the initial assumption be that this function blocks or does not block?

So I was curious about the definition and how it is used in the broader community. I started with an assumption that the term "poll" in Computer Science comes from "poll" in relation to voting. According to Websters the definition of poll is:

"a sampling or collection of opinions on a subject, taken from either a selected or a random group of persons, as for the purpose of analysis."

The question is, can "poll" be applied to a single individual or must it be applied to a group? For example, can you say, "I want to poll a person about this vote?" Naturally, I have no background in linguistics or philology, so I'm just going on a guess here. When speaking of "poll" and voting, I think it is synonymous with gathering the opinions of multiple people and the singular case makes no sense. Otherwise, a "poll" accomplishes nothing towards determing majority vote/opinion. So in my opinion, this might lend the definition of "poll" to include the waiting/loop/whatever instead of a single status check.

Now, "poll" can be twisted when moved into a different field (such as a "bug"). When I initially thought of the syscall poll(), I was thinking predominantly about how it returns to the user after a specified timeout and returns event statuses. However, perhaps it is not named "poll" for this reason. Perhaps it is named "poll" due to its management of multiple file descriptors. In some respect, poll() gathers the "opinions" of each of the file descriptors and records them. So it sort of matches the classic voting definition of poll above.

But this doesn't help our discussion. Lets go back to the case where we're talking about "polling" a state or status. Wikipedia says:

"Polling, or polled operation, in computer science, refers to actively sampling the status of an external device by a client program as a synchronous activity."

and

"Polling is sometimes used synonymously with busy-wait polling (busy waiting)."

Here, it is implied that "poll" implies active multiple-checking and not a singular check of a status. However, I notice that in the above definitions from Wikipedia it is "polling" and "polled", but not "poll". In fact, as I look around the web, virtually every definition or description of "polling" in reference to this topic is listed as "polling". With rare occasion is this topic discussed using "poll" or "to poll" or "polls" (discounting the situations where it appears people are referencing poll() specifically).

So does "poll" refer to a singular check or an active sampling? In all liklihood different niches of the technical community eventually came up with their own definitions/meanings. It's one of those funny things that just happens (Perhaps I'll write someday of the confusion I've had with colleagues over magic numbers). I bet it comes down to common programming scenarios and expectations. Lets go back to my imaginary function:

poll_for_event()

In some communities, such as network programmers or GUI programmers, the assumption might be that the function blocks. The reasoning is simple. Under most circumstances, there's not much to do until the function returns (e.g. http request arrives, GUI item is clicked). So a person in this community who says, "poll", you are likely to believe that you wait until an event occurs.

Then perhaps you have people in the kernel community, where you have many other things you would rather be doing if an event isn't ready. So the natural assumption is the function won't block.

Mellanox acquires Voltaire

So I learned a few days ago that Mellanox acquired Voltaire.

http://www.networkcomputing.com/data-networking-management/mellanox-acquires-voltaire-and-creates-an-infiniband-duopoly.php

This wasn't too much of a surprise, somebody was going to pick them up at some point.  The question is, will someone now go after acquiring Mellanox?  A friend of mine at work noted that Oracle already owns about 10% of Mellanox.