There's been discussion out there about comparing the HDFS filesystem to a traditional parallel filesystem like Lustre. The problem is it's really difficult to compare apples to apples.
As an example, I saw a white paper awhile back (sorry, I can't find it online) that compared HDFS to Lustre. HDFS beat Lustre in this person's performance tests by a good margin. After digging into the paper I saw why. This fellow ran Lustre over a 1 GigE ethernet network.
Is this a fair test? On the one hand it isn't because Lustre is a network based filesystem. If you simple choose to bottleneck Lustre, of course it will lose. On the other hand, it's a fair test, because it uses the same hardware most use with HDFS.
So lets say we replaced the GigE with Infiniband. Would it now be a fair test? Perhaps its slightly fairer, but HDFS people can say HDFS wasn't designed for more expensive hardware and therefore doesn't take advantage of it. In the case of Infiniband, HDFS isn't using RDMA during replication.
I don't know the right comparison. However, HDFS vs Lustre may not be the correct comparison to think about. At the end of the day, I could probably concoct an HDFS setup that will always beat a Lustre setup and vice versa.
I believe thinking about this as HDFS vs Lustre isn't the right approach. It's really Hadoop Cluster vs HPC Cluster. At the end of the day, while Hadoop is famous for handling large data, the reality is because of shuffle/sorting/scheduling/etc. in Hadoop, it also reads/writes tons of small files. The memory for a Hadoop Cluster vs HPC cluster may also be different. That affects spilling of data, page cache, etc.
Update: See "Big Data vs HPC" follow up.
Update: See "HPC vs Big Data" follow up.
Update 6/2/15:
Not so long ago I was talking to someone about the HDFS vs Lustre comparison.
Many people have done HDFS vs "Some Networked Filesystem" experiments.
However, I think these experiments are inherently flawed. The experiments always look something like this.
Datanodes
8 nodes
4 SATA disks
8 core
32G RAM
Networked Storage
4 nodes
8 SATA disks
8 core each
32G RAM
with additional hardware details beyond this.
The comparison will be HDFS using the Datanodes for data & map reduce. Then it'll be a comparison to the Networked Storage, also using the Datanodes as the computation facility.
Do you see the inherent problem in the above comparison?
.
.
It's staring you right in the eyes.
.
.
.
It's an 8 node test vs a 12 node test.
It's a 256G RAM test vs a 384G RAM test.
This isn't to say that the comparison is poorly done. But this is part of the inherent problem of comparing HDFS vs Networked File systems. What is a fair comparison?
Thursday, February 13, 2014
Wednesday, February 12, 2014
Big Data vs. HPC/Supercomputing
There's been a lot of articles about what is "Big Data" and how does it compare to traditional Supercomputing and High Performance Computing. I thought about it, and devolved it into a simple mathematical statement.
In Supercomputing / HPC
Computation Time >> IO Time
and in Big Data
IO Time >> Computation Time
The architecture of the hardware, the networking solutions, the software you use, how you design your software, etc. etc. is centered around this simple statement.
In Supercomputing / HPC
Computation Time >> IO Time
and in Big Data
IO Time >> Computation Time
The architecture of the hardware, the networking solutions, the software you use, how you design your software, etc. etc. is centered around this simple statement.
Monday, January 20, 2014
High performance through strange correlations
Awhile back I learned of a really interesting statistic in Malcolm Gladwell's book Outliers.
In some standardized tests, test administrators like to ask students survey questions before the test so they can try and correlate test performance to other social factors. These questions are pretty normal, things about your social life, friends, and family. However, this survey is so long and tedious that many students just give up and don't even complete the survey before taking the test.
What several researchers found was that there is a strong correlation between percent of the survey completed and performance on the test. Students that complete a higher percentage of the survey perform better on the test. Note that it's not how they answered the questions on the survey, simply the fact that bothered to complete to the survey.
The suggestion is that patience, diligence, and work ethic are actually more important than smarts. If you can stomach through this tedious survey, you've probably stomached through a lot of homework assignments to be able to learn the material well.
Not so long ago, I was on a committee to review a contract for a procurement. While reviewing the bids, I noticed that there were bids that were horrifically bad. At the same time I noticed the bad bids were much shorter in length than the better bids. In fact, by the time the committee was done with the selection process, the ranked order of the bids was strongly correlated to the thickness of the printed bids. My recollection is that if only two bids had been flipped in their ranking, the correlation between thickness of printed bids & rank would have been perfect.
It got me thinking, the correlation was similar to the correlation with the standardized tests. The companies that put in way more effort and time into their bids, probably cared about their bid a lot more. At the end of the day, the fact that they cared about the bid a lot more, probably meant they wanted to win the contract a lot more, and meant they would care about fulfilling the contract a lot more.
In some standardized tests, test administrators like to ask students survey questions before the test so they can try and correlate test performance to other social factors. These questions are pretty normal, things about your social life, friends, and family. However, this survey is so long and tedious that many students just give up and don't even complete the survey before taking the test.
What several researchers found was that there is a strong correlation between percent of the survey completed and performance on the test. Students that complete a higher percentage of the survey perform better on the test. Note that it's not how they answered the questions on the survey, simply the fact that bothered to complete to the survey.
The suggestion is that patience, diligence, and work ethic are actually more important than smarts. If you can stomach through this tedious survey, you've probably stomached through a lot of homework assignments to be able to learn the material well.
Not so long ago, I was on a committee to review a contract for a procurement. While reviewing the bids, I noticed that there were bids that were horrifically bad. At the same time I noticed the bad bids were much shorter in length than the better bids. In fact, by the time the committee was done with the selection process, the ranked order of the bids was strongly correlated to the thickness of the printed bids. My recollection is that if only two bids had been flipped in their ranking, the correlation between thickness of printed bids & rank would have been perfect.
It got me thinking, the correlation was similar to the correlation with the standardized tests. The companies that put in way more effort and time into their bids, probably cared about their bid a lot more. At the end of the day, the fact that they cared about the bid a lot more, probably meant they wanted to win the contract a lot more, and meant they would care about fulfilling the contract a lot more.
Sunday, January 12, 2014
Would you trade 5 first round draft picks for 14 years of Hall of Famers?
Back in 1990, the St. Louis Blues signed future Hall of Fame defensemen Scott Stevens as a free agent. As compensation to the Washington Capitals, the Blues had to give up 5 first round draft picks to the Capitals.
In 1991, the Blues signed future Hall of Fame forward Brendan Shanahan. As compensation, the New Jersey Devils demanded Scott Stevens and won him in arbitration.
In 1995, Brendan Shanahan was traded for future Norris trophy winner and MVP Chris Pronger. While not a Hall of Famer yet, he's considered a strong candidate for the Hall of Fame. (Edit: He was inducted into the Hall of Fame in 2015).
So if you're following along, effectively the Blues give up five first round draft picks for Stevens, which led to Shanahan (although unintentionally), and then led to Pronger.
If Pronger indeed makes it to the Hall of Fame, the Blues would have effectively given up five first round draft picks to get 14 years out of three future Hall of Famers.
The Blues were incredibly smart or (likely) incredibly lucky that these string of signings and trades could net them three future Hall of Famers.
Was it worth it? Well, on the one hand, many first round draft picks end up as busts. While some of those first round draft picks led to quality players like Sergei Gonchar, it didn't net the Capitals any significant progress towards a Stanley Cup. The three players weren't able to get the Blues a Stanley Cup either, or even a trip to the Conference championships.
Many teams have done way worse with their first round draft picks. Getting 14 years of Hall of Famer performance is probably way better than you could do on average with draft picks.
In 1991, the Blues signed future Hall of Fame forward Brendan Shanahan. As compensation, the New Jersey Devils demanded Scott Stevens and won him in arbitration.
In 1995, Brendan Shanahan was traded for future Norris trophy winner and MVP Chris Pronger. While not a Hall of Famer yet, he's considered a strong candidate for the Hall of Fame. (Edit: He was inducted into the Hall of Fame in 2015).
So if you're following along, effectively the Blues give up five first round draft picks for Stevens, which led to Shanahan (although unintentionally), and then led to Pronger.
If Pronger indeed makes it to the Hall of Fame, the Blues would have effectively given up five first round draft picks to get 14 years out of three future Hall of Famers.
The Blues were incredibly smart or (likely) incredibly lucky that these string of signings and trades could net them three future Hall of Famers.
Was it worth it? Well, on the one hand, many first round draft picks end up as busts. While some of those first round draft picks led to quality players like Sergei Gonchar, it didn't net the Capitals any significant progress towards a Stanley Cup. The three players weren't able to get the Blues a Stanley Cup either, or even a trip to the Conference championships.
Many teams have done way worse with their first round draft picks. Getting 14 years of Hall of Famer performance is probably way better than you could do on average with draft picks.
Friday, December 13, 2013
Why bother to have specifications ...
Ugh ... a very annoying e-mail thread from earlier this week. For the record, I'm not attacking the other fellow on the other end of this thread. They're just doing their job too.
Company: We noticed in your open-source software that you output X incorrectly. Please look at this other open-source software Y to see how it should be output.
Me: Please look at standard Z. According to standard Z, I'm doing things correctly. Here's the code snippets to show it.
Company: Hmm, you're right. I guess a number of vendors are not properly sending the data in the right format. Unfortunately, we can't get all those vendors to change the format.
Me: Well, how about I add a workaround option on the command line. Those who are knowledgeable of this subject matter can specify it if they want to.
Company: We tried out your workaround option and it's almost correct. A few bytes were flipped in the output.
Me: Huh, that's strange. The first few fields are sent little-endian, but the latter few fields are sent big-endian. This is really weird.
Company: For legacy reasons, it appears large company A has been doing this, so I guess vendors have followed suit.
Company: We noticed in your open-source software that you output X incorrectly. Please look at this other open-source software Y to see how it should be output.
Me: Please look at standard Z. According to standard Z, I'm doing things correctly. Here's the code snippets to show it.
Company: Hmm, you're right. I guess a number of vendors are not properly sending the data in the right format. Unfortunately, we can't get all those vendors to change the format.
Me: Well, how about I add a workaround option on the command line. Those who are knowledgeable of this subject matter can specify it if they want to.
Company: We tried out your workaround option and it's almost correct. A few bytes were flipped in the output.
Me: Huh, that's strange. The first few fields are sent little-endian, but the latter few fields are sent big-endian. This is really weird.
Company: For legacy reasons, it appears large company A has been doing this, so I guess vendors have followed suit.
Monday, November 25, 2013
Cardinals 2014 Off Season Moves Analysis
This is a follow up to my prior blog post "Cardinals Questions for 2014".
The Cardinals made two big moves that pretty much answer all of their off-season questions.
1) Who plays right field next year?
2) How do you handle David Freese, Matt Carpenter, and prospect Kolten Wong?
The Cardinals solved these two problems with one trade. They traded David Freese & Fernando Salas for Peter Bourjos, Randal Grichuk.
I love this trade. It gives the Cardinals so many options.
First, it allows the Cardinals to move Allen Craig to right field and put Matt Adams at first base.
Second, it moves Matt Carpenter to third and gives Kolten Wong the starting job at second base.
With Allen Craig in right field, the Cardinals were concerned with their outfield defense. They improved it greatly by getting Peter Bourjos. When healthy, he is a gold glove calibre centerfielder.
Bourjos' offense is questionable, however, he's a right handed hitter. In certain games the Cardinals can platoon Bourjos with Jon Jay, who is left handed.
In addition, Jon Jay can play right field on certain days. Jon Jay would be a huge upgrade in defense in right field over Allen Craig.
So the Cardinals improve the defense of centerfield, third base, and second base with this move. The defense in right field presumably goes down (Allen Craig < Carlos Beltran), but it's an overall big win.
In addition, it was an incredibly smart trade. The Cardinals traded a 30 year old third baseman who appeared to be in the decline on offense and defense, and only had two years of control left. They also gave up a relief pitcher who had already been demoted back to AAA.
In return they got a 26 year old centerfielder with three years of control left and potential to grow. They also got a nice mid-tier prospect to refill their minor league system with.
The only risk the Cardinals take is that Bourjos does have injury history. That is the risk they take, however the overall risk/reward appears to be favorable to the Cardinals.
3) Can the Cardinals upgrade shortstop?
The Cardinals solved this by signing Jhonny Peralta to a four year $52M contract.
While I personally preferred the signing of Stephen Drew, the signing is a decent one. It's a bit of an overpay, but not by a ridiculous amount. The Cardinals had Carlos Beltran, Chris Carpenter, Rafael Furcal, and Jake Westbrook's contracts coming off the books. So they had more than enough money to make a push for Peralta. They had enough money to even stretch the contract a little bigger than they'd be comfortable with. A 3 year contract would would have been better, but it's not a crazy stretch.
There are natural concerns over how Peralta will fair after his PED suspension last year. Regardless, his offense will be a huge upgrade over Pete Kozma in 2013. Even if Peralta regresses to his 2012 form (.239/.305/.384) it's still a good upgrade over Kozma in 2013 (.217/.275/.273).
Overall:
A great thing about these offseason moves (so far) is that the Cardinals didn't give up a single high value prospect. They kept Taveras, Wong, Wacha, Miller, Rosenthal, Kelly, Lynn, Siegrist, Martinez, Piscotty, etc
In addition, by not signing Stephen Drew (who had received a qualifying offer from Boston) the Cardinals retain their first round draft pick in 2014. With the Carlos Beltran signing, they'll get another first round draft pick in 2014 too.
Overall, a great set of moves that improves the offense and defense of the team, keeps the depth of the minor league system strong, doesn't overburden the team with a huge contract, and keeps draft picks for the future.
The Cardinals made two big moves that pretty much answer all of their off-season questions.
1) Who plays right field next year?
2) How do you handle David Freese, Matt Carpenter, and prospect Kolten Wong?
The Cardinals solved these two problems with one trade. They traded David Freese & Fernando Salas for Peter Bourjos, Randal Grichuk.
I love this trade. It gives the Cardinals so many options.
First, it allows the Cardinals to move Allen Craig to right field and put Matt Adams at first base.
Second, it moves Matt Carpenter to third and gives Kolten Wong the starting job at second base.
With Allen Craig in right field, the Cardinals were concerned with their outfield defense. They improved it greatly by getting Peter Bourjos. When healthy, he is a gold glove calibre centerfielder.
Bourjos' offense is questionable, however, he's a right handed hitter. In certain games the Cardinals can platoon Bourjos with Jon Jay, who is left handed.
In addition, Jon Jay can play right field on certain days. Jon Jay would be a huge upgrade in defense in right field over Allen Craig.
So the Cardinals improve the defense of centerfield, third base, and second base with this move. The defense in right field presumably goes down (Allen Craig < Carlos Beltran), but it's an overall big win.
In addition, it was an incredibly smart trade. The Cardinals traded a 30 year old third baseman who appeared to be in the decline on offense and defense, and only had two years of control left. They also gave up a relief pitcher who had already been demoted back to AAA.
In return they got a 26 year old centerfielder with three years of control left and potential to grow. They also got a nice mid-tier prospect to refill their minor league system with.
The only risk the Cardinals take is that Bourjos does have injury history. That is the risk they take, however the overall risk/reward appears to be favorable to the Cardinals.
3) Can the Cardinals upgrade shortstop?
The Cardinals solved this by signing Jhonny Peralta to a four year $52M contract.
While I personally preferred the signing of Stephen Drew, the signing is a decent one. It's a bit of an overpay, but not by a ridiculous amount. The Cardinals had Carlos Beltran, Chris Carpenter, Rafael Furcal, and Jake Westbrook's contracts coming off the books. So they had more than enough money to make a push for Peralta. They had enough money to even stretch the contract a little bigger than they'd be comfortable with. A 3 year contract would would have been better, but it's not a crazy stretch.
There are natural concerns over how Peralta will fair after his PED suspension last year. Regardless, his offense will be a huge upgrade over Pete Kozma in 2013. Even if Peralta regresses to his 2012 form (.239/.305/.384) it's still a good upgrade over Kozma in 2013 (.217/.275/.273).
Overall:
A great thing about these offseason moves (so far) is that the Cardinals didn't give up a single high value prospect. They kept Taveras, Wong, Wacha, Miller, Rosenthal, Kelly, Lynn, Siegrist, Martinez, Piscotty, etc
In addition, by not signing Stephen Drew (who had received a qualifying offer from Boston) the Cardinals retain their first round draft pick in 2014. With the Carlos Beltran signing, they'll get another first round draft pick in 2014 too.
Overall, a great set of moves that improves the offense and defense of the team, keeps the depth of the minor league system strong, doesn't overburden the team with a huge contract, and keeps draft picks for the future.
Thursday, November 7, 2013
The Baseball Scheduling Problem
Came upon this ESPN video at random the other day.
http://www.grantland.com/story/_/id/9930115/our-latest-30-30-documentary-short-tells-story-how-husband-wife-ended-creating-schedule-mlb
For years I had assumed that baseball (and all professional sports) had used computers to determine their season schedules, but apparently the scheduling problem was so difficult and so complex, human beings did Major League Baseball's schedule by hand until 2004. The couple highlighted in the video did it for about 20 years. Apparently multiple companies and research universities (MIT, Stanford, Carnegie Mellon, IBM are mentioned) failed to come up w/ schedules in the 80s and 90s. In some cases, it was not failed to come up with "good schedules", but apparently failed to come up with a schedule at all.
The scheduling problem seems mighty complex. The scheduling conditions below are from the video and my own knowledge of baseball.
http://www.grantland.com/story/_/id/9930115/our-latest-30-30-documentary-short-tells-story-how-husband-wife-ended-creating-schedule-mlb
For years I had assumed that baseball (and all professional sports) had used computers to determine their season schedules, but apparently the scheduling problem was so difficult and so complex, human beings did Major League Baseball's schedule by hand until 2004. The couple highlighted in the video did it for about 20 years. Apparently multiple companies and research universities (MIT, Stanford, Carnegie Mellon, IBM are mentioned) failed to come up w/ schedules in the 80s and 90s. In some cases, it was not failed to come up with "good schedules", but apparently failed to come up with a schedule at all.
The scheduling problem seems mighty complex. The scheduling conditions below are from the video and my own knowledge of baseball.
- 30 baseball teams, each plays 162 games (2430 games total)
- The schedule is unbalanced, teams in the NL or AL play within their league more often than the other (15 teams per league).
- Teams within divisions play each other more often than other divisions in the league (5 teams per division).
- Teams cannot play any other team in the other leagues (interleague play) but only specific ones in certain divisions.
- Cities w/ two teams (Chicago, NY, etc.) must not be in the city at the same time on specific days (especially weekends and such)
- Special games are scheduled in for specific rivalries. e.g. New York Yankees and New York Mets will play each other each year no matter what.
- For ratings/ticket sales, specific rivalries (intra division or otherwise) must be scheduled at specific times. e.g. intra-division battles almost always must be scheduled at the end of the year in September. e.g. Boston vs NY or St. Louis vs Chicago games should almost always be on weekends.
- A specific number of rest days are mandated for each team. They must be distributed evenly throughout the year (i.e. not back to back).
- Rest days must be scheduled in when long travel occurs (e.g. west coast teams going to east coast).
- Traveling must not be unbearable (e.g. you can't schedule team to play in NY, then LA, then Miami back to back to back).
- Each team has special requests to be home/away on certain dates b/c of special occasions or city specific events. For example the SF Giants may not want to be in town during Fleet Week.
- Scheduling must be fair and equitable for holidays. For example, it would be unfair to never allow a team to have a home game during the July 4th weekend. In addition, you must handle Canadian holidays.
- Some teams may share their stadium with other professional sports teams, so that must be handled. Although this constraint certainly existed in the past, it may not exist anymore (off the top of my head, St. Louis, San Diego, San Francisco, and Houston used to share their stadiums but no longer do). Although I'm sure special case handling probably still exists (e.g. state championships, conventions, etc.).
Subscribe to:
Posts (Atom)