Hadoop / MapReduce alternatives for parallel computing? (opens in new tab)

(tompinckney.com)

11 pointstom_pinckney16y ago10 comments

10 comments

Deepak Singh of Amazon Web Services maintains a great list of (cloud-focused) parallel computing frameworks and platforms: http://deepaksingh.net/Resources/Computing_in_the_Cloud

He's on twitter, too: http://twitter.com/mndoci

tom_pinckneyOP16y ago

In the name of completeness, there are also great packages like OpenMPI and OpenMP.

At least for my particular applications, though, there's either 1) a steep learning curve for programmers 2) language support issues 3) they're designed for batch processing.

Personally, I find shared memory interfaces the easiest to program when there're complicated data access patterns. But that just might be personal preference.

amock16y ago

Shared memory interfaces are easy to use, but they don't support the same platform as Hadoop and MapReduce because you can't efficiently split them up across machines. With a distribute system like Hadoop you can build a cluster of cheap machines and spread the computation across them. With a shared memory architecture you have to scale up with multi-million dollar machines like SGI's altix line. So if you want to be able to use a cloud of cheap computers you need something like MPI or Hadoop.

1 more reply

wmf16y ago

What do you think about VoltDB? Have you applied for the beta?

tom_pinckneyOP16y ago

Anything Stonebreaker does is interesting, but I don't know enough about VoltDB to have anything to say. Definitely curious where it goes.

j / k navigate · click thread line to collapse

10 comments

nethergoat16y ago

Deepak Singh of Amazon Web Services maintains a great list of (cloud-focused) parallel computing frameworks and platforms: http://deepaksingh.net/Resources/Computing_in_the_Cloud

He's on twitter, too: http://twitter.com/mndoci

tom_pinckneyOP16y ago

In the name of completeness, there are also great packages like OpenMPI and OpenMP.

At least for my particular applications, though, there's either 1) a steep learning curve for programmers 2) language support issues 3) they're designed for batch processing.

Personally, I find shared memory interfaces the easiest to program when there're complicated data access patterns. But that just might be personal preference.

amock16y ago

1 more reply

wmf16y ago

What do you think about VoltDB? Have you applied for the beta?

tom_pinckneyOP16y ago

Anything Stonebreaker does is interesting, but I don't know enough about VoltDB to have anything to say. Definitely curious where it goes.

j / k navigate · click thread line to collapse