"- Just because Google thinks something is a good idea does not mean you should adopt it.
- Disbelieve all marketing spin, and figure out what benefit any given product actually has. This should be especially applied to performance claims.
- The community of programmers has a love affair with “the next shiny object”. This is likely to create “churn” in your organization, as the “half-life” of shiny objects may be quite short."
Map() is not equivalent to a SQL GROUP BY clause, it is equivalent to a user-defined Table Function that is used in a FROM clause. This mimics the Extract and Transform stages in a SQL ETL pipeline. The Extract is implied by the input format.
The Reduce() is very much equivalent to a user-defined Aggregate Function. D&S accurately criticize the sub-optimal materialization of intermediate data sets but they under appreciate the implicit input split and distributed sorting mechanism which dominated the Terasort benchmark at the time (a Jim Gray creation).
On-Premise commodity Hadoop clusters lost out to public Infrastructure-as-a-Service clusters. None of the five takedown categories turned out to be important. The tools have evolved and cloud-native data warehouses and ETL systems are now the best of both worlds.
[1] https://homes.cs.washington.edu/~billhowe/mapreduce_a_major_...
No, the projection doesn't remove redundancy under most cases. There also isn't any reason you couldn't have UDF's in the GROUP BY clause. I've written implementations of both and I think the GROUP BY is an excellent comparison for understanding Map in MapReduce Systems.
People always looked down on those that used Hadoop or somesuch for <1GB of data, but while it wasn't needed from a technology perspective it gave a structure to the project.
Now many places are back in the world of one-off scripts, and I think something of value was lost (even if it was a little ridiculous to fire up a cluster for something Excel or SQLite could handle).
What 'structure'? Why is it so important that it makes it worthwhile firing up a large, complex framework? I'm beyond baffled.
I know above comment will be lost - but this is such a genuine truth.
This is an interesting thought. A company uses shiny tech because programmers like using them for whatever reason. This attracts employees who want to use this tech too. The half-life for shiny tech is short and so these developers move on to shinier pastures. I wonder if this explains why people change jobs so often in tech? I’m sure I read the average tenure is much lower (~1.5years) compared to other industries.
The thing that makes the redbook special in my opinion is that the editors have been able to apply their research to solve actual problems for paying customers! You don't get to see enough of that in academia.
That's debatable. It's a book, not a blog post.
Its an interesting book in that 2015 was in the middle of the noSQL hype. Since then, people have started looking for results and being more critical.
There's a gazillion technologies that we could list that are newer, and claims that any of them are the next big thing and will fundamentally change everything are, obviously, exaggerated.
[1] Concept-oriented model: Modeling and processing data using functions: https://www.researchgate.net/publication/337336089_Concept-o...
(Disclaimer: I am the author)
I also found this paper in the refs, which seems really good. [0]
