undefined | Better HN

0 pointscbsmith9y ago0 comments

> What database would you recommend for solving the taxi problem using the JVM?

You have me at a disadvantage here... The only taxi problem that comes to mind is a probability problem that I'd not likely use a database for at all...

> If this is a business problem, and I solve it in 1/1000th the time, for roughly the same cost, then what exactly is your complaint?

If you came to the conclusion that your business problem runs 1000x faster because of differences in the runtime... you've made a mistake. It is far more likely your benchmark is flawed, or there are significant differences in the compared solutions beyond just the runtimes.

Seriously, I've spent a career dealing with situations exactly like that: "hey, this is 1000x slower than what we are doing before... can you fix that?". Once you are dealing with optimized runtimes, while there can be important differences between them, there just isn't that much room left for improvement.

> It's not ambiguous. I'm pointing to the timings for a specific, and realistic business problem.

The problem is perhaps not ambiguous to you, but you haven't described it in terribly specific terms. More importantly though, you haven't described what you mean by "faster"? That's the ambiguity.

> Does Java include glibc?

> What exactly is your point here?

C programs do. Lots of very efficient, high performance C programs.

0 comments

3 comments · 1 top-level

geocar9y ago· 2 in thread

> The only taxi problem that comes to mind

It's the problem that I linked to previously.

http://tech.marksblogg.com/benchmarks.html

Finding good benchmarks is hard: Business problems are a good one because these are the ways experts will solve problems using these tools, and we can discuss the choice of tooling, whether this is the right way to solve the problem, and even what the best tools for this problem is -- in this case, GPU beats CPU, but what's amazing is just how close a CPU-powered solution gets by turning it into a memory-streaming problem (which the GPU needs to do anyway).

> If you came to the conclusion that your business problem runs 1000x faster because of differences in the runtime...

I haven't come to any conclusion.

There are a lot of differences between a JVM-powered business solution and a KDB-powered business solution, however one striking difference is the cache-effect.

However the question remains: What exactly do we get by having a big runtime? That we get to write loops?

cbsmithOP9y ago

> what's amazing is just how close a CPU-powered solution gets by turning it into a memory-streaming problem (which the GPU needs to do anyway).

Yes, it turns out the algorithmic approach you use to solve the problem tends to dwarf other factors.

> There are a lot of differences between a JVM-powered business solution and a KDB-powered business solution, however one striking difference is the cache-effect.

Wait, you looked at those benchmarks and came to the conclusion that the language runtimes were the key to the differences?

> However the question remains: What exactly do we get by having a big runtime? That we get to write loops?

There is absolutely no intrinsic value in a big runtime.

Now, one can trivially make a <1KB read-eval-print runtime. So I'll answer your question with a question: why do people not use <1KB runtimes?

geocar9y ago

> Wait, you looked at those benchmarks and came to the conclusion that the language runtimes were the key to the differences?

At the risk of repeating myself: I don't have any conclusions.

> There is absolutely no intrinsic value in a big runtime.

And yet there is cost. It is unclear if that cost is a factor.

> Now, one can trivially make a <1KB read-eval-print runtime. So I'll answer your question with a question: why do people not use <1KB runtimes?

Because they are not useful.

We are looking at a business problem, think about the ways people can solve that problem, and cross-comparing the tooling used by those different solutions.

Is there really nothing to be gained here?

The memory-central approach clearly wins out so heavily (and the fact we can map-reduce across cores or machines as our problem gets bigger) is a huge advantage in the KDB-powered solution. It's also the obvious implementation for a KDB-powered solution.

Is this Spark-based solution not the typical way Spark is implemented?

Could a 10mb solution do the same if it can't get into L1? Is it worth trying to figure out how to make Spark work correctly if the JVM has a size limit? Is that a size limit?

There are a lot of questions here that require more experiments to answer, but one thing stands out to me: Why bother?

If I've got a faster tool, that encourages the correct approach, why should I bother trying to figure these things out? Or put perhaps more clearly: What do I gain with that 10mb?

That CUDA solution is exciting... There is stuff to think about there.

1 more reply

j / k navigate · click thread line to collapse