undefined | Better HN

0 pointskiyoto11y ago0 comments

For several reasons, some more legitimate than others.

1. kdb+ was (and maybe is) a good solution to the problem that we had: doing complex data manipulation/simple statistical calculations against billions of rows of time series data. Hadoop is the term du jour for data processing, but truth of the matter is that finance doesn't have really huge data. At best, it's a couple of terabytes, and most of the time, you are working with a small subset of it. Running KDB+ on a beefy server or two would usually do the job (rather well).

2. Maybe because I studied math, but I find k/q's vectorial/functional sematics appealing. I think the syntax is horrible, but the semantics is very neat.

3. Finally, because it helped me keep my job. It was rather amazing to me that all these Ph.D. statisticians that I worked with couldn't bring themselves to learn kdb+ effectively. Apparently this stuff can be very hard for even the smartest people (or maybe they thought it was such a niche skill with a low ROI).

0 comments

mahyarm11y ago

It almost sounds like kdb needs an alternative syntax that is more human readable.

beagle311y ago

It has one: q. But once you get over the syntax, you realize that you also need to grok different semantics that you are used to.

Some q is readable english - e.g., an expression like

   sum price where size>3

is (to the uninitiated) more readable than the equivalent k

   +/price@&size>3

but that only works for simple stuff. The (idiomatic!) computation of maximum-subarray-sum[0]

   |/0(0|+)\

becomes

   max over 0 (0 max +) scan

which is not more readable. And you can drive the point ad absurdum by making it even more verbose:

   max over zero (zero max plus) scan

The syntax seems like it is what stops you from understanding it because it is the first thing you meet. But it's the semantics that you need to grok, and the syntax just matches them.

[0] http://en.wikipedia.org/wiki/Maximum_subarray_problem

sqrt1711y ago

I would find something like

over(max, scan(0, max(+, 0))

slightly more readable, even if that still gives me a higher-order-headache.

Because, y'know, it's confusing to have symbol salad with a weird mixture of infix and postfix operators, even more so if you have type raising (or the moral equivalent thereof) thrown in for good measure.

bastawhiz11y ago

The q in your latter example is the same complexity in my book as a dense Python list comprehension. Could be worse.

1 more reply

j / k navigate · click thread line to collapse