A Quick Puzzle to Test Your Problem Solving (opens in new tab)

People familiar with unit testing and test driven development will feel at home with this kind of puzzle. That doesn't mean that they will be less biased in social/political decisions, it just means that this test will fail to prove a point.

This is now the fourth time I've seen this exact rule. Another example: https://www.youtube.com/watch?v=vKA4w2O61Xo

It's probably not "fair" to say I got it in zero... but I did. :)

Now that the NYT has done it, this puzzle has probably attained enough popularity now that you really ought to change it up a bit now if you're going to run it yourself. Granted, the space of hypotheses as simple as "increasing/decreasing" is pretty small, but your ability to fool people with the first sample run is almost unbounded, so that helps.

If you look at the raw answers, a lot of people who guessed the wrong answer according to: "What answer did you give?" said they got it right:

Right? What answer did you give?

Yes, double the previous number

Yes, Each number much larger than the previous

Yes, sequence must always increase by 1

Yes, Powers of 2

Yes, h = 2n, i = 2(n+1), j = 2(n+2) . n is an integer

Yes, Powers of 2

There's a selection bias - those of us who got it right are more likely to fill that out (:

As a result we can't really rely on overall accuracy, but we can break it out by yes/no to account for the selection bias to get a profile for how a HN correct and incorrect differ.

This crowd needs one more question:

How many "I know the answer, but can I find a flaw in their code" questions did you ask?

to be pedantic: "monotonically increasing" is incorrect. It should be "strictly increasing"

Cool idea, thanks!

Personally, I spent most of my answers playing with the inputs. The form was happy to report that "1e1, 15, 0x10" was a valid sequence. :D

This is a good idea. I submitted the survey but here are my tests: http://imgur.com/hYKudyl

FWIW, I've seen this kind of game before and I was expecting it to be something simple.

My answer was: "The sequence is of an increasing real variable, where each subsequent value is greater than the preceding value. It's monotonically varying."

When I clicked "I think I know it", nothing happened. I don't want to click their "I don't want to play; just tell me the answer". But it seems like the right answer. I can't answer your form question if it is the right anwer, since I haven't clicked on their link and don't know for a fact whether it is or not.

Although I used the wrong term, it's strictly increasing.

Your last question is "have you seen a test about confirmation bias before?" But the text of this test says that it's about "why no-one likes to be wrong", which means it's pretty obviously about confirmation bias (and therefore that the test-taker should be wary about just confirming their first intuition).

[ 456 , 42 , 48 , No ] [ 1 , 2 , 3 , Yes ] [ 3 , 2 , 1 , No ] [ 45 , 46 , 47 , Yes ] [ -1 , 999 ,1238978 , Yes ] [ 1+1 , 1+2 , 1+3 , Fail ] [ 7 , 654 , 653 , No ] [ 7 , 654 , 655 , Yes ] [ 1.1 , 1.2 , 1.3 , Yes ] [ 1 , e , pi , Fail ] [ 1 , e , 42 , Fail ] [ -1.1 , -1.0 , -0.9 , Yes ] [ 2i , 3i , 4i , Fail ] [ 1 , e , pi , Fail ] [ 0 , 1 , 2 , Yes ] [ -1 , 0 , 1 , Yes ]

If anyone's tallying

Intersting that the split of correct/wrong answers from the HN crowd is 78%/22% - the exact opposite of the general population : 22% / 78%! The HN community does think different :)

I wanted to answer "Probably" to the last question about having seen a similar question before.

I don't specifically recall seeing one, but it is likely that I have.

Guessed after just one "No"

  3  9 27  yes  (is it exponential series?)
  4 16 64  yes  (is it only odd numbers?)
  5  7  9  yes  (is it any numbers of the same parity?)
  6  7  8  yes  (is it any set of increasing numbers?)
  6  7  6  no   (just to confirm that it's x<y<z, and not something like x<=y<=z)

It would be relevant to include a histogram of #yes_answers - #no_answers in the summary, to test whether people are biased towards positive rather than negative tests. I think the raw data suggests that it does although I totally failed to create a histogram in google spreadsheets within 5 minutes.

My fuzzing went:

8 4 2 (no) 1 2 3 (yes) 1 1 1 (no) 1 100 123 (yes) 1.0 1.1 1.2 (yes)

answer: incrementing numbers

Thanks for setting up the survey!

Could you perhaps move to a bar chart instead of pie charts?

This has nothing to do with Gödel's incompleteness theorem. It's much simpler than that: https://en.wikipedia.org/wiki/Wittgenstein_on_Rules_and_Priv...

I think the core idea is much simpler than that. It's the notion that falsifiability is the most powerful tool in our arsenal when we try to conceive of theories that explain a certain state of affairs.

The idea you're getting at is that no number of confirming observations can verify a universal generalization -- the reason why we continue to call generally accepted "truths" theories. But we can increase our certainty to the greatest degree possible by trying to test our hypothesis to the greatest degree we can.

Remotely related: I've been interested for awhile in how the same initial terms of a sequence could possibly be generated by multiple rules.

For example, you might have

2,3...

And the rest of the sequence might look like either

2,3,4,5,6...

or

2,3,5,8,13...

or

2,3,5,7,11...

or even

2,3,5,10,20...

Clearly, on some level those sequences are all much less complicated than one defined as "The first term is 2, the second term is 3, the third term is 919243, the fourth term is -1234..."

It's unclear to me how one might rank them in complexity, though. The maximum amount of memory necessary to get an arbitrary nth term? The number of operations necessary to get to the next term?

Another interesting question to me: if there is an ordering of ways to generate a sequence of numbers, given the first couple terms of a sequence of numbers, what are the simplest N ways to generate the full sequence?

It's not ironic. Dijkstra's assertion is, in a convoluted way, related to the main point of the quiz and the article :)

With an equal protection clause, i.e., no particular number or a group of numbers can appear in the rule, is the problem solvable?

Tests like this are frustrating even if you don't recognize the logical fallacy! Just because it's always possible you're wrong. You're either right because you're lucky, or wrong because you messed up!

This does not hold - the test uses Javascript's numbers, which are finite. So it's theoretically possible to test every combination of inputs and convincingly answer what a possible rule is.

It's amusing that you went to a website on confirmation bias, did the puzzle incorrectly, presumably read the material on confirmation bias, but still suffer from the effects of confirmation bias.

The puzzle is not "Find a rule that matches everything you tested". It's figure out what rule they are using. It may be true that there are multiple ways to state the rule, such as "x_2 >= x_1 + 1 and x_3 is >= x_1 + 2" but they are equivalent.

Try the sequence: 0,0,0. It would give "yes" for (x,2x,4x), but the actual rule gives it "no".

Your task is to learn a class of objects by example. Without seeing any negative examples, you're unlikely to learn the right class (which is unique, it is precisely the class of increasing sequences of length 3). (x, 2x, 4x) is not "a correct answer", as it does not describe the class you were supposed to learn to distinguish.

It's not an assumption; the article tells you up front that there is one correct answer.

It is an analog for how science works. When it comes to a natural phenomenon, humans can come up with multiple explanations that fit a given set of observations, but presumably (I mean, this is a basic tenet of science) nature only works in one consistent way.

Thus, the importance of a falsifying test. You form a hypothesis based on the initial observations (in this case, the number sequence 2, 4, 8), and then you propose a test that could falsify your hypothesis.

The trick is that a hypothesis can fail in several ways. It can be outright wrong, like saying "the rule is that the numbers decrease from left to right." That's obviously just wrong.

But it can also be too specific, like saying "the rule is that the exponent increments by one with each step to the right." That matches the given evidence, and tests with other base numbers will succeed too. But it's over-fitting.

Here's a concrete example: a man wearing a red shirt drops a weight and measures gravitational acceleration as 9.8 m/s^2. So he formulates a hypothesis that gravity always produces an acceleration of 9.8 m/s^2 in the presence of a red shirt.

And if he always wears a red shirt, and always tests gravity on the surface of the Earth, he'll always find supporting evidence for that hypothesis.

But of course we know that gravitational acceleration varies depending on mass and distance, and that it's the same no matter what color your shirt is. But he would only find that out if he varied his experiment beyond what his hypothesis predicts.

Well, yes, in this situation random guessing would be very likely to provide an immediate check on the solution.

(1x, 2x, 4x), as indicated in the video below, is not sufficient. It represents a subset of the values that are valid.

Think of it this way. When asked to write a unit test, do you only test the positive outcomes? No, you test to make sure the failures are as you expect as well. Otherwise, you are likely to have what you think is a failure end up as a success.

You got the problem completely wrong.

The idea isn't to come up with tuples that satisfy the predicate. The idea is to figure out what the predicate is in the first place.

Also, you're not in any way, shape, or form reduced to random guessing. If you have an idea of what the rule might be, you build a counterexample. There's a ton of value in _trying_ to get a no but getting a yes instead.

From a Computational Learning Theory perspective, we are faced with an infinite hypothesis space, with an infinite VC dimension. So yes, there's not much strategy that can be employed here.

But, the fact that there is one correct answer is not really an assumption that the puzzle makes, it is information we have been given:

> We've chosen a rule that some sequences of three numbers obey -- and some do not.

This simply means that the solution is realisable. No matter how many ways (in English) we have to describe that solution it is still the same solution.

Consider the sequence

1, 3, 5, 7

what comes next? 9 right? Or is the sequence generated by 2n − 1 + (n − 1)(n − 2)(n − 3)(n − 4) for n ∈ N. Then we've got 33.

"among all hypotheses consistent with the observations, the simplest is the most likely"

33 is correct, but it's less likely to be the basis for the generation of the sequence.

Your answer of (x, 2x, 4x) proves the puzzle illustrates the confirmation bias, at least in your case.

Does the unit test that confirms your function returns the expected result given one set of arguments prove it correct?

(x,2x,4x) does not actually give a "yes" every time. In particular, it won't work if x is negative. But, in that case, (4x,2x,x) will work.

> (x, 2x, 4x) gives you a "yes" every time, therefore it is a correct answer

I think this logic is a bit wonky - if there are sequences that get a "yes", but don't match (x, 2x, 4x) then the correct rule cannot be (x, 2x, 4x), can it?

I've read this comment three times and I still can't figure out what you're trying to say. It sounds like you're arguing that all "puzzles" should exist on finite domains so that a brute force solution exists.

"(x, 2x, 4x) gives you a "yes" every time, therefore it is a correct answer, at least as automatically checkable."

Well, no: you can type 1, 2, 3 into the system and it will tell you "yes", but your rule says that it should tell you "no".

It is crucial to the definition of "correct answer" here that your rule should not just say "yes" only for tuples which the widget also says yes to, but also your rule should say "no" only for tuples which the widget also says no to. That is what the puzzle means when it's asking, "can you guess the rule that we've created?"

This makes it very, very different from what I think you're thinking about, which is situations where someone tells you, "what is the next number in this sequence? 4, 7, 13, 25, ...?" where technically there are an infinite number of rules which will generate those 4 numbers first and an arbitrary number afterwards. Technically one of them is "simplest" in the sense that it can be expressed in 7 symbols, but in general it's a complicated problem and there is no best solution.

"To find a 'no' you're reduced to random guessing. That's not a puzzle, that's crap."

In many ways it still is a puzzle but the space that it lives in is richer. If you think about typical "puzzles" they're things like: "here's a grid with some spaces filled in with numbers,

    2 . . 2 . 2 .
    . . . . . . .
    1 . 3 . . 2 .
    . . . . . . .
    3 . . . 2 . 3
    . . 2 . . . .
    . . . . . . .

Each number is a block in a block wall. We want you to turn this into a block maze so that each 'block wall' (set of blocks connected by adjacency) contains exactly one numbered-block whose number says how many total blocks are in the wall. Furthermore the path (non-block space) of the block maze should be connected and should not contain any 'rooms' -- that is, any 2x2 or larger segments of open space."

This 7x7 grid has 10 spaces which are known to be blocks and exactly 12 more blocks scattered in the remaining 39 spaces, so just by those factors alone we're searching only (39 choose 12) ~= 3.91 billion possibilities; we can also use a quick heuristic to identify 6 places which must be "space" to break apart adjacent numbered walls, removing 91% of that search space.

The puzzles, "I have a set of integers where inclusion in the set is governed by a short rule, you can ask me any integer and I will tell you whether it is in my set", by contrast, have an infinite search space. This means that any solution is going to be more interesting, as will the means for checking that solution's validity. You could require, for instance, a Haskell expression of 140 characters or fewer which turns a nonnegative Int named `n` into a Bool, to be judged as "valid" or "invalid" if it properly filters `[1..10000000] :: [Int]`. You could even give the first 100 numbers in the set, e.g.:

    ghci> take 100 $ filter trueFn [0..]
    [2,5,8,9,13,14,18,19,20,25,26,27,32,33,34,35,41,42,43,44,50,51,52,53,54,61,62,63,64,65,
    72,73,74,75,76,77,85,86,87,88,89,90,98,99,100,101,102,103,104,113,114,115,116,117,118,
    119,128,129,130,131,132,133,134,135,145,146,147,148,149,150,151,152,162,163,164,165,
    166,167,168,169,170,181,182,183,184,185,186,187,188,189,200,201,202,203,204,205,206,
    207,208,209]

In this case that's pretty much enough to see the general pattern; the verification covers 10 million bits while the 140-character limit probably limits your search space to 1000 bits or so, so it's going to be hard to get an "incorrect" answer which agrees on that subspace of the whole.

> There is not one correct answer. (x, 2x, 4x) gives you a "yes" every time

Not true. What if x is negative?

Responds "Yes" to

  9007199254740990, 9007199254740991, 9007199254740992

but "No" to

  9007199254740991, 9007199254740992, 9007199254740993

Presumably this is due to how Javascript handles integers, i.e. it uses the integer part of a float64, to wit

  > parseInt('9007199254740992')
  9007199254740992
  > parseInt('9007199254740993')
  9007199254740992

Edit: I think this is the code that actually reads the numbers the user enters, see [0]

  function l(){
      var a=h.exec(m[1]),f=null,g=null,n=null;
      return a&&(null!==a[1]&&a[1]&&(f=parseInt(a[1],10)),
          null!==a[2]&&a[2]&&(g=parseInt(a[2],10)),
          null!==a[3]&&a[3]&&(n=parseInt(a[3],10))),
      new e(f,g,n)
  }

Edit(2): Actually, I'm not so sure that's the correct code at all. They NYT game is capable of parsing floats correctly (e.g. it accepts 1.1, 1.2, 1.3 as a "Yes") so it's not just using parseInt.

[0] http://a1.nyt.com/assets/interactive/20150612-151638/js/foun...

Further subtleties:

* The number may have optional sign and digits after a decimal point, and may use exponential notation. Example: (-1.2e1, .0E+0, 1.e-3) => "Yes". As seen in the second and third number here, there may be no digits before or after the decimal point, but both at the same time (i.e., ".0" and "0." parse but not ".").

* If the number begins with "0x" or "0X" it is read in hexadecimal, where the digits a-f may be in either case. Hexadecimal notation must not be accompanied by decimal point, sign, or exponential notation.

* No whitespace is permitted within the numeral, even between the sign and the digits as in "+ 11", but both tabs and spaces may be used before and after the numeral without changing its value. In particular, by using a input of the form "1 " it is possible to make rectangular display empty while still parsing it as number. Note that pressing "Check" leads to the numbers being displayed in the rectangle in exactly the same way as they were displayed in the text box, which may depend on the position of the cursor in the text box.

ETA: Also, you mentioned rounding, but there is also exponent overflow and underflow. The application refuses to parse numbers greater or equal to 1.7976932e308. It parses arbitrary negative exponents fine, but it does not recognize that 1e-324 is greater than 0.

A test engineer walks into a bar. He orders a beer. He orders two beers. He orders 999999999 beers. He orders 1.00001 beers. He orders -42 beers. He orders 1048576 beers...

Only on hackernews :O

Good old IEEE 64bit floating point numbers =)

That also means that for(i=0;i<j;++i){} doesn't necessarily terminate for an arbitrary j smaller than infinity, which I find hilarious.

Any idea why? Could it be an error from the input being too large? Or is there some other magic at work here...

Nedit: One of the other responses nailed it. IEEE standards on 64 bit fp ops

I tried a negative series and a decimal series, just in case the rule was increasing natural numbers. I tried very large numbers to see if there was a limit to the rule, and a series that had large contrast in between each element. For fun I tried to see if the app would recognize "pi", "i", or "e", but, perhaps unsurprisingly, it did not.

I did test with negatives, but I did not think to test with decimals. 15 yes's, 6 no's, and I successfully determined the rule.

I tested with negative integers, but didn't think to test with real numbers unfortunately. (Still got it right, but I should've tested that.)

That said, I did make sure to test all the edge cases I could think of. I was actually going to guess (n^1, n^2, n^3) at first, until it failed for (1, 1, 1).

10 "yes" tests, 6 "no" tests.

I wonder if somebody tested the rule to be deterministic by entering same number over and over. But I think the rule a<b<c is too simple to invite to use creative tests.

Yes and yes. It wouldn't activate the "check" button with complex numbers.

I got it right, using decimals - I was probably a little too exhaustive...

http://imgur.com/6adtvqz

I tested with real numbers to be curious, just tried 1.1, 1.2, and 1.3.

Full method just for fun:

    function judgeSentence(sentence, numNo) {

        sentence = sentence.toLowerCase();

        // no nos -> wrong.
        if (numNo === 0 || sentence == "") {
            return false;
        }

        // if have any fancy words -> wrong.
        var probablyWrong = ["doubl", "expon", "multipl", "^", "**", "power", "two", "2", "twice", "as big", "nth", "rais"];
        if (hasAny(probablyWrong, sentence)) {
            return false;
        }

        // if you have the right words, and no buts.
        var seemsRight = ["larger", "increas", "greater", "small", "less", "big", ">", "<", "go up", "ascending"],
            weaselWords = ['but ', 'not ', 'odd'];
        if (hasAny(seemsRight, sentence) & !hasAny(weaselWords, sentence)) {
            return true;
        }

        // // no nouns, verbs or adjectives in your sentence -> wrong.
        // var s = nlp.pos(sentence).sentences[0],
        //     verbs = s.verbs().map(getWords),
        //     nouns = s.nouns().map(getWords),
        //     adj = s.adjectives().map(getWords),
        //     numWords = verbs.length + nouns.length + adj.length;
        // if (numWords === 0) {
        //     return false;
        // }


        return false;
    }

It's surprising how far you can go by such heuristics. I run an IRC bot that uses matches like this (I'm working on a proper solution right now though) to parse natural language queries and I managed to trick few people into thinking they're talking with human. As long as it's ok for 90% of most common cases, people often won't notice.

include "monoton" in probablyWrong.

The sequence is not monotonically increasing.

Same here - I was also guilty of feeling a bit clever and avoiding the obvious "oh, the numbers double every time" answer. So, when I found out that if the spelling of each number was one letter longer (4 = four letters, 8 = five letters, 11 = six) I got smug, and didn't bother testing further.

Good article - and a humbling experience. :)

I think it's a reasonable mistake.

Most people will make assumptions about what's required based on previous experience. And hardly anyone will have previous experience where a question formatted in this way isn't asking you to find a series rule.

That's not quite the same thing as confirmation bias. With a bias you're just as likely to discount significant evidence as you are to mismodel the problem space.

> It's trivially easy to fool the user into finding a wrong rule.

While that's true, the real thing being tested is the user's readiness to prove themselves wrong.

It is true that, in general, this quiz is impossible to guess, for the reasons you explain.

But in this case -- which is the main point of the article -- it was actually a trivial rule. No tricks, no special cases. The real purpose wasn't for you to find the actual rule, but to learn about your biases. The only trick here lies in the human mind, and its tendency to validate patterns (and claim an early victory) instead of trying to refute them.

Yeah I thought it would be something like "Numbers must strictly increase unless the first number is 15326". Then the point of the article would be that some government rules are not well defined.

Yes, I'm curious what heuristics it used there.

Playing the confirmation bias game on "playthroughs of the confirmation bias game": it looks like you get that message if you have a 'no' answer, and the word 'increasing' in your guess. ("Increasing by the same amount" is still accepted.) I wouldn't be surprised if other words also count.

Edit: the words '<', '>', 'increas', 'big' and 'larger' seem to count. Looks like they're accepted as substrings, not just words - so 'increase' and 'increasing' are accepted. I could look for a long time, so I'm giving up now.

I simply said: "a < b < c", with no other words, and it gave me the same answer as yours.

I said "The numbers are in ascending order", which it didn't seem to recognise.

> though presumably that's because it was linked from LessWrong

No surprise there, given that this experiment is discussed in Sequences, the link to the post present in the article you link to :).

Thank you! I forgot the name of the game, and it is totally unsearchable.

Zendo is fantastic. Watching people unaccustomed to its sort of problem-solving struggle with and improve their methodologies for being good at the game is really enlightening.

Maybe some positive/negative feedback built into the campaign could help; e.g. for each act of benevolence/violence, add/subtract a 'karma' point from some running total, and alter the gameplay as needed.

You must be awfully tempted to constantly give them powerful adversaries who are willing to become friends if given the smallest chance.

I guessed correctly with only one no.

You can enter all kinds of crazy random sequences which only have The Rule in common and get a yes, which seemed to be enough assurance. If you're trying to get it to say "no" but failing, is that still confirmation bias? Doesn't sound like it.

I also guessed correctly with only 2 nos -- my real confirmation came from the fact that "-1, 0, 300000000" (or some number of 0s) was correct, meaning it couldn't be any really meaningful sequence.

Isn't your answer technically wrong, though?

The sequence is not monotonically increasing. It's strictly increasing. If you test [1, 1, 2] or [1, 1, 1] or [1, 2, 2], you'll get "No" answers even though those sequences are monotonically increasing.

I got so many nos. I can't believe that "Remarkably, 77 percent of people who have played this game so far have guessed the answer without first hearing a single no." That's crazy.

"Larger" in this context means "greater than" in the usual ordering of the real numbers.

It works for floating point numbers -- 0.01 0.02 0.04 for example. So it's a geometric series that has to start with a positive number and doubles. (The submit button and the show the answer button were broken for me.)

Maybe it should be "a puzzle many people already know the answer to" in which case the conclusion and results are already obviously biased.

I was able to solve the puzzle without testing any numbers at all. Which really skews the relevance of "only nine percent of people saw three 'no's before answering."

Eliminating assumptions baby.

https://www.youtube.com/watch?v=9-9VLVkm8R4

this is exactly what I did. I wonder what the psychology/cognitive science explanation of this might be.

Why is it technically true?

I'm curious about how you came to your answer and what led you there?

Were you testing a pre-supposed hypothesis that confirmed itself?

The answer they were looking for was: Let the numbers be x, y, and z; x < y < z must be true.

Their hypothesis was that most people would guess as you did (they mentioned 78% of people did so).

I had to turn off µBlock to make it work.

There is an implied penalty because it says about the answer box "Make sure you’re right; you won’t get a second chance" in contrast to "You can test as many sequences as you want" about the sequences.

Well, after all they tell you that the sequence consists of numbers:

>We’ve chosen a rule that some sequences of three numbers obey

>Now it’s your turn. Enter a number sequence in the boxes below, and we’ll tell you whether it satisfies the rule or not.

I entered A, B, C. The form didn't let me submit the sequence, so they are doing some verification that the entries are numbers. Rational numbers work, though: 1.1, 1.2, 1.3.

Every nytimes.com page displays that in the JS console.

Were you guessing random area codes of midwestern states? :)

Interesting totals. I had 4 No's and would have had at least 5 had I considered negative digits.

That's a great point. Perhaps we should show something like this to new developers who don't understand the value.

I guessed that immediately (but still verified it). It was obvious it was going to be a trick question, so...

I'd trust the older guy.

Pretty good point. That's how I felt.

Additionally, this setting is probably too close to usual situations you get in school where there is little to no interaction and negative answers from the teacher are seen as failures by students. (Speaking about education in my country only.)

Having just re-read HPMOR a few weeks ago, I could answer it right away, but there was actually a difference from HPMOR's version, which required three positive increasing numbers, while this allowed negatives.

What's funny is that I felt smug and self satisfied for getting the correct answer even though I watched that video when it came out.

Looking at my result there, seems to be something I need to start doing. TIL :P

To be fair, I did check 3/6/12 - just to make sure it was double and not powers of two. Guess that's the articles point though isn't it!