Add experimental fuzz test support for Go 1.17 (opens in new tab)

Yes, I remember, thank you! That's how I got started with go-fuzz.

It's because the Go philosophy is "tests are code", so you use the standard language features like != instead of a whole new domain-specific language that you have to learn. I've written more about that here: https://lwn.net/Articles/821358/

Personally I find the "cutesy" names like Expect(a).To.Equal(b) and It("should ...") patronizing and too English-like (code is not a human language). Similarly, I wouldn't like a language that made you write:

   Please.Print(1.Plus.2)
   Log.To(Console).Text("did something")

The vanilla Go approach can get a bit verbose when you're doing lots of asserts. In that case, you can use standard language features like refactoring things to variables or helper functions, or table-driven tests.

But if you really want assert-style code, I like the https://labix.org/gocheck approach:

   c.Assert(a, Equals, b)

Because there's only one function (Assert) and a bunch of comparator names to learn.

The fluent API flavor is how jest works in javascript-land, and while it makes the code read ok, it makes for some lousy assertion failure messages.

Typically, you get some autogenerated variation of "expected 3, got 5" - which in a large codebase is kinda useless because you don't necessarily have context on what the error refers to without going back to the test source code to see where the failure originated from. This is especially problematic with table-based tests or tests with large numbers of assertions where the closest human-readable label might have been described too generically.

I much prefer to be able write out "foo broke in such such way: expected %s, got %s" than have the team rely on a machine attempting to generate a description and getting a bad description because the human is encouraged to be too lazy to write a proper description and the test framework is not smart enough to write a proper one for them.

I'm not sure what you find ugly, actually I find your example ugly because it seems to be full of magic

My answer would be: flexibility.

And I'd argue the alternative is more verbose since you're not invoking the error/fatal.

I feel like the downside is that those libraries are always one step behind the assertion you actually want. Our codebase at work uses something called "testify" for whatever reason, and it's always missing the things I want to test. For errors, it doesn't have any As/Is support; it just does an exact string match. That's all you get, so people are forced to write worse tests (or change the library, but guess what 99.9% of "line programmers" are going to do.)

If you demand your pet assertion, you have to maintain some extension on top of the library you use, or wait for upstream to add it -- leading to the maintainer having to maintain a 2000 function library of everyone's pet assertions, which isn't fun, and which they probably won't do.

Another problem with assertion libraries is that edge cases are buried in details far away from the test you're writing. What equality function does To.BeEqual() use? Is null equal to undefined? You really have to hope it doesn't matter. But if the assertion is the if condition, you're not adding any new semantics to your language; you know what == does, and the code says ==, so you don't have to guess.

Finally, error output suffers the more generic you make it. You can review a screenful of hand-written assertions, but autogenerated error output always seems to be one screen per error, meaning that you have to do more looking around to fix multiple tests. Minor complaint, but it's the nature of writing a formatter for interface{} vs. exactly what your test tests.

Overall I find it not worth it. Most of the time you'll be writing a table-driven test, and you're only writing the assertions a handful of times for tens of test cases. (A comment below complains that table-driven tests don't point you at the failure; giving you a vague error like "test 42: fail". I personally add a free-form "name" to every test case to avoid that particular problem.)

My main complaint with Go is how the official style guides don't make conventions around the hand-rolled assertions clear. At least when I was learning Go at Google, you pretty much had to use the form:

   if got, want := len(foobars), test.wantFoobarCount; got != want {
      t.Errorf("foobar count:\n  got: %v\n want: %v", got, want)
   }

or your code would not be approved. (The style points here are: the variables should always be called got and want, the output should be in got, want order, and the colons on got and want should be aligned.) Nobody in the real world does this, for whatever reason, and there is no "Code Review Comments" to point to to say why it should be got, want and not want, got. (Why? Because it's arbitrary and someone else already picked.)

I would like to +1 jacque's request. I would also be really interested in a rundown of gopter.

> My colleague implemented an equivalent of `openssl passwd` in go and used gopter to test it (comparing outputs between his implementation and what openssl returns).

Could you talk more about this? I'd love to see the code or any other docs you might have.

The stretch goal of structured fizzing beyond []byte sounds like property testing to me

Just to add, we're a heavy user of fuzzing in Diem[1] and we found a good number of bugs thanks to it : )

https://github.com/diem/diem/