undefined | Better HN

0 pointsjstimpfle7y ago0 comments

Here is the musl implementation.

> https://github.com/esmil/musl/blob/master/src/string/strtok....

It's a bit longer than 4 lines because strtok does things you should not want. If you insist on parsing that configuration line with strtok, go ahead and write that brittle code. It breaks as soon as you want empty strings (try "keyword=value1, , value3" with strtok) or escape sequences or other transformations, or as soon as you want to do something as basic as parsing from a stream instead of a string that is completely in memory.

So to clarify, of course you are never done with parsing in 4 lines. But even if it wasn't as braindead to overwrite the input string, the functionality strtok provides would not be worth more than 4 lines.

0 comments

yason7y ago

So, here's that implementation:

    static char *p;
    if (!s && !(s = p)) return NULL;
    s += strspn(s, sep);
    if (!*s) return p = 0;
    p = s + strcspn(s, sep);
    if (*p) *p++ = 0;
    else p = 0;
    return s;

Instead of carrying that code, or something similar, with my source code or my own utility library I'd much rather have the already debugged version from the standard library.

Overwriting the input in C is more efficient than maintaining more internal state and returning a pointer and the length of each token which you would need to strncpy() to get the token into a C string. strtok() does not want to do the initial strdup() for you because only you will know whether your input can already be mutated or whether you need to use a copy.

As I pointed in the other reply, strtok() does not break on strings like "keyword=value1,, , value3" unless you skipped RTFM and expect it to do something completely different. And more often than not that's exactly what you want when parsing non-computer readable input which you can expect to take a specific form.

If you want to handle escape sequences, parse from a stream (without having the option to fgets() the next line into memory), or parse CSV tables without collapsin colunms then you will want to use something more specific to that. Luckily, strtok() was not advertised as a Swiss army knife so it's off the hook for specific parsing purposes like those.

jstimpfleOP7y ago

As someone else pointed out what you really want is the implementation of strspn/strcspn which is where the loop is. You don't "carry" that code along. You just write

    while (i < len && !is_token_begin(buf[i]))
        i++;
    if (i == len)
        error("End of input\n");
    start_token(tok);
    while (i < len && is_token_char(buf[i])) {
        i++;
        add_to_token(tok, buf[i]);
    }
    end_token(tok);

or something along those lines. Whatever you need. It's not rocket science. Putting highly fluctuating and project-specific code like this in a library would only have disadvantages. Not everything should be in a library. In fact, most things should not be.

j / k navigate · click thread line to collapse