\d less efficient than [0-9] (opens in new tab)

Short answer: \d includes all the Unicode characters from http://www.fileformat.info/info/unicode/category/Nd/list.htm

I was a bit surprised that Perl does not seem to be matching Unicode digits. Anyone know why?

    $ echo '0' | perl -pe 'print "yes: " if m/\d/'
    yes: 0
    $ echo '੧' | perl -pe 'print "yes: " if m/\d/'
    ੧

I wonder what kind of security vulnerabilities could be looming in validators not expecting non-ascii 0-9 digits and using this regex?

Regex is a really powerful tool, but sometimes I wonder just how well people actually understand it as the vast majority of people (myself included) seem to be self taught in the syntax - only learning the bits they need as and when they need it.

The problem is, regular expressions is packed full of counter intuitive idiosyncrasies which make perfect sense once they're explained, but are far from obvious. Take this for example:

    s/(^\s+|\s+$)//g

is slower than running two separate regex, like so:

    s/^\s+//;
    s/\s+$//;

So it does make me wonder the number of bugs that have been introduced to software by bad regex.

The quoted benchmarks all complete in fractions of a second. Not a good sign. They may be reliable results, performed accurately, but why risk it?

IMO you should be running something for much longer, to protect against random short spurious events. e.g. a task reschedule, interrupts, etc could add significant variances. It wouldn't hurt to add a few more zeros to the loop and wait a minute for the results.

in C#

The test code creates a new regex every time, would be interesting to see how it works with a compiled and reused regex.

Interesting to see these missing, which are 1 and 1, respectively: [一, 壹]

What is the need for those "mathematical monospace/bold/sans" characters? Should that be a font issue?

Python's methods on unicode strings also apply this logic. E.g.:

     >>>  u'١٣٦٨'.isdigit()
     True

    >>> int(u'١٣٦٨')
    1368

I suppose this could be potentially abused if you are storing and displayeing what is supposed to used as a number as unicode text, but later convert it to a number. E.g. an online shop where you are asked whether you want to pay '5꯸' for some item which looks like 5 plus some weird square, but is really int(u'5꯸') => 58 -- http://www.fileformat.info/info/unicode/char/abf8/index.htm

The fact that character ranges like [a-z] can depend on the value of LC_COLLATE is also something not many people are aware of.

  $ echo "ä" | LC_COLLATE=C grep '[a-z]'
  $ echo "ä" | LC_COLLATE=en_US.UTF-8 grep '[a-z]'
  ä

For common values of LC_COLLATE, the range [a-z] does not exclude accented characters and umlauts.

There seems to be a tiny bit of difference in Ruby too. This code:

    require 'benchmark'

    def random_string(length)
      result = (1..length).map { (65+rand(26)).chr }.join
      result[rand(length)] = rand(10).to_s if rand > 0.5
      result
    end

    Benchmark.bmbm do |b|
      b.report("\\d") do 
        (1..1000).count { random_string(1000).match(/\d/) }     
      end

      b.report("[0-9]") do 
        (1..1000).count { random_string(1000).match(/[0-9]/) }
      end

      b.report("[0123456789]") do 
        (1..1000).count { random_string(1000).match(/[0123456789]/) }
      end
    end

gives:

    ~/Code/ruby% ruby regex.rb
    Rehearsal ------------------------------------------------
    \d             0.690000   0.000000   0.690000 (  0.712500)
    [0-9]          0.690000   0.000000   0.690000 (  0.703990)
    [0123456789]   0.680000   0.010000   0.690000 (  0.705759)
    --------------------------------------- total: 2.070000sec
    
                       user     system      total        real
    \d             0.710000   0.000000   0.710000 (  0.791722)
    [0-9]          0.700000   0.000000   0.700000 (  0.708210)
    [0123456789]   0.690000   0.010000   0.700000 (  0.713355)

Noted here that modifying \d to only include [0-9] yields \d more efficient http://stackoverflow.com/a/16622773/1943429

I tend to use ranges (e.g. [0-9]) as they seem to me to be more standard than the token for "any digit" (often \d, but in elisp (Emacs) it's [:digit:])

Maybe the order in which the Regexes are evaluated is also important due to caching etc. Has anyone tested if results are different when changing the order?