A while back i remember looking at the kernel source code, when overcommit is enabled, malloc would not fail if it couldnt allocate memory, it would ONLY fail if you attempted to allocate memory larger than the available memory space.
I not think you can deal with the failure condition the way you think on Linux (and I imagine other operating systems too).