It's Linux (see [1] for why) and from the git repos it looks like it uses OpenNSL [2] (which is not open source but open in the sense of "this used to be under NDA and now it's not so be grateful we're giving you that much").

[1] http://www.wired.com/2015/09/microsoft-built-linux-everyone-... [2] http://www.broadcom.com/blog/network-infrastructure/inside-o...