If there is code in there that turns out to be copy-pasted from somewhere else, open sourcing makes it more likely that people will find it. That could be expensive (e.g. when some BigCo owns the copyright on code they have been selling for decades) and/or have even more serious consequences (e.g. when there's a GPL-licensed code fragment in there, and they linked it with a part they want to keep commercial)
Answering that question conclusively can be very expensive. They may not have a full history of the code, and if they have, it may not contain all metadata needed.k
They may fear releasing the source opens them for patent lawsuits or may have patents they aren't willing to give up, and fear that open sourcing it without any patent clause will not give them much goodwill.
Copyright. That intern who worked at SGI or NetApp who (un)knowingly reused some of a project they had on their laptop at their new place of employment that was actually software owned by their old place of employment. Scrubbing that or getting proper (open source) licensing for that bit, along with all the legal headaches that entails. Remember that we're talking about code that has been around since effectively 1982 with Sun UNIX 0.7 (or SunOS 4 as Solaris 1.x in 1991).
Licensed from other. I recall that various parts of non-Sun operating systems had licensing for parts of NFS. It wouldn't be surprising to find that parts of Solaris had licensing from other companies too. Including code directly licensed likely wouldn't be compatible with the license from the other company. Removing the licensed code to make it linkable is an option, but a time consuming one that diminishes the value of the overall project ("what do you mean I need a license from HP for something that DEC wrote?")
Your patents. Some open source projects have patent clauses with them. Sure, you can do the stuff to license those patents under the terms of the open source license... or chose one that doesn't have them. The former isn't at all in the interest of Oracle; the later is "Here's some BSD code... we don't know what patents are in there, but if you use them we will sue you."
Other patents (part 1). Surprise! In open sourcing the software, it is discovered that some intern reused the methods learned at another company in part of the product that has made its way to today. Now you've got the lawyers looking for blood for a decade of royalties.
Other patents (part 2) Recognizing that NetApp didn't want ZFS to be open source and there were some cross licensing aspects with Oracle with the ZFS settlement... they'd probably have something to say about it. There's probably some WAFL patents in it now.
Competitors. There are things in Solaris that would help competitors to Oracle products. Yes, open sourcing with a copy left license would mean that those competitors would be more challenged to use the software, but its out there.
Parteners. There are things in Solaris that help partners. Open sourcing Solaris, while providing good will to the community reduces the leveraging power with the partners for good deals. Some of those partners may also be interested in parts of Solaris not being open source.
So... nope. There are lots of legal issues and many business reasons to keep it closed source.
Consider also that this is development from March 2010 (first release under Oracle closed source - prior to that it was CDDL) until October 2015 (the last release) and the amount of value that has been added compared to the amount of effort for the above.
