> Are you talking about the ability to NXDOMAIN [star].example.com while also whitelisting this.example.com?

Yeah, you can totally do that! Details are here [0][1], but in your RPZ zone file, you use a CNAME with a value of . to return NXDOMAIN, and a CNAME with a value of rpz-passthru. to process the query normally:

  ;allow www.sinfest.net, but deny all others, including sinfest.net
  www.sinfest.net CNAME   rpz-passthru.
  sinfest.net CNAME       .
  *.sinfest.net CNAME     .

And to demonstrate:

  $ dig +short www.sinfest.net ; dig +short sinfest.net; \
  dig +short www.sinfest.net @8.8.8.8 ; dig +short sinfest.net @8.8.8.8
  64.29.145.9
  64.29.145.9
  64.29.145.9
  $

If you're interested in a complete, but simple RPZ zone file I can provide one. If you have more questions, feel free to ask, and I'll try to answer.

[0] http://www.zytrax.com/books/dns/ch7/rpz.html

[1] http://www.zytrax.com/books/dns/ch7/rpz.html#rpz-examples