You're right, I misspoke, "random" data is nothing, but "random" as in "arbitrary selections of network data" it will work on. And no, it doesn't unpack layers per-se, but it detects signatures and can be used to find patterns and eventually separate layers of protocols.

The two oldest and most successful methods that I know of are matching on payload length and models trained on the initialization of network application protocols, neither of which requires constantly re-sampling and re-classifying to get a hit. And blocking traffic is often more about terminating an existing connection once you detect something bad going over it (deep content inspection).

Yes, commercial firewalls do look for tunneled applications. Palo Alto Networks has a patent on it (App-ID), Websense/Forcepoint does it (Content Gateway Analysis), Cisco sort of implements it (Network Based Application Recognition/Application Visibility and Control).

A bunch of open source software implements it, too. Some commercial proprietary software is also out there, with PACE leading the pack. Wikileaks has one of their product data sheets: https://wikileaks.org/spyfiles/docs/IPOQUE-PACEProtAppl-en.p...

Honestly, a lot of customers simply force proxies on their users and inspect all their traffic and drop anything that it can't inspect, so there probably isn't a lot of commercial need for this. But it is out there.