No, YouTube does have audio-only streams. You can use "youtube-dl -F [url]" to view the available formats and their codes.
They're meant to be paired with the video-only streams used for some resolutions (most notably 1080p). Even browser extensions such as YouTube Center can download the streams, and they certainly don't use ffmpeg.
(In fact, I do use ffmpeg to combine audio-only and video-only streams via youtube-dl, e.g. "youtube-dl -f 137+141 [url]" which takes 1080p video and 256k aac audio.)
