undefined | Better HN

0 pointsopenasocket9y ago0 comments

> Are you saying that doing reverse lookups on every IP address, where some of these IPs will have many virtual hostnames, is easier than extractng the plaintext hostname from a certain offset in a Client Hello packet?

If the SNI info was at a fixed offset in a packet, it would be easy. But, per the RFC, it goes at the end of the client hello, after the list of supported cipher suites and compression methods. Not only does that mean it's not at a fixed offset, the actual client hello message may not be contained in a single packet, but rather several. So the ISP has to gather the packets and put them in order to re-construct the TCP stream, and then compute the offset. That is not trivial to do, especially at scale. Reverse DNS lookups are much much easier. Trust me: in my work I've helped implement both TCP sessionization and reverse DNS lookup infrastructure, and the latter is far more scalable.

0 comments

gwu789y ago

Are you saying that programs that extract hostnames like "sniproxy" cannot scale?

And you are saying that all hosts have set up reverse DNS and the data is complete and accurate?

openasocketOP9y ago

No and No.

I'm talking about a hypothetical ISP that wants to extract all the hostnames its customers are connecting to. It has to analyze the traffic off a live stream and re-construct the TCP stream to do this. Rebuilding the TCP stream on a 100Gbps switch is pretty hard to do. Something like "sniproxy" is only extracting the hostname for all traffic connecting to it, so it doesn't have to try and re-build the tcp stream.

For the reverse DNS stuff, yeah you can't count on PTR records. The easiest thing is to use a third party like Domain Tools (https://www.domaintools.com/), or you can roll your own. The quick and dirty way to do this is to get your hands on regularly updated zone files with all the hostnames, do a DNS lookup for that domain name, and store that data in an index. Assuming you get regular updates to your zone files the daily load is manageable. From memory, for .com you only need to evaluate about 400K domain names a day.

j / k navigate · click thread line to collapse