All I can think of is that an "IP address" does not have a port component and that's all that IP deals with.
When a machine receives a packet, after the Ethernet and IP layers have made sure the packet is addressed to this machine, the very next thing that happens is checking what transport layer implementation should receive the packet - and this is done based on the "transport" bits in the IP header. If the packet is, say, TCP, then the TCP layer starts reading its own header and finds out the port number, and proceeds from there.
OH woah woah. Okay that's the root of my confusion then. Nevermind. I thought the OS network stack exposes ports and packets come on to ports
Nevermind then :)
Networks are built in layers. There's a physical layer underneath IP, then there's IP, and then there's TCP and UDP on top of IP.
The OS network stack has components that handle all of these layers. That's why it's called a stack.
Port numbers are part of the individual protocol (TCP or UDP) because there are a lot of things you can do with networking, and port numbers don't necessarily make sense with all of them.
For example, when you ping another computer, that uses ICMP, and there is no need for ports with ICMP. You're pinging the whole computer, not trying to connect with one of several applications running on it. So ports are not really needed.
If you mean socket as in `my_socket = socket(AF_INET, SOCK_STREAM, 0);` that's TCP/IP, not raw IP, so it will have port numbers in the TCP part of the packet. `SOCK_STREAM` and `SOCK_DGRAM` respectively correspond to TCP and UDP. Raw IP sockets on Linux can be created by `socket(AF_INET, SOCK_RAW, protocol);` and that will have no TCP/UDP header attached by the Kernel after the IP header in the packet.
(googling a bit as I write, forgive errors)
The socket API has different types of sockets, typically you use stream (TCP) or dgram (UDP), but you can also get a completely raw socket where you need to construct your own ethernet and IP headers, or create your own replacement of IP protocol. So socket does not mean only TCP/UDP or something with ports, unix sockets are another example of this.