If the words are just unaligned, you not only run into issues spanning cache lines, but spanning pages as well. The prefetcher can detect general linear patterns (+/-c, etc..) after a couple misses, but it can't prefetch across pages. Misses are still an order of magnitude slower than hits though, and this would imply at least two misses per page (~100 cycles), plus at least the number of pages you span. Assuming they were tightly packed, and depending on the number of instructions per value, it can nearly double the cycle count. Thankfully, unalinged access is generally as fast as aligned access with Intel though.

When you span the word size (e.g. add a single bit), you not only add two (dependent) instructions to access the value (consume a register), but without a linear access pattern (i.e. id index access), you can add anything up to ~200 cycles per value access. So, depending on the number of instructions per value, this could be anywhere up to 200x slower. This also doesn't consider the other possible issues like polluting the cache.

After re-reading what I wrote, it is kind of misleading. Yes, CPUs generally can't work with anything smaller than a cache line (see coherence). Although, you don't need to wait for an entire cache line to transfer before you can use a word on the cache line.

So, where values fall on (across) cache lines does matter. Straddling word sizes has additional problems, and generally implies straddling the cache line size as well (unless it's a multiple).