And it kills your sperm as well. I think that these are long-term effects though, I was talking about the direct effects (the dose probably matters as well).

It actually makes sense that the use of cannabis improves attention but impairs attention in the long run. Compare it with drinking coffee. The first time I drank a strong cup, I had a strong boost of energy and felt a little euphoric even. I am now a heavy caffeine user, and when I don't drink coffee I actually feel more tired than usual.

Of course, it's dose-dependent, because eventually the detrimental effects on attention and memory (as you correctly point out) will outweigh the beneficial anxiolytic effects.