But it's difficult to click the leftmost or rightmost character to position the cursor.
So after clicking, I usually tap Ctrl+A (Cmd+A) to "select all" anyway, and then tap left-arrow or right-arrow to deselect and jump to the chosen end of the URL.
Cursor keys only junp to the end if it's selected. My keyboard doesn't have dedicated home/end keys, but I'd probably use the select-all-then-arrow trick anyway.