Sure, by themselves, but replicate that across everyone repeatedly downloading the library in various build processess, in various projects and with different versions.
Sure, by itself a sourcemap might not add up to much. But then there is various projects with different versions, who download the library repeatedly in different build steps. Then it starts to add up both in bandwidth, and in storage taken.
Luckily, there are still package and library maintainers who care about making things less, instead of more, both in architectures and size.
> Just so that people can modify the source code directly a little bit easier?
This is a real thing that is useful to think about, how to make your software easier to change in the future, while just solving the current concern without over-engineering the solution.
