But you are already trusting Apple to manufacture your device and the operating system the app runs on top of.

Apple owns the hardware, OS and distribution, they can do whatever they want. Your signature is neither here nor there.

The only thing it can do do is show Apple you compiled the code.

There's no way you as the dev or the end user can verify the installed software really originated from you.

Yeah, it's not like Apple is in charge of the entire operating system running the app and verifying said signatures. Having them sign your app is definitely what gives them too much power.

Because it is MY source code, I don't want to send it anywhere. (Yes I do use GitHub/Bitbucket, only for public code)

For him, and imagine how Microsoft or Google would feel if they have to submit their source code to Apple...