What's preventing them from serving a backdoored version? xz was open source as well, that didn't stop the backdoor. There might be reproducible builds on android, but you can't even inspect the executable on iOS without jailbreaking.
Signal designs their systems from the ground up to deliver verifiable trust mechanisms (via remote attestation) along with data minimization/zero-access encryption techniques.
It is neither against the signal software’s license, nor it is against the signal service’s terms of service.
This is a false meme spread because the Signal founder (who is no longer with the company) didn’t like people making forks without changing the API server URL and running their own servers.
Open source software doesn’t work like that, however.