You dont need to go open source to allow self hosting, and, youll likely lose money that way. Its entirely possible to allow self hosting without being open source. Plenty of other projects do it.
That being said, self hosting doesnt solve all issues.
You, and whoever uses this product, will need MANY IPs because of blacklisting and throttling limits applied almost worldwide to all mail servers. Theres a reason why the marketing spam hosts have their own /24 or /25 and have 80% of them cycling through usage as a mail sender.
Im not knocking your want to have something better than MailChimp and the like, but i feel like you didnt really dig into the real underlying tech and security before you went to work on this.