Well, know what your secret sauce is. I think performance is really the best differentiator. Make a fully behaviourally compatible (maybe not bug for bug) version available and then sell a proprietary faster version.
Think an compiler that doesn't due any optimisation and outputs naive code. You know have a useful OSI project, and a clear value add, and a clear boundary between the two.
This is really applicable for databases, and it still leaves you with something useful for learning small projects, and for developers to run locally on their own machines.