If you are an artisan doing commodity work (read: bored at work, your open source project is fun), you are absolutely right. You are underutilized and easily replaceable, and your boss will want to replace you with someone cheaper. Regardless of the size of the counteroffer, you should leave.

If you are doing work that few others can do, and you make it clear that your only problem is money, they will accept that you looked outside just to get a market valuation. This is especially true if you accept a counteroffer lower than your external offer (i.e., "I like this place enough to accept $10k less").