Denver, Salt Lake, Boise, and Reno are pretty good. I'm not sure that they fit in your "without temperature extremes", though. The upside is, with the temperature extremes comes some pretty good skiing.
Without temperature extremes... maybe San Diego or Seattle. I'm not sure that either of those are great for affordability, though. San Diego might be better than Seattle, but I'm not sure.
My experience: I live in Salt Lake. World-class skiing is 45 minutes from my door. I can literally wake up, look at the weather report, decide to go that day, email my boss, and go. I don't waste half the day getting to or from the slopes.
In the summer, there's a lot fewer mountains than in Colorado, but they're closer. There's a couple of good trout rivers an hour away.
Moab is four hours away. Yellowstone is 6 1/2 hours away. The Grand Canyon (North Rim) is 6 1/2 hours away.
If you want to get away (like, really away), in two hours you can be the only person in the surrounding square mile.
Salt Lake is pretty affordable, at least compared to NYC or the Bay area.
The weather, though... it gets up to 100 in the summer. It can get down well below 30 in the winter. And it's dry. 20% humidity is a regular thing in the summer. 12-16 inches of water in a year, and some of that comes in the form of snow.
Also, Salt Lake sits in a valley. That valley has a nasty tendency to trap air pollution. It's not as bad as Los Angeles, but it's still an issue.
Oh, yeah, and the valley floor is at 4200 feet elevation. The mountains go up to 11000 feet outside of town. You may be in good aerobic shape in New York, but expect to take a little bit to get adjusted if you come here.
