Well, I'm not talking about a major refactor (squash them when you see them). This should be an enum, this is named poorly, this variable is scoped wrong, inline sql when should be a sproc, that kinda stuff. A few lines a fix, the low hanging fruit.
Major refactors certainly need to be accounted for in a sprint(s), but keep features a priority when needed in sprints.