You see the most extreme versions of dangers of this in stories about governments and older companies having to pay insane salaries to bring FORTRAN or COBOL developers out of retirement to keep systems running. If you keep doing the simple solutions within the existing system, you risk creating a system so inbred that only the folks who built it can maintain it effectively.
For less extreme setups, it's still a balancing act to consider how much your unique and specific solution that is the simple option for you company starts closing you off of the larger hiring pools in more common technologies and patterns.