I think it can come from natural curiosity, strong fundamentals, and experience. Search for problems that don't have canned answers and try to solve them.
I think anyone can improve if they're willing to put in the time, though of course natural ability does help.
Think about learning to write a compiler for a programming language. You'd have to learn:
- Computer architecture
- Assembly language
- Caching
- Recursion and Parsing
- Type theory
- Translations
- Performance testing
Or attempt to write your own database. That would cover at least:
- File Systems
- Transaction semantics
- Algorithms
I think tackling problems like this would help a great deal. Or just look for areas where you don't know how to do something, and work on them.
Things like:
- How deeply do you know your favorite programming language?
- How easily can you setup a Linux server to host a web application?
- Are you an expert at your editor?
- Can you write a usable GUI, either for the web or a phone, etc.