What exactly is making you feel bad - what are you thinking when you have that feeling? i.e. what thought is stopping you and why is that bad?
[OK, I'm guessing in the following - you'll find the true answer in the above questions] You say you feel bad because of a lack of mastery, but what level of mastery counts?
I think it's the lack of a goal: a defined point where you say "ah! I've done it!", instead of a continuous never-ending slog. There's always more you can do in the bigger picture, but you can choose a goal at a point that in some sense is a completion. Artificial in a way, but also on top of something, a logical boundary.
That is, a specific doable task that can be done pretty quickly; with some challenge, not too much - a level that, from your experience, is definitely doable by you; that is clear and definite, so you have a direction/guide to keep you on track, and so you know when you've reached it (so you can stop, and see how far you've come, and a take a rest to choose the next one).
Finally, it must have some meaning to you - it could be to just learn some technique to some standard... but better is an actual project that you want to do: a simple parser, an animation, a tool that you'd like to have, implement an idea, copy a game or tool or library or framework or language or app that impresses you (or some part of it). But note that it must be doable, by you, in reasonable time. So, if too ambitious (or you realize it's too ambitious as you go along!), break out simpler and simpler versions of it, or sub-parts of it - provided it still has meaning, a definite end-point, and is doable.
Of course, as you say, you are learning stuff in this spiral, and you'll get cross-insights from doing different things. It's just that you feel bad about it - I think having goals would help. [But see the questions above...]
What the OP describes is a system which he acknowledges improves his expertise in the studied knowledge.
My opinion is that system has the advantage over goals in that it is an automatic and permanent drive to stick to some rule. It can be learning, exploring new ideas, do fitness, healthy food, keep a good weight, etc.
Goals require a boost of effort and focus wich can only last for a limited duration. You feel good if the goal is achieved and bad if not.
If you work with goals and sprints, there is a high risk of burnout. So better go into system mode for the long run.
The change of topic of interest is something I saw my self and was also puzzled about. I attribute this to some "fatigue" of the brain where it's learning capacity is kind of saturated. Changing topic of interest, changes the neurons and connections in play which let some of them to rest and activate others. Learning is also a matter of repetition and pruning details. You get this by switching back and forth between topics and forgetting the details in between.
Another benefit of having different topic of interest is that at some point we will make the connections between them. We'll use the knowledge gained on one topic on the other. This is powerful and by the combination of topics explored it will also make you most probably unique.
pain [...] consistently high [...] don’t have the energy or enthusiasm
The key idea here is that a goal not something to strive for, but a limit. Not to force you on, but to let know when you have accomplished something - when to stop! Now, if you're a perfectionist, you may say, "but there's so much more! I've barely scratched the surface. I've done nothing! I'm not worthy!"
However, if you had a goal, and you can remember how it looked when you set it (that it did seem challenging), then when you reach it, you can compare where you were, and where you are now. And you are forced to acknowledge the fact that you have accomplished something.
This gives a flood of satisfaction, like it or not.
Now, turning to "sprint" vs. "system". Firstly, I stressed that the goal must be doable. There's minimal risk that you can't complete it, if it is "doable". And if it's too hard, pick a smaller one.
Note also that you can shift topics between goals, just as you can with a system. I'm talking about small goals, not huge overreaching ambitious goals. Baby steps - but steps nonetheless!
Sprints seem very compatible with human psychology: movies, books and music usually consist of alternating tension and release. There's a series of difficult obstacles, not just one, but a series of problems and solutions. People really respond to this.
Curiously, it seems that literal sprints (i.e. running) are also much better for building fitness than consistent paces. And they even have "sprints" in some agile methodologies (not sure if that counts for or against my argument...).
Finally, there's absolutely a magic in programming, that you code something... and something happens! It's a child-like thrill that (in my opinion) it is absolutely crucial to respect and preserve, especially as ones goals become ambitious and abstract. Fred Brooks talks about the "sheer joy of making things" - and compares us both to a child making mud-pies and to God creating nature. (just found an excerpt online: http://cs.calvin.edu/books/processing/references/brooks/myth...)
I would guess that, evolutionarily, the human body and mind is adapted to sprints - and this is what makes goals feel so satisfying to us, just as we enjoy food, water etc. The "system" approach makes logical sense, and what you say about it seems true. It's just that you haven't addressed how it engages with human motivation (but assume it is "an automatic and permanent drive") - which is the issue OP is struggling with.
Lastly - I think OP's personal solution will actually come from the thoughts he has when he feels that pain. Part mistaken, part profound - only OP will know which is which. I don't claim my suggestion is right for OP, just adding another perspective.
Anyway, I distinctly remember learning about Clojure, and being completely enamored with the ideas Rich Hickey talked about. I would still get bored of doing 4clojure problems, but instead of abandoning them for a different subject entirely I would watch a ClojureConj video, read a blog on a Clojure library, or even just practice some of the tooling like emacs key-chords.
I believe the learning process is better when it is erratic, both for you're own sanity and the overall completeness of a knowledge domain you can acquire after jumping around from topic to topic.
5 months in, I am trying to push Clojure/Cljs ideas into the C# shop I work at, and upholding my promise to do SOMETHING regarding the language or tooling every night.
Our brains seem to work best this way, storing an idea in our subconscious after rigorous study allows us to process it in the background. I find myself frequently daydreaming about language constructs or architectural designs.
Would appreciate your feedback on that!
I think on some level you cant make yourself feel bad for being a jack of all trades, and you can setup methods to reinforce going back to the well of whatever you were working on.
One of my compatriots who does this has a dedicated but somewhat more closed off friend who doesn't look around for new things. They challenge each other to see who can accomplish the most in some specific language, style, whatever . The flighty but open one now is challenged by someone who is more dedicated to the task, and as they keep it light hearted. Together they make some interesting stuff and are a pretty good team.
I think there are only two things.
1.) You shouldn't feel bad about it, and now you are realizing that it's an overall upwards spiral after all, you probably don't feel so bad about it! :-)
2.) You forget to include a lot of other things. What about sporting activities, or social activities. There was another HN post on "investing time", that explains that concept. I can tell you my spiral is way bigger. And especially Wikipedia I see as a big corner stone of learning things. It's one of the fastest ways to get a feeling for what the Bellman equation is, or convex optimization, or the different notions of consciousness, or the axiom of choice in set theory, or a morphism in category theory, Feynman diagrams, or a renormalization group, just to list a few things I've checked in 2013. I bet you could follow my intellectual spiral by the specificity of topic terms I check on Wikipedia. :-)
The world is a wonderful place, there is so much to discover! You will never be done. It's great to not just accept that, but to appreciate it!
Maybe you need to broaden your horizons a bit to find something that truly interests you and dramatically increases the things you can do as an engineering/scientifically-minded person?
Leaving something incomplete makes it interesting, and gives one the feeling that there is room for growth. Someone once told me, "Even when building the imperial palace, they always leave one place unfinished." In both Buddhist and Confucian writings of the philosophers of former times, there are also many missing chapters.
This is because a "Dabbler's" skill goes back down to 0 in the short term, and this is not what is happening to the OP.
One of the key things here is to think about what you want and clearly define your goals. But you may find yourself in the situation when it's hard to understand what you want. Then, I think, it's ok to play around with small projects shorterm without strict commitment. This helps to try many different things and finally find what you like.
The focus here being on the word "problem". That's where I find I get the enthusiasm to learn something or finish that I started, because the problem is interesting or useful.
The main problem with "the spiral of learning" is the wasted energy required to switch tasks.
If I could just stick with one thing until I mastered it, I would be much more efficient. Trying to go back to things I haven't done in a while requires a long period of time for re-familiarization with material I already used to know.
You should try and do/create cool/interesting things/projects which help you learn the skill a little more.
Rather than going at it with the intention of learning xyz