What I've seen several times are engineers that are able to tackle problems that require so much creativity or skill that giving them to a less qualified engineer would most likely be a waste of everyone's time. In instances like that there can be a 10x difference in the value created by a quite good and an exceptional engineer. I've seen it in the static analysis software domain.
Arrive at 9:30, start out with 30 minutes at the coffee machine. They get mad if you discuss work at this point, or god forbid, talk about interesting algorithms or the like. They expect to be told exactly what to do, or they won't even sit down. If you don't give them an exact list of methods to implement in a class, nothing will happen. You can give them documents about the design, but they won't read them, let alone provide feedback. Not even after asking them 5 times. Similarly, when I started my recent project, I asked one of the customers for the canonical book on the subject and read (most) of it. I do not consider this special behaviour, but I'm definitely the only one to have done this. They treat this as normal, and if they make basic problem interpretation mistakes that would make you think a highschooler had mental problems, the reply is "it's in the design". Similarly classifying bugs correctly is impossible. A letter out of place in the UI they treat as equally serious as getting the wrong outcome in a financial calculation. If some programmer put "fucking" in some UI text (the one I talk about below), then we drop everything else, right ? They work to satisfy the exact wording of the policies, and never ask for (or tolerate in code reviews) exceptions. If you don't have 5 exceptions to the style guide and 5 things where they claim "sure, but the obvious unit test would be worthless, so I tested ...", chances are you don't understand the problem, don't understand the language well, or there is something wrong.
Of course, this is the "average" programmer at a place like a bank. I'm sure at Google or a few other places, it doesn't quite work like this. Being a 10x programmer is easy in a team filled with this kind of staffers.
I once had a guy that I'm totally unsure whether he was a fantastic programmer, or totally worthless. He'd be sitting behind his desk for the first week of the project. Almost without exception he wouldn't have his IDE open, but have an ipython notebook with some part of the problem open. Alarmingly often, he'd have hacker news, or dzone open and would be reading articles (I mean 50%+ of the time, not once or twice a day). Meanwhile his output in code was exactly zero for that week. Then, suddenly, in a little over a day (when he had claimed he could do it in an afternoon), he'd checked in the source for an entire module (that would have taken us probably a month to write, at least half a month). It wasn't 100% bug-free, but it was good enough to make the product functional. And while he took care of any bugs pointed out to him in minutes flat, for the rest of the week it was back to reading and experimenting (I think) mostly unrelated things. The problem is, this guy is not failure-free. He has these bursts of productivity, and sometimes he writes something beside the point (all programmers do), however it is really hard to ignore the week of wasted time when something he does gets rejected. We have a -sort of- working relationship, but it's not exactly smooth sailing. Any relatively independent and hard problem, he gets to do. And if something just needs to start working, we throw it his way. The result of that second part tends to infuriate other developers, but at the demo the UI works, with the project only half finished, which is something customers and managers appreciate. He tends to have his IDE open, modifying code, often while we're walking into the demo room, which doesn't inspire confidence to me. His reputation with the team is on a constant change. He's lazy, self-centered bordering on egoistic, can't follow instructions (granted, he generally makes good cases that the problem is with the instructions, not with him. But the frequency of him not following instructions is just so much higher than with the rest of the team it's not realistic). He ignores his coworkers, and of course they can't deal very well with the fact that there regularly are weeks with his productivity at zero. They don't feel to well about his productivity being off the scale for a few days either, by the way. Oh and once, he wrote 10 pages of code that implemented a complex algorithm, when we needed it, and all variable names were girl's names. Every single one. As in "for (int debby : kaithlin) {" type code. Why ? Because he had an argument with the architect.
So tell me, is this a good engineer ? Great ? Or a disaster ? He's not exactly a stabilizing force, that's for sure. There regularly are times when I wouldn't want to do without him, but most of the time, I'd like to make a hole in the window with his exact silhouette. More than one of his coworkers have mentioned something along the lines of "can't we fire him 4 weeks out of 5 ?".
But you know what, at the weekly group meetings, my team leader could clearly see that my bug list always was empty while my team "mates" lists were overflowing. I told the department manager I had a lot of unused capacity. Not once, but several times. It didn't lead nowhere. When I fixed unreported bugs or did the "refactor the whole program while fixing my 3 assigned bugs" plan I was heavily criticised for that. All my suggestions for how the systems could be improved or what features could be implemented was shot down because they weren't in line with the department managers great plan for how things should be done. It just lead to long dick-size measuring arguments.
So exactly what should I have done? Sit with VS open, tap keys and simulate working all day long? Helping my colleagues with their problems was out of the question. As they, just like your department and this guy, talked behind my back and despised me because I only had to work 1/10th as hard as they did.
Eventually I quit because the situation became unhandleable. I got verbally warned for showing up late and people stopped even saying hello to me altogether. I was sick very often because it was better than reading hn all day which actually is fucking boring. My new job is like a breath of fresh air because the old one was really driving me crazy.
Truthfully, I can understand how it can suck to be average and have to work with someone much more skilled than you. I would also feel threatened if I knew about a colleague that could replicate what I do in a month in a few days. However, that's what HR departments, project managers and team leaders are there for. It's their job to make it work and if they can't and istead lose their truly great engineer, they suck.
Much of the description (yours and the GP) is POV, though. I don't know you; maybe your suggestions for improvements were interesting engineering problems but horrible business decisions (and you didn't realize it); maybe your "several" requests for more tasks were actually 3 in 5 years, and were mostly actually scornful complaints about the low quality of your colleagues, though that's not quite how you remember them. POV.
There are more concrete checks, though -- can you list some of the things you did specifically to annoy/frustrate your colleagues or bosses, or even partly with that in mind?
E.g., did you ever implement a complicated algorithm with all variable names being girls' names? Did you ever intentionally fix this at the last possible second just because you enjoyed how uncomfortable it made the others? Etc..
Edit: My hope is that you won't come up with anything that silly, of course. I'm not actually trying to paint you in a bad light, just show how it's possible the situations could be quite different or quite similar.
Side note, though -- some situations really are untenable, some cultures simply don't reasonably allow improvement, but honestly most of the time I see one dev who's much more capable than their colleagues who is seriously resented that's because they are failing as a mentor... often because they simply don't think it's worthwhile ("I'm great, those people suck, and that can never change"). It takes some effort put into skillful social interaction, but most people want to improve their skills if given the chance in the right way. There will be some who just want to do exactly what they're doing -- and let you handle the hard problems -- and if you handle that relationship correctly they will appreciate you as well... but most resentment comes from people who want to keep advancing at a reasonable pace, and get some credit for doing good/interesting work, but instead they have the asshole super-dev who either says "oh, that's too hard for you, I'll do it" or "you think can do it, really?! well, go ahead, but don't come crying to me if you screw it up".
It is rather easy to check though - you should forget about time spent on work and need to compare the outcome, i.e. result, of this guy vs every other engineer on the team.
If the check above shows that this guy is better than everybody else, than this:
>> Alarmingly often, he'd have hacker news, or dzone open and would be reading articles (I mean 50%+ of the time, not once or twice a day).
and this:
>> More than one of his coworkers have mentioned something along the lines of "can't we fire him 4 weeks out of 5 ?".
indicates one simple thing - he is not managed properly by his manager and by the company.
It as simple as that...
Another observation: Being 10x is total contribution, not individual contribution. If you're fast, but others can't read your code, subtract that from the productivity. If you cause other decent performers to quit, subtract that from productivity. If you piss off clients, subtract that from productivity.
That's not to say one can't find a way to make eclectic people useful, but the smart jackass rarely ever lifts an entire team's productivity.
Sounds like he doesn't want to be working there. Sorry, but it's probably the right decision to encourage him to find a job that he wants. Elsewhere.
In a few of the behaviors he described, I find some sympathy. I do have a high opinion of myself, and at one job would end up in loud arguments with the CTO over how to do things. Sounds like this guy takes it farther, though, doing things that are spiteful when he doesn't get his way. That's a scary, pathological behavior, and that alone would make me think you'd be better off without him. I wasn't happy when I didn't get my way, but I'd suck it up and deal; that's the only professional response.
And speaking of pathological: It sounds like he's intentionally misinterpreting instructions when he can? Again, that speaks of passive-aggressive behavior. And he avoids other employees? Not to say that anyone is in danger, but you've painted a scenario that strikes me as the backstory for "the guy who cracked, came in to the office, and went on a shooting rampage."
Or he's just Aspergers. Can't really tell the difference from here.
This is a highly irrational conclusion, even with the meaningless caveat.
Does he have any chance of being promoted, for not being a complete moron? Or do you promote morons into lead roles, so they can hire even bigger morons, and make moronic decisions?
If he's not going to get promoted, then he's got no reason to not just dick around, and behave like an asshole. Yes, he's probably hurting his chances of promotion (and might risk getting fired), but he probably had no hope anyway (since there's no way you could afford to lose him by kicking him upstairs). And what are you going to do? Fire the only guy who can actually solve problems?
I don't really mean to do it that way, and I usually feel guilty during that first week. But apparently my subconscious brain is sorting things out. I try to work and it all just seems too hard and confusing. I make little attempts to start and they just seem wrong. And then one day everything suddenly clears up and it's all obvious.
His value to you depends on how you structure his environment to make doing a good job more fun to him than torturing his co-workers.
There's a degree to which putting exceptional people with people who just do the job can cause both to resent the other.
