My best projects have been full on XP projects. However, XP is a subtle beast. It's tempting to look at the 12 practices of XP and say, "We're doing XP. We will be awesome", but it doesn't really work that way (in my experience). The 12 practices in XP are "generative practices". What that means is that they are meant to lead you to a process that will be "hyper productive" (I think Ron Jeffries coined that term). The 12 practices are not the process itself. It's a bit like the difference between pointing at something and the thing itself. The practices only point to the process -- which will be different for each team.

And in case you are wondering what I mean by "hyper productive" (and I apologise if this is not the original meaning, but I think it is...): it means being able to go faster as time progresses. On normal projects it is easiest to work on the project when it is still greenfield -- you have no constraints to work around. As you work on the project, the decisions you make create constraints and make it more difficult to change what it does.

With a hyper productive team it works the other way around. Imagine working on an embedded system where you may have a compiler, but you have no standard libraries, or frameworks or third party code to help you. It's very slow because you have to build all of the tools that you need to go forward. You have to consider all of the abstractions and build designs that make it easy to follow those abstractions. On modern systems we have lots of tools that allow our "green field" projects to be boot strapped quickly. Now imagine that you are constantly keeping your code in a state where every future change is like that. As the code base gains capability, it actually makes it easier to make the next thing. This, of course, is the holy grail of programming :-) You can make a lot of money selling maps to the holy grail, so you shouldn't believe me!

With that disclaimer, I will rashly state that I have been on "hyper productive" projects before. But it's a pretty ridiculous tightrope. For anything I could say, there will be someone who will reply, "Yes, that sounds great in theory, but what if X happens? It will all come crashing down". Those people are correct. That's why XP is not the 12 practices. They are just something that will hopefully make you aware of what you need to do when X happens.

I don't think XP is the only way to get to "hyper productive", but it is they way I have used to get there. I don't know any other way, personally, but that doesn't mean that it doesn't exist. Usually when people ask me the question you have asked, I direct them to XP and suggest that they study it thoroughly (including running projects in full-on XP mode). The problem is the "Agile" attitude that people usually bring with them. Because they think, "Well, that doesn't make sense", or "I can't do that because Y" and they use their experience to insert something else. In the end they create a kind of "We tried baseball and it didn't work" situation (to borrow from Ron Jeffries again). It's a catch 22. Blindly applying the XP practices won't give you success, but you probably can't get the experience to understand where the practices are pointing without doing just that.

Even for me, where I've had success with those techniques, I often find myself in situations where I can't actually get there from here. So I try things I haven't tried before (usually without success :-) ). Or I wait until the situation changes and I can get there from here. It's really hard.

I'd love to talk more about politics and the difficulty of just getting people to walk in the same direction. For example, I often tell the story about 5 people all walking in different directions. One of the people is walking in the right direction, but every one of them thinks they are walking in the right direction, so they can't agree. Let's take the position of the person walking in the right direction. If that person just pushes on, then they will arrive safely, but the 4 other team members will fail -- and hence the project will fail. So it makes more sense for everyone to walk together even if it is in the wrong direction, because you have a chance that you can change course and succeed. If everyone goes in different directions, then you will fail, almost by definition -- even though one of them might be right.

The problem about talking about things like the above is that I don't fully understand what to do. Kent Beck's famous C3 project was a technical success (though, I'm not sure it was every fully implemented) because all of the team members were fed up with the previous process and were willing to do pretty much anything else. My most successful project was actually similar -- the previous manager had been overbearing and couldn't work with the team members. He was removed from the project and I was inserted "to make everyone happy". So I did -- they wanted to try "this XP thing" and my job was to make them happy. (Side note: you usually don't go far wrong if you make your workers happy).

But there is far more to it than that. I always say that the most dangerous actors in a project are senior management. Virtually every failed project I have been on (and I've been on a lot) has failed because it was cancelled by senior management. Now, I say that partially tongue in cheek because projects don't fail until they stop and they don't stop until they are cancelled :-). But it's actually true in a very real fashion. Most poor choices actually stem from trying to appease stake holders or managers who are worried about something. Again, I will say that XP can really help you there, but I will forgive you if you look at the 12 practices and say, "I don't see how".

I didn't really intend to write a long response to this question, but... you can see what happened. It's really a difficult question to answer and I'm not sure that my answer sheds any light. However, just keep pressing on. The holy grail is there. I've seen it. Your path will be different than mine, but you can get there if you keep believing.