As good as he was in Incredible Hulk, rumors were even then that he was a pain in the backside. Based on his performance alone, I'm sad that he won't be in Avengers. But on the other hand, Avengers is a team movie both on camera and off. If he can't play nice then I don't blame them for looking for somebody else to take his role.
Ed Norton appeared to be quite the control freak in "Incredible Hulk", demanding script changes and re-shoots. I think whatever happens, the studio's primary goals will be making a great film and keeping Robert Downey Jr happy!