I actually wouldn't mind seeing the fast pass go away. It does really slow down the stand by line, a lot on some rides. While it's great when you have a fast pass, not so fun to stand in line even longer when you don't! I'm not sure all those people would be in that line anyway, they may be on different rides, having that one time slot to return attracts a lot more people at once to a ride. Not to mention even if they were all in the stand by line, they still wouldn't be cutting in front of everyone else that has been waiting for ages! I personally never found the system to be very fair.
Even though we do use it here and there, it's not something I'd miss.