I think it's easy for people to jump to conclusions. Reading studies on this, I have read instances where people believe they can hear a difference between bitrates and when they do a blind test, it turns out they can't. I myself turned out to be worse at this than I thought.
In order to be certain you can or cannot, you have to subject yourself to a blind test. Hearing a difference is not really the question either - the real question is can you hear which one is the higher quality one? I know that I'm not as good at this as I previously suspected and I think many would be surprised.
edit :
Actually, I can set up a blind test in the games forum, I'll encode a couple of mp3s in varying qualities and then people can download them and check if they can guess which ones are the correct ones.
.. Although they'd have to promise not to look it up on the file, though - or filesize. Hm.
I'd have to find a way to make people stream it I guess.