It's not necessary to have a headphone amp in terms of volume.
As far as sound quality is concerned, think of it in terms of car audio systems where people buy amps for component speakers like tweeters, mids, etc... A dedicated power source provides cleaner sound even at lower volumes and the speakers benefit from having enough juice to drive them.
If you're not worried about "audiophile BS", then don't get a headphone amp.
__________________
|