Very true. There are two schools of thought there-
The first is that our ears have been trained to hear tonal music from the music from about 1400's to about the early 1900's. Atonal music and music using things such as quarter steps has sprung up. Because this is new to our ears, we don't care for it, similar to any new variety of something - food, art, etc. It takes time to adjust.
The other school of thought is that people like the ones we have now because they do sound good and aren't full of dissonances. The octave is a nice 2:1 (1:2) ratio. This is pleasing. A perfect fifth is 3:2. These are relatively simple. The more dissonant you get, the crazier the ratios. A perfect fourth is 4:3, while a minor second is 16:15. The most dissonant interval of the 12-tone language is the tritone (augmented 4th / diminished 5th). This is a ratio of 45:32 or 64:45. With the numbers alone, it's easy to see why dissonant intervals, and therefor atonal music, doesn't work well.
Further, researchers have found that newborn babies, when played lovely basic two-note harmonies, such as a perfect octave or perfect fifth have been relatively passive, but when played more dissonant intervals have been found to have elevated heart rate, and have become more uncomfortable. We're talking newborns here, so there's something to be said about nature vs. nurture.
As far as the realm goes with "we're just not used to it yet", a terrible argument is "we're not used to eating rat poison either".
