Interestingly enough, I never really listen with patterns or forms immediately in mind. If I hear something I like, my ear latches onto it. What that thing might be is very... broad. I have a tendency to play genre roulette when I'm listening to music because my expectations of how melody, harmony, and rhythm/meter work together are fluid. Whether or not something is challenging to listen to depends on how you choose to listen to something.
For instance, comparing two figures like Bach and Merzbow, one of the two is typically considered "easy listening." The other, not so much. You have a cultural context that makes Bach's music far easier to digest than Merzbow's. And yet it's extremely misleading to paint Bach with the brush of "easy listening" when you have works of his like
The Musical Offering that pretty solidly demonstrate that Bach is anything
but easy.
Obviously Bach and Merzbow are two extremely disparate musicians, but the point is that music is only "easy" or "difficult" because of relative exposure and familiarity. Western music hinges on a combination of melody and harmony, whereas rhythm and meter tend to be very simplistic and/or static. To many Western ears, as soon as you remove traditional notions of melody and harmony from the equation, music will tend to move into the realm of impenetrability. When you understand that limit and shift around it, it starts to blur and fade. Or at least, that was the case for me.