Well, you'll definitely enjoy the scenery while traveling horseback in Skyrim.
The worst thing about horses in Skyrim, though, is that they have a tendency to just wander off when you dismount. And you have to dismount to harvest, loot, or protect yourself. And they don't really run much faster than you can on foot.
For me, horses are just an added complication. I guess that would be OK if there was a way to actually address some of the issues, particularly being able to tie your horse up somewhere, like to a tree or something, so he doesn't run off on you and you end up spending the rest of the day trying to find him.
I just abandoned my horse after all the retardedness with him.