A few more...
The Gardiner Expressway, a raised highway along the lake shore into the downtown core. It's an absolute eyesore, traffic is usually bad, and it's crumbling right down to the rebar.
OCAD University - The premier art and designe college in Canada. This is one of their buildings (it's a small school). Locals call it the "pencil box" and I can't decide if it's an eyesore or a great architectural and engineering accomplishment. Perhaps both.
Maple Leaf Gardens... now practically gutted and most of it is a supermarket now.
My vote for the ugliest, plainest landmark building in Toronto goes to the US Embassy. Sorry, 'Muricans.