I am not sure why population size (if you could isolate the effect of just "size") should matter so much. If you tax 1 million people 10 dollar, you get 10 million dollars f.ex to fund public schools. If you tax 10 million people, you get 100 million dollars. In both cases, the return from the government to the people is generally the same. That's a very simplified example, but you know. Rather, I think other things turn the government into a "they" rather than an "us". Class differences, cultural differences, money spent on war/debt (little or no return to the people), the fact the minority who don't get their wish in a country of hundreds millions population often equals a huge amount of people, etc.
I could write a lot about how I think all that affects the US, but I'd be getting too far off topic. To get back on track, if you don't see the possibility of cannabis taxation as a return to the people, then I'd think that would sway most people towards a no. I'm not that sceptical to taxation, though, I think it can be good. Here in Norway, they just raised taxes on "unhealthy" foods, something I'm fine with. Not only does it generate money to the government (us), but it should help give people a little extra motivation to eat healthy which will be good for all of us. I just wish they'd reduce taxes on healthy food like fruit or vegetables, but I think that's right around the corner.
Anyways, cannabis taxation could be a return to the people, although I agree with Duga that I'd be a bit pissed if my tax money funded a war. I know how expensive some of the cannon ammunition on our navy vessels are and the thought of a year of wages worth of tax money disappearing in a single cannon blast is in itself frustrating .. And then of course you get the associated suffering.