it's paranoia. i went for years and years without a virus scanner. started using one maybe a year ago because i liked its firewall and decided to just go for the whole suite (ESET). it has gone off a handful of times, but it's usually just quarantining keygens that i run in Sandboxie anyway so it's more annoying than anything...
as for my firewall, i never notice anything either. inbound stuff doesn't really get past my router, and i keep all outbound under tight control...
it's more about being aware about where you're clicking on than anything else