It's kind of like refusing to have a fire extinguisher in your home because you heard the dust poses a risk of cancer (note: I'm making this up for an example). Your house catches fire and you lose a large chunk of your house because you had nothing to put out the initially managable blaze. Do you still feel anti-fire extinguisher in such a circumstance? 

Now replace fire extinguisher with vaccine and fire with disease and it's the same argument. Don't ignore an actual legitimate threat for a "just in case, probably not".