Well, people do have differing opinion about everything. One might think Witches are ok if they are harmless and all, and might grow up to actually be nice, but others might want to cull them before they grow up and become powerful enough to become a threat.

So, you might think its weird, but what of our characters? Each of them are unique in their own ways and form their own opinions in different ways. So being thought of as an idiot is perfectly plausible. Remember it is our characters' opinion, not necessarily ours.