Well I expect ref's were from the outside at one point. Exposed to the horrors of the wasteland. They were either victims subjected to the torments of creatures like raiders/fiends, super mutants, or worse they were a raider/fiend forced by some lunatic to kill their friends in a deadly brawl just to see who would gain the right to join the gang. Plus there are always lies and rumors most likely spread by the vaulties. Telling false stories about them just to frighten the others.