I imagine most Scavs know one another.
Other than that, some might know Cass' trough Jacob, or some other affiliation. 
either or.