The vygorns saw the group in front of the cave. If cora wasnt there, or beau, they wouldn't know. The people standing by the cave must have heard/seen them. Unless they were listening to music with headphones, while either looking the other way, or with shut eyes, but i doubt anyone was doing that.
And the Vygorns didn't necessarily come from Riverdale. Maybe they approached from the south, not the east.