Yeah, it is probably a scent thing but that is very weird. Either that or you might look enough like one of your family members that the wrong scent throws them off
One of my cats literally watches out the window. Not like lounging in it but seriously watches out the window and when nothing else is happening, they leave.