I think chukkle is saying that you are part of the four of us, which is kind of what "Is it just the four of us?" means, to include oneself