Probably the longest running and definitely most stable game I've ever been in is three players and four characters. Also a modern military one, so personally I'd be fine with just the three of us and growing if more people come later.

Quality over quantity sort of thing.