Yes, that puzzled us too. The castle might be water-inaccessible, but it is not that plausible for the inner city. A sea wall could make sense, though. I.e. A wall right behind the wharves. Maybe a smaller inner harbor with a raised chain to seal out anything above rowboat / small canoe in size.