If you really want to get down to it, there are numerous armies that fought with little to no armor. Greeks, Romans, a lot of barbarian tribes from around that time, off the top of my head.
Even after heavy armor was developed, the poorest foot soldiers had little in the way of protection besides what they could loot, sometimes nothing at all.

Also, Musashi defeated an expert martial artist with a wooden sword. Regardless of his intention, the fact that he used a wooden substitute MAKES it impressive. It's indicative of his skill as a high caliber martial artist.