I always thought skendar's clothes and weapons were made frome his essence, meaning that if he lost his sword his maximum size would decrease.