Well if he's still owned by his old master, he wouldn't have to be confined to the Coliseum barracks.  In fact, the Coliseum would treat him as if he were a willing athlete.  Even though he's a slave, the Coliseum doesn't own him so they have no rights over him.