[@Fetzen] Mayhaps if you go the Skyrim route, one of his fellow adventurers was accused of Talos worship by a Thalmor patrol and everyone with him was seen as an accomplice? It might be able to keep your original idea intact.