If not ex-military, I'm maybe leaning more into the idea that he has a past life in organized crime; really favoring a relatively violent/dangerous history to have brought him to where he is now. Or maybe even some kind of occult involvement.