I mean we're not supposed to have much in terms of memories to start off with anyways. 

I've kinda of leaned into memories of his time in the service being the key memories to his power. It'd take some doing to re-imagine him another way.