I've figured out a way to make Harrison a bit more durable I think. Make his memory ghosts able to trade places with him. So the he is the invisible memory ghost while whatever historical figure puppeteers his body. 

I'm going to bed now so please let me know if it's any good or any changes to be done (I'll also accept suggestions).