Yeah, Saber was an exception because she wasn't actually a proper Heroic Spirit at the time of her summoning; Grail taking her from just before she died and all. Every other Servant is/was a copy of the original Heroic Spirit and wouldn't retain memories if they were summoned again.