I guess I'll add in my piece. Yue will tolerate everyone as long as they don't get between him and his orders of protecting the Avatar. Except maybe for Tenkai due to her dislike of none-benders. Even then he'll simply regard her a bit more coldly than the others.

Due to Yue's more serious nature, he'd probably be most annoyed by Masaru, despite his orders to protect him.