Turns out when you build up a character of you being overly serious with a side of self-deprecating humor, people are going to take your one-off jokes as overly serious. I really should've seen that coming.