You kinda went awol with him at a moment where both him AND Frisk would've been freed, so he had to stay behind to keep an eye on Snowdin.