Hm?  According to the wiki, it takes .75 seconds to break cobblestone with a stone pickaxe.  Perhaps my [url=http://minecraft.gamepedia.com/Breaking]source[/url] is inaccurate?

Anyway, Slayer would rather target the annoyance first so as to have a good fight with the remaining individual, though it does leave him open for a potential cheap shot from James.