The Imperial army is actually fairly capable. They have to be to keep expanding it's borders and keep the keep. the Stormtroopers are more of a faceless and nameless symbol of Imperial might. The hammer of the Emperor's will given form. It has less to do with being elite and more that people perceive them as such. the intimidation factor is part of the key to the Stormtrooper corps successes. 

The true power of the galactic Empire lies with it's navy which is by far it's biggest military powerhouse. It's ships are arguably the largest and most destructive ever built in the Star wars universe. 

In reality the storm troopers were 'par for the course' at best. They were originally the remnants of the old clone army. But as they started to age and certain problems/concerns with mass cloning became more apparent the empire transitioned away from clones(who were being churned out in ever shittier quality anyway) to willing volunteers.