With Jedha, currently the Empire is in the process of harvesting the crystals.  Getting into the temple is all but impossible.  However, as was mentioned above, smugglers hitting convoys wouldn't be that out of the question.  Stormtooper armor is pretty weak...obviously (their helmets don't protect them from being knocked out by a staff).  So, by all means, rob them :)