Per a certain request, we'll expand on the above...

Raw crystals suitable for lightsabers need to be attuned by a Jedi/Sith before they are safe to put into a lightsaber. Its minute adjustments to the molecular structure, aligning it properly and attuning it to the Force. As this requires the Force, it can only be done by a force-user.

Most blaster crystals aren't even remotely close to the quality that can be attuned to use in a lightsaber. The easiest source for the prisoners for lightsaber crystals, other than getting one from the Sith, is to make them from scratch. Such creation requires that the Jedi guide the growth/forging/whatever with the Force for the crystal to be usable in a saber.