No, you will not lose the stone if unequipped but Unequipping Stone does cost Gold Coins.

 

The stone will remain in your inventory unless consumed when crafting a higher stone.