It's also on a cooldown, you'd have to do something else while waiting for it anyway, even if it's just spamming spacebar. This may be a shared "safety" feature between item-affecting skills, as I think "it belongs in a museum" does the same; difference being you can't (normally) do anything with a gem like you could right-clicking a wearable, museumable item.
