I've run into the same problem, but deleting the element set warns me:
Are you sure?
This will delete the element set and all elements assigned to the element set. Items will lose all metadata that is specific to this element set.
I really don't want to reenter all the metadata that we've already entered. Is there another solution or a work around?