People are annoyed over the loss or theft of their personal information, a new survey from the Ponemon Institute reveals.