If you're not seeing it, then there may well be something else wrong. I've blown a driver (in my case, a woofer in an ancient speaker) by playing too much bass through them. What happens is the coil warps. Not really anything can be done about it, other than replace the driver. I don't know how to verify this to be the case, though. The coils remain housed down to the degree of discombobulation described by Ari.
The only other choice is that you accidentally pulled a Kinks on when removing the pads. Doubtful, though, you'd know. (I'm thinking of the images of The Kinks creating distortion by actually plunging knives into the surrounds of the drivers in their guitar cabinets.)