I'm not familiar with Virtual Box, but the solution with VMWare would be to recreate the VM -- that's not as bad as it sounds, just start up the VM creation process, and when you get to the part about a boot disk, instead of saying "create a new one", say "use an existing one", then select the one you have. VMWare is smart enough to just use it after that. The drivers are all virtual on the hosted OS, so it's only VMWare that needs access to the "new" (different) drivers, then the software does the translation.
You may want to look and see if VB has the same sort of thing. Of course, I'd use a copy rather than the actual .VMDK file (it'll be the biggest one), just in case it erases it, thinking that's what you meant.