And to go with what Nate said, actually have a lot of vinyl on hand. And of course make sure you enjoy listening to the format.
Someone on Vinyl Asylum recommended to start slow since you really might dislike something that doesn't have an auto-arm, having to stay there to flip sides, change records, go through the cleaning process, etc.