Take into consideration what software you want to use for adding music, and whether that software will allow you to rip CDs from your collection. I've heard that many, if not all, Mp3 / Mp4 players are software-specific. Check for compatibility.
For the player itself, how much storage do you think you need? IPod Shuffles are down to $79, and I'm sure equivalent models in other brands are even cheaper. Shuffles (and some other brands' models) use flash memory instead of hard drives, though. Most Mp3-storing cell phones use flash memory. A friend of mine, who has both a Shuffle and a regular iPod (received as a gift) said the sound quality on the regular iPod (with a hard drive) was superior. Your mileage may vary.
If at all possible, listen to a demo of whatever model you're considering. Are the controls easy to figure out and use? How bulky is it or how sturdy does it feel? Also look up reviews. I used to use epinions.com; don't know if they're still around. CNet, enGadget, even some merchants like Amazon.com have reviews.
But most importantly, when you finally choose your player and start using it, PLEASE DON'T SING ALONG--at least when you're out in public!