Overselling is when a host sells more resources than a host server has, physically. Generally it refers to Ram, but can also be done with diskspace and CPU. It does not automatically imply 'poor management.' It is simply a tool that, much like everything else, needs to be used with responsibility.

How does it work?
It works like this; You have a host server with 8GB (8192MB) ram. and you sell 256MB Guaranteed ram to 32 clients in a 1:1 ratio. Each client on that host can use that 256MB ram whenever they want, with absolutely no swapping. However, 98% of them won't be using all of their Ram at any given time. So you end up with a lot of idle resources.
Now, those 32 customers have to pay a premium price for their resources. The host still has to make money off of them, otherwise there is no reason to continue providing the service. Now, the host says... heck, we've got 8GB Ram sold to these 32 people and only 2GB is being used on average... let's sell 256MB guaranteed Ram to 8 more people on that host server (who, with the above numbers probably won't even notice). So we have 40 customers and 10GB of guaranteed Ram SOLD to 8GB of physical Ram. Each of those 40 customers have to pay a bit less for their resources.

It can be abused, and when it is, you will notice it.
Overselling is playing the Law of Averages. Sometimes you have outliers, these are the people who use what they are given. When this happens the disks will start swapping, IO waits go up, etc, because there is no longer a 1:1 ratio of Guaranteed resources to Physical resources and when too many people try to use what they're given, well.. there's a finite amount for them to access. The more you oversell, the greater the amount of customers that fall into the 2% of those who use what they pay for. Host servers become sluggish and non-responsive, everyone complains, nobody is happy.

The key here is; it is the hosts responsibility to use this tool in a very controlled and monitored fashion. Treat the host node like a living being that needs constant supervision to ensure a certain level of service quality.

Overselling is NOT bad. Hosts who oversell irresponsibly are the culprit. But I guarantee you that without it, you wouldn't be paying what you do now for your shared/vps hosting. VPS wouldn't be able to price-cut dedicateds and come damn near in price to Shared. Shared wouldn't be able to offer insane packages for $5/month. The hosting world, in short, would be completely different.