According to Dictionary.com, Speculate means "to engage in any business transaction involving considerable risk or the chance of large gains." So typically, speculation has been associated with Wall Street traders and other evil doers. However, most people are engaged in speculation in one way or another.
Anyone who has bought stock or any commodity expecting the item to increase in price has engaged in speculation. I have a couple thousand baseball, football, and hockey trading cards that I hope one day will be worth more than I paid for them. That is a business transaction with hopefully the chance of a large gain. But I don't hear anyone complaining about all those kids sitting on their trading cards waiting for the price to go up.
Finally, as we have seen by the "collapse" of the housing market, people buying a house are engaging in a transaction involving considerable risk. I'd say that losing your house and damaging your credit is a considerable risk. Again, most people don't think of a house that way (for various reasons) nor do you see anyone railing against people for taking such a risk with their finances and ultimately our economy.
Instead, people are railing against "Wall Street" for taking these risks and trying to make "speculation" into a four-letter word. When in fact, many more people speculate much more often than they think and it isn't always a bad thing. At least that's what I tell my better half now that I've got an out of the way place to keep my baseball cards.