Albert Einstein said, "Everything should be made as simple as possible, but not simpler."

The Interaction Designer's Coffee Break discusses this in terms of website design:

Usability is based on principles such as "Less is more" and "Keep it simple, stupid". But there is more to simplicity than meets the eye. By reducing visual complexity at the cost of structural simplicity, you will give your users a hard time understanding and navigating the content of a web site

For example:

One of the most misleading arguments used in favour of reducing visual complexity is the rule of 7 +/- 2. The rule states that the human brain can't handle more than 7 +/- 2 items at a time. If you apply the rule to visual design, it would mean that things such as lists of menu items or items in a bulleted list can be no more than nine.

The trouble with this rule is that the psychologist George Miller who formulated it was studying the limitations of short-term memory – not limitations of what people can perceive visually at a time. Humans can only retain 7 +/- 2 items in the immediate memory, but have no problem in dealing with great amounts of information in the field of vision. As long as you have information present for continuous reference, immediate memory plays no significant role in your perception.

...via elearningpost