Elegance is a form of balance. No single detail should draw excessive attention to itself. Consider the most elegant of outfits -- the dinner jacket. It has just two colors, black and white. Everything is simple and classical. The viewer sees the entire outfit as a balanced and unified whole.
Imagine if the wearer replaced his black bow tie with a red one, and wore pants with a gold stripe down the side. Suddenly the balance is thrown off. The viewer immediately focuses on the red bow tie and the gold stripe. The impression is not of a whole outfit but of separate parts, with two of those pieces -- the tie and stripe -- overwhelming the rest. The effect is jarring.
It is important to avoid visual noise not only in color, but in cut. Pagoda shoulders and their frequent partner, roped shoulders, shout for attention. Elegance is much more subdued. Each detail is quiet but combines into a single piece of music.
Cary Grant is exemplary on this point. If you look at most of his suits, you won't find pagoda or roped shoulders, ticket or slanted pockets, extremely wide or narrow lapels, velvet collars, contrast colored stitching, or any number of other loud features. What you will find are simple details, perfect fit, and unsurpassed elegance.
Some examples from Grant:
http://www.doctormacro1.info/Images/Gra ... ef)_02.jpg
http://www.doctormacro1.info/Images/Gra ... lk)_01.jpg
http://www.doctormacro1.info/Images/Gra ... us)_09.jpg
http://www.doctormacro1.info/Images/Gra ... us)_11.jpg