I often find that on the weekends/when traveling I prefer to ditch the dress shirt when not called for, but love to wear my blazers. During the colder months, I love to wear a long sleeve polo under my blazers. The collar, while not usually a stiff as button downs gives some visual structure to the look as opposed to a crew-neck sweater.
Polo/Belt: Ralph Lauren, Blazer/Socks: J. Crew, Pants: Brooks Brothers, Shoes: Johnston & Murphy, Glasses: Tom Ford, Watch: Maurice Lacroix.