It's an unfortunate fact of traveling, but a man who travels is a man bound to his suitcase. Your suitcase is the thing calling the shots as to what you will and won't wear on your vacation, and let's just say that your suitcase isn't going to appreciate that bulky sweater taking up half of the space. Sadly, some of our favorite items of clothing are unpackable, but that's just life.
On the other hand, think of all that extra room you're saving by not packing your favorite bulky sweater. That's just the law of equivalent exchange for you: you win some, you lose some.