If you are an American, you know that if you take a taxi, it is expected for the person to sit in the back seat of the taxi, especially if you are alone. The only time that you venture to the front seat is if you are a large group and all three back seats are already taken. However, in other countries avoiding sitting in the front seat can actually come across as quite rude.
Taxi drivers will think that you're a bit snobbish if you don't sit in the front seat and opt for the back. Maybe they are more understanding once they realize that you're American.