There are a ton of Japanese traditions that are centered on being polite and courteous to others, especially when you're in a public setting. So, it shouldn't come as too much of a surprise that being especially loud while you're on your phone in public is frowned upon. If someone does have to answer a phone, then they usually tend to talk in a low or quiet voice while out in public.
If the conversation is especially personal, then a person may even wait until they're in a place that's private or at home to answer the phone and have their conversation.