Games : Do you earn less XP per character with a larger party?

I noticed that the game got significantly easier after I found a few more people to join my party. I'm at five party members now, and just using as many as possible seems like the best option right now.

I'm wondering though if there is a drawback to using the maximum number of characters in your party? Does the party size affect XP gain in any way? Is there any reason to use less than the maximum number of characters in your party?