The correct answer is 'fifty-man'. In English, when describing a group of people doing something collectively, we use the singular form of the noun with a numeral. Therefore, 'fifty-man' is the correct phrase to use.