Whenever I go to fight the big guy in armour (The Taros), he just tells me to go away. How do I fight him?

You need the Achilles Sword. Look at the list of important items for how to get this. Visit Rohde when you’ve got it, and he’ll go to the Taros (the guy in Armour) with you. You can fight him now, but only the Achilles Sword can damage him.

Meeting Rohde in Shining Force II