Games : Pillars of eternity monk armor

I chose a monk and am struggling to make him viable early game.

With his limited array of skills and the melee requirement he seems uniquely fragile.

I also cant find much of a downside to running him in medium/heavy armor save for a small damage nerf.

Is there any reason not to armor up an early game monk? And is the lack of defense otherwise overcome differently?