Games : Is it normal that Diablo 3 is very easy

I started playing with a friend of mine and we got to somewhat level 20 and we started to up our difficulty from normal on up. (no idea what it's called exactly) After that we still noticed it was very easy and we went to the highest diffculty possible(Not the one you unlock at level 60) and it was still way too easy. We never die or get in danger. Is this normal in Diablo 3 or are we missing something? We're level 40 now. We played from level 30 - level 40 in the highest possible difficulty