Native speakers: 128 million
Where it's spoken: Japan
Why it's hard: Japanese is difficult for English speakers because, among other reasons, it has more than one set of written characters, and readers need to memorize thousands of characters to achieve fluency.
Japanese is also a highly contextual language that requires different forms depending on the formality of the social situation. Even a pronoun like "you" could have more than 10 different translations depending on the person you're talking to.
お前と言ってるモルモン教の宣教師がいた。
張本いいこと言ってた。