While the origins of both phrases may be similar, their recent use is very different.

In recent times, the expression "to call him out" has means the escalation from an argument to a physical altercation and is the demand that a person goes outside premises for a fight. The current use, common in social media and spreading, is capable of misinterpretation and, even, as being an incitement to violence.

To "take him to task" may have similar origins as in to give someone a beating (as distinct from a fight) but it has long been used in a non-violent sense to mean verbally berate. In fact, it's gone beyond that to be the equivalent of a "dressing down" which is where a person in some form of superior rank or position delivers a good telling off.

So, unless you intend to invite someone to a fight, don't say "call him out." Use one of the equally powerful alternatives.