Those that have low self-esteem typically don't do well in life, or aren't doing well in life, and hence have low self-esteem. However, not everyone that has low self-esteem got there by themselves, and for some people the path that got them to that low self-esteem, came primarily from somebody close to them that basically destroyed their self-esteem to the extent that the one suffering low self-esteem has still not been able to successfully recover from.
While it might seem a stretch to believe that mere words, sarcasm, and cutting remarks can break anyone's self-esteem, the truth of the matter is, that the more respect and the more direct knowledge that a given person has of someone, basically means the more credibility and legitimacy that they have, in which, no stranger, or passerby could ever have that equivalency. What this indicates is that while cutting remarks from strangers or acquaintances may indeed sting, and may indeed shake a given person up, for the most part, people consider the source, and are usually able to successfully brush such off. On the other hand, if your spouse, or if your parents, or if your best friend, basically severely criticizes and harangues you about the most important things in your life and of your being, all of which pretty much define you as a person, in a manner, in which there is no equivocation in what they say, those words, spoken in anger of not, can annihilate a given person's self-esteem.
Everyone wants to believe that they have value, and they especially want to believe that the things that they are most noted for, are of value, to other people, and especially are of value to those that they are closest to; so then, when somebody that is very close to a person, states in a very specific way, why that given person is so useless or in error or wrong about something that is quite important to that other person, because of the respect that the other person has for the one so talking, it therefore makes it very hard to walk away from such an encounter, unaffected, though some do.
There is all sorts of abuse that one person can do to another, of which physical abuse is especially noxious; but those that make it a point to destroy another person's self-esteem because they have been able to ingratiate themselves to another, learning all that they can about another, and what is of most importance and what defines another, is, in its own way, just as noxious, for when one person takes someone of good character, and fair self-esteem, and belittles them, and thereby breaks their confidence, then that person has, to a large extent, changed the very being of another.
While criticism, and constructive criticism, has its place, the true point of any criticism should always be to be of actual benefit to the person so being criticized; and those that hide behind false fronts, as in "always being honest and upfront", are actually masking their true intentions, which is often to breakdown another person's self-esteem. The intent behind certain words that are being spoken, aren't always owned up to, but when that intent is fundamentally about breaking down another, what seems to go unrecognized by the offender, is those words, perhaps spoken in anger, haste or without discernment, can and often do have long-term negative consequences.