The problem with an insidious philosophy that believes that human society reflects in its natural order the survival of the fittest, is that such a philosophy leads to a rather convenient conclusion -- though flawed and quite fallacious -- that those at the pinnacle of power within society are therefore the fittest. Additionally, it then so follows that such provides the basis for circular reasoning and becomes the raison d'être for all sorts of laws, injustices, unfairness, and inequality created between peoples, races, creeds, and sexes; all under the guise that the fittest have the exclusive right to do whatever they so will, because they are the fittest.
That is to say, within any society, in which the reality is that society is both unfair and non-egalitarian; there will not be any design for real improvement of the general conditions for the population at large, because the prevailing sentiment of those that are in power, is that such a country, is built fundamentally upon a belief that those that are at the top, are quite obviously the most fit of the peoples in that society, and because they are the most fit, it is therefore their right to do what they so will, for the betterment of those so fit, foremost, and only do what can be done for the unfit, as an afterthought or from charity, if that.
Lost somehow to the closed minds of those that so often have been in positions of power and authority, is the real salient fact that the reason that so many believed for so long, and still believe even today, that white people, are of superior intellect in comparison to black people or brown people or red people or yellow people, fundamentally comes down to those oppressed peoples' historic lack of opportunity, lack of education, lack of healthcare, lack of family structure, lack of safety, lack of justice, lack of capital, lack of land ownership, lack of entrepreneurship, and lack of just about anything of value, in comparison to their overseers.
Not too surprisingly, when the results are stacked to favor one race to the exclusion of all others, than those that are judged to be unfit, have an arduous and daunting task to get their selves to even the starting line of opportunity, and are handicapped in just about every conceivable way, thereon out. Further, the supposed unbiased results of the testing of the intelligence of different races are going to quite obviously favor those that have been prepared for such, as compared to all those that have not been.
All of this combined leads intelligent people, even well-meaning intelligent people, to conclude all sorts of nonsense, about other races, and other peoples, without seeming to recognize the most fundamental fact that the color of a person's skin has absolutely nothing to do with their intelligence; and further that the survival of the fittest, is applicable, at best, to the animal kingdom, and has nary to do with the human race, of which the mind and spirit properly developed is the builder of great societies and great nations.
It is social Darwinism, especially as propagated in the late 1800s that lead to the unnecessary delay and late integration of societies, a legacy that is still being dealt with, today, for far too many that are in positions of authority are truly unfit to be there, though they act as if they are fit to be.