Willingness to do something is not the same as ability to do something. In theory, there exists a break-even price for anything one values. It may fail in practice for any number of reasons (e.g. transaction costs are too high; the good is not excludable; that price is more than the total value produced by all humans ever; people find this kind of thinking icky; etc.).
But is that general aversion consistent? We know that people’s beliefs are often inconsistent depending on how a choice is framed, so it is not a given that people have a general aversion that turns out to be irrational upon examination. Particularly where the general aversion is among non-philosophers in a non-reflective mode, it isn’t clear that we should put much weight in the moral consensus.
And one may simply reject consequentialist morality, or prefers not to swallow its hard pills. If the outcome isn’t the basis of whether or not an action is moral, then whether or not World A is better than World B is irrelevant to the question of which action is required or permissible. So, maybe put this a different way: if we take as a given that the correct morality is consequentialist and that that entails that we should push the fat man, in that case do you agree that X must exist?
One response to this line is to point out that, whatever other consequences you want to load into World A, there should still be an X that outweighs those consequences. Count up all the children in the world who will die of malnutrition in the next ten years, figure out how much they need to not die of malnutrition, plus the cost of distributing that much to each one. Is the life of every child who would die of malnutrition in the next ten years really not worth another source of existential dread and a little blood on your cuffs?
But I prefer another approach: to quote my torts professor, “don’t buck the hypo” (not sure if that’s original to her). You can make the math not work by adding additional terms, but those aren’t the hypothetical being considered. If what you’re saying is, “yes, in the case you presented, it’s morally permissible to kill for money, but that case could never happen in the real world for [reasons]”, then fine, say that. But if you don’t agree that the hypo as presented justifies killing for money, then lets keep discussing the hypo as presented before we embellish it.
I think the fat man problem suffers from a similar problem: people implicitly read reality into a fanciful intuition pump. It’s actually difficult to conceive of a situation where you know 100% that pushing a fat person in front of a trolley will stop the trolley and save lives, and so even when people’s conscious minds acknowledge that’s a given here, their moral intuitions can’t be readily separated from the real-world in which they were honed, in which the fat man problem is outlandish because there’s no one in the world fat enough to stop a speeding trolley. (Maybe a way to test the general aversion under this hypothesis: do a survey where you ask half the people the traditional fat man problem, and ask the other half a variation in which the fat man is sitting above the switch and pushing him off will kill him and hit the switch. If I’m right, there should be less aversion to the latter.)