Strategically (Im)moral: Stated Preferences in Personalized AI Advice

Abstract

I study whether personalized AI recommendations based on stated moral preferences affect moral behavior. In a preregistered experiment, 1,250 subjects choose between a profit-maximizing option and a costlier moral option that generates a UNICEF donation, deciding themselves or delegating to a profit-maximizing or moral algorithm. Delegation reduces moral choices by about 13 percentage points, and personalized recommendations based on structured or free-text preference elicitation do not offset this decline. Identical questions elicit weaker moral preferences when subjects know that their answers shape recommendations, generating more profit-maximizing advice. When subjects do not anticipate the link between elicited preferences and recommendations, moral choices return to the no-delegation benchmark. Personalized advice treats stated preferences as stable inputs; in moral domains, subjects strategically shade those inputs to obtain self-serving advice. The results reveal a tradeoff for AI regulation: transparency about the preference-recommendation link can distort the inputs on which personalized advice relies.