Cake or death
Calculations related to the "Cake or death?" problem: how to make an artificial intelligence that can learn that cakes (which are harder to make) are morally better than producing death (easy to produce). With the wrong representations it turns out to be better (higher expected utility) to do a lot of killing and then ask the programmer if it was right (and if it wasn't, make one cake). Getting the motivation system right is surprisingly tricky.