I've finally been able to get my Mac Mini to display proper 1360x768 output, without horrible scaling. It turned out that the framebuffer wouldn't accept the new resolution X asked for, and stayed on 848x480, while X would think the new resolution had been set. This, combined with the fact that X.org doesn't seem to have a built-in modeline for 1360x768, caused the breakage.
Now my "Monitor" section looks like this:
Section "Monitor"
Identifier "Generic Monitor"
Option "DPMS"
HorizSync 30-61
VertRefresh 50-75
Modeline "1360x768" 84.50 1360 1392 1712 1744 768 783 791 807
EndSection
And my "Device" section like this:
Section "Device"
Identifier "ATI Technologies, Inc. Radeon 9200 (RV280)"
Driver "ati"
BusID "PCI:0:16:0"
Option "UseFBDev" "false"
Option "MonitorLayout" "TMDS,NONE"
EndSection
Op dit werk is een Creative Commons Licentie van toepassing.