if you hit 165Mhz and unstable, I'm afraid you can't push your RAM that high since the CPU will conk out even before you hit a pretty good high frequency for the RAM.
at 54-56c, you're pretty hot already. i suggest you review your case ventilation, and of course, your room ventilation. no need for costly aircon, and don't place a fan to blow inside the case. A nice circulating air within the room, with a good path/exhaust for the escaping hot air will be sufficient.
as for the vtt, this is the main reason why i skipped Abit. vtt is buggy with Abit, I'm not sure with the later releases, but early ones report a wrong value. guides on vtt mod abound so it's up to you to try and mod your board.