The VESA video driver has a maximum resolution of 1024x768 from what I understand. In order to utilize a higher resolution you will need to switch over your video driver to match your system's hardware. You can do this by starting up the display wizard on boot-up (I think it is option 6 on the boot menu).
With PC-BSD 9.0, the best supported graphics devices are from NVIDIA, but there has been a significant amount of progress on the Intel graphics drivers for PC-BSD 9.1. If your computer has Intel graphics, I would recommend that you upgrade to the PC-BSD 9.1-RC1 development image instead.
~ Ken Moore ~