If you've got the Linux kernel booting off NAND then that's a good start.
I found the easiest way to get the UBI rootfs going was to do the next easy step, which was to recompile the Linux kernel with an embedded initramfs containing the busybox shell. Then boot this kernel off your NAND, and at least you now have a running Linux kernel on the target.
To format the NAND UBIFS rootfs, I used the tools in arago-base-tisdk-image-am335x-evm.tgz (i.e. from the ti-sdk-06.00), which is a minimal rootfs (built using arago/bitbake). I loaded this tar file off a USB stick attached to the target and extracted to the initramfs.
To format the NAND (i.e. on the target):
# flash_erase /dev/mtd7 0 0
# ubiformat /dev/mtd7 -O 2048 -s 2048 -y
# ubiattach /dev/ubi_ctrl -m 7 -O 2048
# ubimkvol /dev/ubi0 -N rootfs -m
# mount -t ubifs ubi0:rootfs /mnt/nand
NB: Make sure you use the ubi* tools from the arago tar file, not the 'cut-down' ones in busybox.
