In board-am335xevm.c, look for the rtc initialization code, try changing am335x_rtc_init. Look for
2050 /*
2051 * Enable the 32K OSc
2052 * TODO: Need a better way to handle this
2053 * Since we want the clock to be running before mmc init
2054 * we need to do it before the rtc probe happens
2055 */
2056 writel(0x48, base + 0x54);
And change the 0x48 to 0x40. If you look in the TRM for the RTC registers, bit 3 of register RTC_OSC_REG selects the clock source for the RTC. 0=internal, 1=external.
Steve K.