It sounds like the radiated emissions from the ESD test is coupling into the person touching the screen and this energy is coupled into the touch screen which injects unexpected current pulses into the AM335x device. The current pulses are apparently strong enough to disrupt normal operation of TSC in such a way that it locks-up the internal logic circuits. Unfortunately, the TSC does not have a soft reset that allows this module to be reset by software. The only way to reset this module is by applying a system reset.
I'm not aware of any other reports of AM335x TSC locking-up as you described so there may be some system level dependency that needs to be understood and resolved. For example, I saw something similar on another TI device once and we found a metal bracket on the board floating that was building up charge and coupling into the device inputs. We were able to solve the issue by grounding the metal bracket.
It sounds like you have explored several filter options without any success. I think this is the correct approach. Does this happen only when touching the screen near the corners where the resistance to the AM335x AIN terminals is lower than it would be in the center of the screen.
Is this ESD test a industry standard test or a custom test to validate your system? If possible, can you describe the ESD test or point me to an industry standard.
The BBB and capes were designed to be low cost evaluation products with minimum noise immunity precautions. So it is not be a big surprise to hear these boards are having issues with this type of test.
Are you having the same problem with the smaller display?
Regards,
Paul