Some months ago I encountered this problem:
during installation of a test Oracle Database 126.96.36.199.0 Net Configuration Assistant failed.
This appened on a 32bit Oracle Linux 5 U4 test VM running on my 64bit AMD desktop PC's VMWare Player.
As stated in Oracle Support it's a known issue on 64bit hardware. (X86 DBCA, NETCA GIVE JAVA HOTSPOT ERROR IF ON X86_64 HARDWARE [ID 942076.1])
The following procedure is the one I followed to solve this problem:
Download and extract patch 8670579 from Oracle Support
So I got p8670579_112010_LINUX.zip then
[oracle@orcl ]$ unzip /home/oracle/p8670579_112010_LINUX.zip
To apply this path we need to use OPatch that is installed as additional software during most Oracle products installation...in this case Database installation. This implied that I couldn't patch the system before installing database so the trick is to install Database until NETCA fails then apply patch.
When Net Configuration Assistant fails run OPatch:
[oracle@orcl ]$ /u01/app/oracle/product/11.2.0/db_1/bin/OPatch/opatch apply /home/oracle/8670579
where as usual /home/oracle/8670579 is the path where I unzipped patch.
When patch is applied I clicked RETRY button and this time Net Configuration Assistant completed successfully!!