Setup of NT 3.1 and 3.5 doesn't detect processors newer than the original Pentium, as well as some models of 386, 486, and Pentium processors. To bypass this, initial.inf and setup.inf needs to be edited: in both files, change the line STF_PROCESSOR = "" ? $(!LIBHANDLE) GetProcessor to STF_PROCESSOR = $(ProcessorID_I586). Use 386 or 486 instead of 586 if the OS is being installed on a 386 or 486. (source: BetaWiki)
While BetaWiki says you should edit the files "after the first stage (in text mode) of setup is done", in my case neither WINNT nor SYSTEM32 existed in that stage, probably because i initiated setup via i386\winnt.exe in DOS. Instead, I copied installation media to the hard disk, and edited the files included in the copy, so that the setup would read the modified files from the very beginning.
To do that, use the expand command:
expand initial.in_ initial.inf expand setup.in_ setup.infDelete the in_ files (otherwise the setup would still read the compressed, unmodified copy), and edit the expanded inf files.
On a VMWare VM with two virtual CPUs, NT 3.5 and 3.51 might say HAL is incorrect (something like "this HAL is only for MPS version 1.1") when booting. Expand hal.dl_ from the i386 folder of the installation media to hal.dll and overwrite %windir%\system32\hal.dll. The system would boot, but the downside is it would only recognize one CPU. Maybe VMWare VMs don't support earlier versions of the MPS?
NT 3.1, even in setup, doesn't recognize partitions that start after the first 4GB of a hard drive.
NT 3.5 ~ 4.0 setups recognize such partitions, but the installed OSes still cannot boot from them. To fix, copy ntldr from an installation of NT 4.0 SP5 or above. (NTLDR is backwards-compatible even with Windows 2000)
NT 4.0 setup cannot recognize IDE hard disks that are 8 GB or larger. To fix, get atapi.sys from KB197667 or SP4 or later, and replace the copy in the i386 folder of the installation media. (see also: https://nt4ref.zcm.com.au/bigdisk.htms)
Choosing to format the partition to NTFS, or convert from FAT to NTFS, might cause setup to say "setup was unable to open or access the setup CD-ROM or floppy disk". If this happens, you have to keep the partition as-is, and convert later within Windows NT if desired.