The maxTime input does have the unit of mS and is used to act as a timeout feature. The ReadWaveform function places the scope into a state where it is waiting for the trigger you have configured in your code, waits for all the data to be acquired, and then reads the data from the hardware into RAM. If the trigger is never received, theoretically, your code could hang at the ReadWaveform function indefinitely, creating the need for a trigger timeout feature.
The waveformSize input specifies how many points you wish to pull from the hardware into computer RAM. The number of samples the hardware will actually acquire is configured with the NI-SCOPE configure horizontal function, but you can either read all of these points, or a subset of these points. The units of
waveform size is number of samples, or data points. You do not need to factor in the size of each sample in memory and convert it kB or MB. Just tell the function how many samples you want to read.