Hi Marlo,
In LabVIEW, to design a FIR filter by windowing, you start with an ideal frequency response, calculate its impulse response, and then truncate the impulse response to produce a finite number of coefficients. To meet the linear-phase constraint, maintain symmetry about the center point of the coefficients. The truncation of the ideal impulse response results in the effect known as the Gibbs phenomenon � oscillatory behavior near abrupt transitions (cutoff frequencies) in the FIR filter frequency response. This might be the reason you are seeing slight distortions outside the passband.
You can reduce the effects of the Gibbs phenomenon by smoothing the truncation of the ideal impulse response using a smoothing window function. By tapering the FIR coefficients at each end, you can diminish the height of the side lobes in the frequency response. The disadvantage to this method, however, is that the main lobe widens, resulting in a wider transition region at the cutoff frequencies. The selection of a window function, then, is similar to the choice between Chebyshev and Butterworth IIR filters in that it is a trade-off between side lobe levels near the cutoff frequencies and width of the transition region.
I guess LabVIEW uses the Parks-McClellan method to compute the coefficients. This method finds the coefficients using iterative techniques based upon an error criterion
As for the IIR filters in LV 6.1, I think I was able to cascade 2 stages with 2nd order. The Advanced Analysis toolset also install IIR Cascade filters which consist of a cascade of 2nd or 4th order filter stages. Each individual stage is implemented using the direct form II structure because it requires the minimum number of arithmetic operations and the minimum number of delay elements (internal filter states).
I think there have been a lot of improvements on the Analysis section in LV 6.1 over the LV 5.0 version, so it might be worth for you to try LV 6.1 instead as it seems to fit your application.
Hope this helps,
Pravin