Hi Boris,
Since you're passing in a 1D array of Y values to the signal input of the Digital FIR Filter, dt is set to 1. This implies that your sample frequency is 1 Hz, violating the requirement of the cutoff frequency (Fc) being less than or equal to half the sample frequency (Fs). Thus, you should use the "Build Waveform" primitive to modify dt to equal 1/Fs. Feed this new waveform into the Digital FIR Filter.vi. Refer to my attached VI.
What are you trying to accomplish with acquiring the analog signal of the incremental encoder? Perhaps you can avoid the whole noise issue by connecting it to a counter?