10-19-2011 03:53 PM
As a company that provides test instruments, we get pulled into providing proof of concepts and side by side comparisons all the time. In February 2011, I decided to make a demo that compared the speed of one of our latest instruments, the NI PXIe-5665 vs. the Agilent PXA vector signal analyzer. We recognized that the Agilent PXA is a world class instrument and wanted to show our customers that we have similar performance, but with the obvious advantages of the PXIe bus and faster processing (amongst other benefits) the NI PXIe-5665 is about 10-15 times faster for measurements such as ACLR.
I recorded a video showing the differences in the two instruments. I picked different benchmarks such as list mode, WCDMA ACP and LTE EVM for the comparison. The link to the video can be found here.
http://www.youtube.com/watch?v=9kBd2JMdFgE
(The handsome guy with the great hair in the video is me 😉 )
I also put in months of effort to make sure that the demo showed an equal and fair comparison, such as performing reads on both instruments, using the same number of averages and using the same attenuation / RBW settings in all cases. I made it a point to mention that we are not doing a spec sheet comparison of the instruments and that both instruments have better measurement capabilities. The demo that I was showing for was for one particular test setup. The numbers shown in the demo pertain to that setup only and better results might be achieved by both instruments for a different setup (using a different filter for example). The speed results are similar irrespective of the setup.
I also had many RF engineers at NI look over the comparison before going public. There are a few instances where both instruments implement things differently, so we picked the typical setup that a test engineer would use. To read more about the setup of the demo, see here.
http://zone.ni.com/devzone/cda/tut/p/id/12651
I expected some rebuttal from Agilent of course, and welcomed the discussion. Initially this rebuttal was professional and courteous, where they made some interesting points about the demo. You can read about it here.
My team and I tried to address all of their questions and posted responses in an act of healthy competition.
However, this all seemed to change at European Microwave Week (EuMW) 2011 hosted in Manchester, UK. I showed the same comparison demo in our booth, while keeping Agilent’s response in mind (for example, I added an option of fast mode instead of IBW mode on the PXA, which loses dynamic range but improves speed on the Agilent PXA). The consequences of showing this demo at EuMW surprised us.
At the show, Agilent told us that we cannot show the demo in Europe (even though it is perfectly legal to do a side by side comparison in the UK). I spent over an hour working with Agilent’s engineers showing them that the demo shows an equal and fair comparison. We reviewed the code and even asked Agilent for suggestions to improve the demo. In spite of this, Agilent asked show organizers to force us to remove the demo whether it was technically accurate or not. We were threatened to be kicked out of the show and to have show technicians take down the demo in our booth. The show organizers would not allow us to keep the Agilent box in our booth, even when it was turned off.
We ended up taking down the demo before the last day of the show. We eventually ended on good terms with the show organizers, but it was still surprising that a large company would behave so aggressively and unprofessionally by trying to take down the demo with such tactics instead of trying to technically prove the demo wrong.
I created this demo thinking that it would help customers make informed decisions, and to provide additional options when selecting test instrumentation.
I want to ask you, our customers, whether side by side comparison demos such as the one talked about above are useful to you or not?
10-19-2011 05:04 PM
Rajiit,
they are very useful indeed.
One thing that I, as a user, would always like to see is a comprehensive list of
This not only contributes to the comparison being fair, but it also helps us, the users, in making the right decisions based on our applications.
10-19-2011 07:55 PM
Interestingly when I clicked on the link to the eebeat site http://www.eebeat.com/?p=3928 I got blocked by my "Norton's safe surf" software, declaring that it had been ID'd as a malicious site! Boy, your interaction with Agilent must have gotten really hot! The Norton says that it has "drive by downloads" of anti-virus redirects"
10-20-2011 10:04 AM - edited 10-20-2011 10:06 AM
Raajit,
As a test engineer I have typically avoided presentations of side by side comparisons. The funny thing is- the presenting companies product ALWAYS seems to be the no-brainier better solution and I (the audience) should have my head examined for even considering the other instrument. That being said, I have listened to your presentations before and found them remarkably open and honest so, I dug into the video and discussion. Here are my key takeaways:
By the way. Nice piece of gear. Congrats to the team.
10-20-2011 10:39 AM
I enjoyed the presentation, even thought I don't know anything about these kind of measurements.
I agree with Jeff that I am usually very suspicious with such comparisons, because the side sponsoring the tests always wins hands down, and often further examination shows great bias. Things are sometimes more believable if done by an independent outside party, but even that can be tricky because you never really know who's behind it (see e.g. astroturfing).
Overall, I truly believe that the time of monolithic instruments are over and NI is on the right track. Congratulations to such a great product. The presentation was cool!
(Maybe it should all have been programmed in VEE, just kidding! ;))
I am curious about the debate at the show. Did the legal teams get involved or was it just a local fistfight?
10-20-2011 11:42 AM
Thank you for your response. My thoughts are below.
· what are the configuration settings for the equipment in the comparison
· what specific parameters/specs are being compared
I tried to include all the high level configuration settings included in this white paper
http://zone.ni.com/devzone/cda/tut/p/id/12651
We tried to reference the white paper everywhere the video was shown. Unfortunately I could not mention all the setup details in the video in the interest of keeping the video short.
As you probably noticed, the two instruments are different architectures and we have had customers ask for even details on the exact configurations. For example we had a customer who asked how to get repeatability to .1 std deviation. This was achieved by both instruments at about 25 averages, and we are more than open to give out more details or tweak the demo if a customer asks for it.
See the comments section on this video for more details. http://www.youtube.com/all_comments?v=AkJ12ny6A2w
The parameters and specs are also mentioned in the white paper.
10-20-2011 11:45 AM
- what workarounds (if any) would be available in the competitor product to bring it closer (if not overcome) the performances of the NI piece being demoed.
We tried to incorporate all the “workarounds” suggested by Agilent to make their instrument faster, and some of these methods did make the instrument faster but at the cost of something else.
This is a great point you bring up. We were aware of most of the workarounds on the PXA to make it faster, unfortunately implementing some of these work around (such as the FAST ACP mode) on the PXA meant losing out on dynamic range. While developing the demo, I wanted to show the best dynamic range possible out of both instruments for all scenarios. We did document the workarounds in the white paper as well. This table in the white paper showing the FAST ACP mode is an example.
10-20-2011 11:45 AM
@LV_Pro wrote:
Interestingly when I clicked on the link to the eebeat site http://www.eebeat.com/?p=3928 I got blocked by my "Norton's safe surf" software, declaring that it had been ID'd as a malicious site! Boy, your interaction with Agilent must have gotten really hot! The Norton says that it has "drive by downloads" of anti-virus redirects"
Yikes! Sorry about the malicious site, not sure why that happened to you. Here is another link on evaluation engineering that mentions the same rebuttal from Agilent. Hope this works
10-20-2011 11:46 AM
@Jeff Bohrer wrote:
Nice to hear from you again, it’s been some time. You bring up some good points for sure.
I agree that I could have done a better job earlier of making it clear that the demo applies more to automated test and not as much to debugging or benchtop testing. The assumptions that I made for the video were
· - The test time is the time taken to make a single measurement on a DUT (moving averaging does not count as a single measurement)
· - The test time includes initiate + acquire + process + return measurement
· To be clear, I am not transferring the entire trace from the PXA, only the ACP measurements, whereas on the PXI module, I have to transfer the entire waveform (just a difference in architectures)
· - Assumed 10 averages for some kind of repeatability
I am glad that you are happy with our support Jeff J. Having been part of our Applications Engineering team myself a few years ago, I can tell you that NI AEs will do our best to help you.
10-20-2011 11:47 AM
altenbach,
I am glad you enjoyed the presentation. Below are my thoughts on your concerns.
· Do the speed differences really matter in practical applications?
For an automated test environment, I believe that they do. We have seen customers such as Triquint and STEricsson who have reduced their characterization times from a couple of weeks to a day by moving from a boxed instrument to PXI. The video above shows just one measurement but a company such as STEricsson repeats the above mentioned ACP measurement 800,000 times and that is where a difference of days comes in to play.
Alternatively, if you are R&D and debugging your DUT for the first time, especially if you are doing it manually, the small difference in a few 100s of ms might not matter.
· Which instrument is easier to setup and configure to somebody who is not familiar with either product?
Honestly, If you are doing a simple measurement, just one time, not automating it, and not transferring any data back to a PC, a boxed instrument might be easier for display purposes. However, if you are
· automating in any way
· synchronizing,
· Transferring data back to a PC
· Performing real time processing
· Doing MIMO applications
The PXI platform provides plenty of benefits over a traditional boxed instrument.
·