I have an application where I need to fetch multiple records, I'm using a PXIe-5185, and basically the problem is that it takes too much time to fetch multiple records, I did a test to measure the time it takes to read the hole 32MB of data and this are the results.
Records to Fetch = 8192 (the maximum 1 Record = 4KB, 8192 Records = 32MB)
Time to Fetch = 1100 ms
Records to Fetch = 1
Samples to Fetch = (2^25 - 1500) = 32MB
Time to Fetch = 70 ms
So, I want to know why it takes more time to fetch the multiplerecords if both are reading the same amount of data? Is there something I can do to decrease the fetching time?
Solved! Go to Solution.
No matter the device (HSDIO, RF, or Digitizer), fetching multiple records will always take longer because there is more communication overhead. With more communication overhead, the longer the host takes with specifying records, and thus the longer it takes to fetch the multiple records. On the flip-side, fetching only one record only has to deal with the communication overhead once, hence the shorter fetch time.
Theres is no way to acces the memory directly, like raw data, and then I can separate it into different records?
I'm not entirely sure what you mean by that.
The only differentiation the NI-SCOPE Fetch function has is "unscaled data" and "scaled data". You may mean "unscaled data" when you say "raw data". To change this, you'll need to specify to fetch I8 data instead of Cluster or Double data. All scaling is done in software, so fetching unscaled data may improve your fetch time, but then you will need to apply scaling
The communication overhead can include processes necessary for providing timestamps with your data, and then there is inherent bus communication with the digitizer depending on how you are fetching (if you are fetching one record at a time, ten records at a time, etc. The more records you fetch at once, the less overhead you should observe, and the more quickly the fetch will return.)
What is your sampling rate in both of these cases (single record and multi-record)?
Above 2GS/s, there is currently a limitiation that does not allow you to fetch while acquiring. In other words, the acquisition will need to complete fully before the Fetch function returns with data.
When you set your 5185 digitizer in the multi-record mode, at what rate are the records acquired? In other words, at what rate do your triggers arrive, on average?
If your whole acquisiton is taking about 1 second, and the Fetch function is taking about 1 second to return, then that probably confirms that you are hitting the above limitation.
The biggest question is: what design constraint do you have? Why is the longer fetching time an issue?
P.S. If you have NI support, you may want to consider calling into 866-ASK-MYNI or e-mailing from ni.com/support in order to fully examine the options.
The idea is to continuosly sample 1000 samples at 6.25GS/s with a trigger frequency of 100Khz, when I did the tests I noticed what you said about the 2GS/s (where is that kind of info?), so know I'm sampling at 2GS/s with a trigger frequency of 4Khz to be able to continuosly Fetch.
To do the comparison, I waited until all the data was acquired so I only measured the time of the Fetch instruction.
What I mean about reading the memory, is to read the memory as if the data were only one record, so acquire the 8192 Records and then Fetch as if they were only one Record to be able to read faster and then I can manipulate the data and build the 8192 records.
Thanks again and I will try to contact our sale representative.
I would definitely recommend contacting your sales contact or setting up a serivce request to get a complete answer for your application. It sounds like you may be working with a particular functionality of the device that is unique to the 5185/5186.
...when I did the tests I noticed what you said about the 2GS/s (where is that kind of info?)
This is not documented as a specification or a characteristic of the device, because it may or may not be permanent behavior. What is currently available in the NI-SCOPE driver and what is possible in the NI-SCOPE driver differ. This is something that we provide pre-sales and post-sales support for to help on an individual application basis.
I waited until all the data was acquired so I only measured the time of the Fetch instruction.
When the "Actual sample rate" is 2GS/s and above, this Fetch instruction will wait on completion of the acquisition before returning. So the time you measure for this instruction will be some fraction of the time to complete the entire acquisition + the time to complete the data transfer from digitizer to host (i.e. the "Fetch" time). In other words, above 2GS/s with this device, if the acquisition has not completed, you will always be measuring additional time while you wait for the acquisition to complete.
What I mean about reading the memory, is to read the memory as if the data were only one record,
Without seeing your code, I'm not certain how you're setting this up--but you have the capability of fetching multiple records at one time (with one Fetch call). This is set by the "Fetch Number of Records" attribute/property which can be called prior to initiating your acquisition, or during the acquisition. See this example to help clarify:
niScope Multi Record Stream to Memory
(In this example, look for the "Number of Records Per Fetch" control in the lower-left part of the front panel.)
If you are only using one channel, increasing the number of records per Fetch will add a dimension to the data type you are currently working with. So if you were reading back a type of "Cluster" or "waveform" you will now be reading back a "1D Cluster" or "1D waveform". If you were reading back "1D I8" you will now be reading back "2D I8".
Hope this helps. Again I would encourage you to call in and create a service request for further support on this use case.
Here is my code, I think I am doing what you say and I think that for my test it doesn't matter the frecquencies, what I don't undestand is why if in both cases I am reading 32MB, I got two complete different reading times.
Maybe I am not explaining my self, I hope that my code help you understand me.
The two settings were:
Number of Records: 1
Number of Samples: 33,552,932
Time to Read: 74ms
Number of Records: 8192
Number of Samples: 1000
Time to Read: 1184ms
It looks like you are fetching multiple records at once--which is good.
I believe your trigger rate is throttling your fetch time. If you are triggering at 4kHz and you are acquiring 8192 records, that means it will take roughly 2 seconds for your acquisition alone to complete:
Acquisition Time [s] ~= # Records [Rec] / Trigger Rate [Rec/s]
= (8192 records) / (4,000 triggers/s) = 2.048 seconds
You won't see this entire 2.048 seconds reflected in the Fetch time, because the Acquisition will have already started by the time your program gets to the Fetch function. However, because these higher rates (>2.0GS/s) inhibt you from fetching while acquiring on this 5185/5186 device, you will need to wait until the acquisition ends before the data transfer from digitizer to host begins. In other words:
Time for Fetch Function to Complete (>2GS/s on 5185/5186) [s] = (some fraction of) Acquisition time + Fetch Time [s].
The Fetch Time is going to be a function of the data transfer rate of the PXIe bus + any additional software calculation time to apply scaling on the data, etc.
Now, compare all this info with your single-record acquisition, which does not rely on your (relatively slow) 4kHz trigger rate:
Acquisition Time [s] = Record Length [Sa] / Sampling Rate [Sa/s]
= 33,552,932 Samples / 12.5GS/s = 2.68 milliseconds
Since the Time for Fetch Function to Complete is still going to be the same, you can see how the Fetch Time term will dominate this use case (as opposed to the multi-record case, where the Acquisition Time term dominated).
So here's what you can try:
1. Speed up your trigger rate in the Multi-Record case (keep in mind that according to the rearm time in the 5185 specifications, there will be a practical limitation you hit at about 100kHz)
2. Put in a delay between when you Initiate your acquisition and when you call your Fetch function. If you do this in both the single record and multi-record cases, you should be essentially removing the "(some fraction of) Acquisition Time" factor from your Time for Fetch Function to Complete equation.
Again, I would recommend contacting NI Support for further options if you need them.
(P.S. - a slightly better approximation for the acquisition time (s) will be longer than what I gave above. This is because above I'm not including the time needed to acquire pre trigger samples on the first record and post trigger samples on the last record. So a slightly better equation would add in the total record length:
Acquisition Time (s) ~= #records [Rec] / trigger rate [Rec/s] + Record Length [s]
If you run these numbers, it doesn't change the total Acquisition Time by much. There are some other factors in this equation to be technically correct, but those factors should be rather insignificant here.)
Actually I think that the trigger rate should be out of the equation because in the while loop I'm waiting until the acquisition is complete, and then I just calculate the time of the fetch instruction alone, I have tried with 100Khz and the times are the same.
Actually I think that the trigger rate should be out of the equation because in the while loop I'm waiting until the acquisition is complete,
You're absolutely correct, I apologize for that oversight.
I'm testing your code and seeing longer Fetch times (roughly 930ms) when only fetching from one channel. As soon as I specified two channels, the fetch time is an order of magnitude smaller (roughly 90ms). Changing the sample rate, fetch timeout, and record length don't appear to affect this order of magnitude disparity.
So to reiterate:
In the single record case, changing from one channel to two channels doubles the fetch time. That is what I expect.
In the multi record case, changing from one channel to two channels decreases the fetch time by an order of magnitude. I don't expect that.
Furthermore, when I test in the standard shipping example niScope EX Configured Acquisition I do not see this effect.
1) Can you confirm that, in the multi record case for your testing, you see the fetch time reduces when you specify two channels instead of one?
2) Can you confirm that you are not able to reproduce this in the Configured Acquisition example?
I filed a CAR 350194 to look into this behavior more with R&D, but unfortunately I can't promise a timeline for resolution. I would recommend at this point that if you modify Configured Acquisition when setting up your code instead of starting from scratch. You can call into support at NI at any time to check the status of that CAR.
To answer your original question of why one use-case is taking more time than another, this will take more time to investigate as per that CAR. We've already mentioned there is some overhead involved with multi-record fetching, but whether or not the shorter or longer times you are seeing are the correct times is something that R&D will have to answer.
Please let me know if you can confirm what I am seeing by testing.