LabVIEW
Zip and Unzip Flattened Double Array fails
Solved!
I need to serialize and zip measurement values stored in double arrays. This method is required and cannot be changed. I am using the OpenG ZLIB Library.
Now the other way around should be pretty straight forward but for some reason this fails every now and then.
The attached vi demonstrates the behaviour. The upper array will produce an error while inflating the lower will work just fine. Both arrays are the same size.
Any advice?
How does it fail? Is the data unexpected? Is there an error out? What is the error out? Check the Deflate error out as well... The error on the inflate might be caused by bad data from the inflate...
The array sizes are the same, but is the data the same? If there is an error, is this reproducible when you input the same data? Or is the error random, and independent of the input data?
Search LabVIEW like a graph!
@t.gru wrote:This is completly reproduceable but i can't tell which exact input data causes the error.
But if there is an error on dataset A, will the exact same dataset A always produce an error?
We'll need to find out if the error is related to the input or random. I'd put my money on the input... If so, we could establish a set of inputs that always produce errors, and then look for a common thing in the datasets.
in practice, run the code with datasets. If there is an error, add the dataset to an array of datasets. That would build a database of erroneous datasets...
I suspect there is a problem with strings being truncated when a \00 or \00\00 is encountered, but that's just a hunch. That means it could still be a problem in the inflate... I've been using the zlib dll, but not though the OpenG VI's... I haven't seen this problem. It might as well be caused by an upgrade to a newer LabVIEW version. I'm not aware of any differences in how CLFN are handled, but then this kind of stuff isn't that common...
What is the LabVIEW version you are using? Are you using the latest OpenG library?
I found out the error disappears when the "expected length"- input is wired correctly.
(And btw. when there is selected compression level 0. This is senseless, but then apparently the algorithm doesn't struggle)
As a workaround you should include the original string length information into the compressed string. Something like that: (the string labels are mixed up a bit, don't be confused)
Wiring the expected length crossed my mind as a potential cause. It should be capable of handling this automatically though. It should try to allocate enough memory to deflate, and if it doesn't fit, it should expand. There is a limit, I think it doubles data size 4 or 5 times... Wiring the expected length will be faster, since deflate won't iterate until the deflated data fits, it simply returns an error if it doesn't (IIRC).
Hey Wiebe,
there must be such limitation with automatic buffer size doubling. I wired all necessary string sizes with the first channel. It shows that the date has to be deflated mor than 100 times.
As you can see, Plot 1 has many parts with values of zero, so this should be very compressible, while the other channel has noise on the signal. And obviously the Inflate.vi doubles the buffer 5 times and then stops with an error.
I cleaned up the vi a bit for this investigation...
You should be able to open the Deflate VI. It's not difficult (or at least the auto size part is not).
Note that doubling the data size grows exponentially, and it might not start with 2X. I'd expect that this sizing is chosen so the theoretical maximum deflation is possible, but that might not be the case (or even possible).
Specifying the length is the best solution from all points of view (except there's 4 bytes more data). Most importantly, it will make your life a lot easier. It's what I do in my code.
The problem is very simple: If you look in the Inflate.vi you see that the loop gets aborted when:
- the inflate() function returns with anything else than -5, which indicates that the buffer was to small
- the loop has iterated already 5 times
- or the input length was greater than 0
Obviously the first reason is useful. If the function returns anything else than -5 it was either successful (returns a value greater or equal to 0) or it had an other error than that the buffer was to small.
The second reason is disputable but I wanted to avoid creating a VI that might never abort and then create eventually (quite quickly since the buffer is increased by doubling it every time) an out of memory error, which is pretty nasty in LabVIEW as it will often simply shutdown without letting you save anything.
The last reason, I figured that if the caller provides a size, he better knows the actual buffer size needed to store the inflated data, or he has messed up somehow.
Increasing the constant inside the loop in the Inflate VI from 3 to 6 would solve this error. I think 6 or 7 would be acceptable but there should be some limit. When this VI was developed (around 1998/1999 I think) 100MB of total memory in a computer was considered a huge amount!!
