.Zip file limit

[nzamora]

Does LabVIEW 2015 x64 or 2016 x64 still have a 2GB limit? I read this post from 8 years ago where this was the case, but was hoping that I was just doing something wrong when my file stopped adding files at 2GB. If I want to package a bunch(1000's) of files into a single easily transported file, am I going to have to use a TDMS?

[Bob_Schor]

I was wondering if Zip file contents all need to be in memory at once, hence giving a 4GB limit for LabVIEW 32.  However, when I tried to read about Zip files, I found something that said the Zip file format also imposed a 4GB limit (which surprised me).

On the other hand, LabVIEW uses a 64-bit Position indicator for file I/O, which suggests (to me) that you could stream text or binary data to a file and definitely exceed 4GB (as long as you did it in smaller "chunks".  Don't know if TDMS supports this ...

Bob Schor

[nzamora]

I considered if contents had to be in memory, but given that it's come from one disk space to another, it doesn't really make sense to. Same for unzipping, you specify where you want to files to go, so that they don't have to be loaded into memory.

I can do TDMS, but most of the data is images, and the only way I can think to store them is by converting to array which is a pretty slow process compared to reading an actual image file.

[nzamora · Accepted solution]

I'm still unsure on LabVIEW's true inherent limitations. I have anecdotal evidence to support it being limited to 2Gb both in 32 and 64-bit LV. 

I have gotten around it by making a wrapper for a .NET(4.0) call to (de)compress a folder. This is a bit more limited than the LV method of putting each file in 1 by 1, but it works for >2Gb worth of files.

[chembo]

I noticed the same thing yesterday with LabVIEW 2014 32-bit and then I tested it with LabVIEW 2016, 32-bit and 64-bit.

Just for the test I was trying to put a 10 GB directory (including all sub-directories) in a zip file using the native LabVIEW ZIP VIs. The VI completes without any errors, but the size of the resulting zip file is about 2 GB and corrupted.

Then I tried with the OpenG ZIP Tools. It doesn't work either, well at least it errors out (Error 7, File not found ......)

I can create a ZIP file from the same directory using 7Zip and the size of the zip file is about 9 GB.

I will probably use the 7Zip command line for compressing big ZIP files in LabVIEW.

[rolfk]

The currently released version on the OpenG ZIP library on the Tools Network is quite old and does not include any 64 bit support. The whole problem is actually twofold.

First the ZIP library must use specifically 64 bit file IO functions in order to support ZIP files bigger than 2 (4) GB. And then it needs to support a ZIP extension that allows for the internal directory structure to span across the 4 GB limit inside the file. Another ZIP extension is necessary for support of individual files being more than 2 GB inside the archive, but that is usually less of a problem. I'm quite sure that the NI ZIP library is also using the older source code from the ZLIB distribution that has not yet officiallly added 64 bit support but simply has been recompiled for 64 bit under 64 bit LabVIEW.

Basically compiling the library for 64 bit does absolutely nothing to support 64 bit ZIP files. On the other hand, adding the necessary 64 bit ZIP extensions to the code would support 64 bit ZIP files even when run under 32 bit LabVIEW. Maybe I get around to update the OpenG ZIP library in the near future.

[DavidAMoore]

The MGI Library has zip file tools that handle large files and have file-at-a-time functionality. It's based on SharpZipLib. Download it using VI Package Manager.

[Cranky]

Any chance you've gotten around to this? I'm using the MGI library at the moment but I'd like to see a native solution so that shared code doesn't require additional externals.

Thanks!

Cranky

[JÞB]

---

@Cranky wrote:

Any chance you've gotten around to this? I'm using the MGI library at the moment but I'd like to see a native solution so that shared code doesn't require additional externals.

Thanks!

Cranky

---

Rolf was not suggesting that he would update native LabVIEW functions.  He or others may get around to updating the OpenG toolkit.  Or, since it is Open, you could go to  LAVA and submit a update yourself. 

[nzamora]

Here is the code as LV 2017 snippets I have been using for zipping/unzipping for the last year or so. The only problems I have ever had is not knowing the status of the action since it's all in 1 node, rather than adding file by file, and if you try to unzip to a non-empty directory, it errors.