spanned file system with zip

[zerotolerance]

Hi

Is there a way of creating a spanned file system with any of the current zip utilities (Native LabView or OpenG etc.)?

Effectively the result should be z01, z02, etc. This format is something that both WinRAR and WinZIP accomplish, so the output would have to be compatible.

The main issue is when compressing a whole directory. Files are rearranged in such a way that the output of all z01, z02 etc. is of the same size (users choice), and I don't quite know how the files are re-arranged in order to achieve that.

Thanks  

[Hooovahh]

I don't think LV, or OpenG can do this nativly.  But you can do this with a command line using 7-zip.

I tested it with the following command.

7z a -tzip -v10000000 Test.zip *.*

This took all the files in the current directory and zipped them into Test.zip in 10MB files.  So in my case I had the following files

Test.zip.001

Test.zip.002

Test.zip.003

[zerotolerance]

First, thanks for your reply.

I actually have a zip utility that compresses strings rather than files. I believe that this utility can be used to create this spanning, but the trouble is it only gives the actual compressed body and not the required headers. 

I don't know much about how the headers and how to create them (i.e. the crc32, relative path etc.) in a zip file.

But more importantly, how are the files within a directory arranged so that constant zip files (i.e. z01, z02 etc.) come out. 

What I mean by this is the convept that WinRAR uses. So, if you have a directory with mixed file sizes in them, and you choose to have split size of 10MB, the output of each compressed file (i.e. z01, z02 etc.) is always 10MB, appart from the last one of course. 

Some help in the headers and the rearrangement of files within a nested direcotry would go a long way.

As for the cmd line, I don't like using that feature with labview because its very heavily limited. It does not give any intermitante control when a job is started. Just to see a simple progress as exe runs we have to jump through hoops, i.e. writting the output to file or clipborad and then reading etc. let alone anything else. and I'm also looking to add alot of other small features along the way, i.e. pausing, cancel, resume progress save, etc. So I definatly cannot take this route.

[Hooovahh]

Well the 7-zip command line program is just a wrapper of a DLL. So you could look up the documentation and perform calllibrary nodes to do the same work but have progress, pause, and stop functionality. Also not a simple task.

[rolfk]

OpenG ZIP does not support spanning or stripped datafiles. The

---

@zerotolerance wrote:

First, thanks for your reply.

I actually have a zip utility that compresses strings rather than files. I believe that this utility can be used to create this spanning, but the trouble is it only gives the actual compressed body and not the required headers. 

I don't know much about how the headers and how to create them (i.e. the crc32, relative path etc.) in a zip file.

---

Most likely you just have the deflate algorrtihme in that "ZIP" utility. It is the compression algorithme normally used in ZIP but has nothing to do with ZIP itself. ZLIB is the most commonly used Open Source library for the deflate() algorithme and is also used in the OpenG ZIP library to implement the ZIP functionality. The function ZLIB Deflate.vi in OpenG ZIP exposes this functionality directly and does mthe same.

---

@zerotolerance wrote:

First, thanks for your reply.

But more importantly, how are the files within a directory arranged so that constant zip files (i.e. z01, z02 etc.) come out. 

What I mean by this is the convept that WinRAR uses. So, if you have a directory with mixed file sizes in them, and you choose to have split size of 10MB, the output of each compressed file (i.e. z01, z02 etc.) is always 10MB, appart from the last one of course. 

Some help in the headers and the rearrangement of files within a nested direcotry would go a long way.

As for the cmd line, I don't like using that feature with labview because its very heavily limited. It does not give any intermitante control when a job is started. Just to see a simple progress as exe runs we have to jump through hoops, i.e. writting the output to file or clipborad and then reading etc. let alone anything else. and I'm also looking to add alot of other small features along the way, i.e. pausing, cancel, resume progress save, etc. So I definatly cannot take this route.

---

So ZIP uses deflate() to compress the individucal file streams but the ZIP format is about creating management around it that allows to store multiple such streams into one physical file. It is a little like a filesystem inside a file.

Adding Spanning and striping to OpenG ZIP would be possible and most of it could be done in the LabVIEW part of it, but there would also need to be some modifications to the underlaying shared library to support that. The spanning or striping is done in such a way that filestreams are cut up when necessary and continued in the next data file. The unzipping algorithme has then to be smart enough to recognize that and combine the streams again when uncompressing. For that the compression changes various flags in the header for each stream, which is the part where the shared library would need to be modified to allow those changed values to be writting (and also recognized on decompression). All in all quite a serious project undertaking.

Refusing to use command line tools because they are ugly is not really a smart decision.

Under Unix/Linux it is totally normal to do so and it makes the whole unix system very powerful as you don't need to write huge monotlithic monsters that do everything from making coffee, to cleaning the house and making the laundry but simply make a tool for each task and tie them together through command line chaining. Divide and master is the magic here, not not conquer and swallow!

Also you have here the choice to use a command line tool and be done in 5 minutes, or write a tool like OpenG ZIP that took me probably several man months of work over time. Or wait until me or someone else changes OpenG to support this feature which could be 10 years.

Alternatively you can also try to write a LabVIEW library to use the ZIPArchive shared library from http://www.artpol-software.com/ziparchive/KB/0610051553.aspx. But that is likely going to be a similar effort as writing OpenG ZIP was.

[zerotolerance]

Hi rolf

You're right, I have the deflate algorithm used for streams. 

Appart for control issues that I have when using cmd, its also the licensing issue. 7-zip is an open source software, but it doesn't provide the corret format. Its format is in the form of (zip.001, zip.002 etc.). This is making me think that it compresses the whole file or directory in ONE zip, then it simply divides the binary format to give the 001, 002 ending. And thats not good. As you mentioned, the zip archive itself has to divide the files inside, and not divide the resulting zip, because that can be done even now. i.e. cmpress everything into one zip, and simply divide the zip into equal parts. 

The format that I'm looking for is the more standard one that WinZip and WinRAR uses, and this is what I believe to be true spanning where files are divided inside the zip archive itself. 

And I don't know an open source software that does that. 

P.S. What I love about the OpenG version is that it allows to step through files and extract a single file inside the archive, without the need to extract the whole thing. 

Thanks

[rolfk]

Aside that I do not believe that 7-ZIP does what you think it does but rather what you think you want to have, you can call other utilities like the tools from the ZIPArchive site that make use of ZIPArchive and call them with the proper command line options. You can even use the original pkZIP over the command line. Just because 7-ZIP doesn't (maybe) do what you want, does not mean that the command line option is not feasable.

There are indeed different options like spanning, striping and binary striping and most likely what you think 7-ZIP is doing is the binary striping. I'm not sure if that is the case and if there is an option in 7-ZIP to specify other options, but I"m sure that finding the command line tool that can do the spanning as you want and incorperating it in your application through the command line is by far the quickest and most easy solution to achieve what you want.

ZIPArchive definitely supports all three modes. It is Open Source and GNU GPL but as long as you call it as command line tool that should have no implications at all on your own application, which is why the command line interface is such a powerful thing. And if you want to incorporate it as shared library and not open source your application you can buy a licence from the developers.

[Hooovahh]

7-zip does not split the way you described. Each file can be opened individually and files in that archive can be pulled out without needing the other archives except for the first AMD last files because they too are split.

[zerotolerance]

Hoovaa and Rolf, thanks for your information. 

I will look into 7-Zip more closely and see if the output can be turned to WinZip & WinRar specs. I keep mentioning these two because my software has to be compatible with them, and those two will have to be able to unzip the output of my spanned files (its a limitation on my end that I won't bore you with).  

When developing closed software, I tend to stay away from GPL licensing because of the shady area in legal terms of what constitutes as open source (i.e. forcing my software to be open source). I've read with mixed reviews that using a GPL software through cmd line would be ok, but then it becomes merky when distributing the exe file with my software etc. So in an effort to avoid any potential risks, I stayed away from GPL licenses. This however may be the one case where I will have to properly look into.