Writing files bigger than 2^32 bytes issue

[LSadava]

I use LabWindows/CVI recently updated to 2017. 

My application decodes small raw data files and writes the data out to one large file. We're attempting to write files in the 10GB range but are having some strange results; The data is not written correctly to the file. The filesize as read by windows is right (~10G) but the data in the file is not right. It seems to get to byte 2^32 and continually overwrite this byte in the file. 

I ported the application over to be 64 bit and the issue remains. 

When I look at the FILE * object that I am using fwrite on, it seems to be only using the top 32 bits for the FILE struct fields buffer, bufferEnd, next, readEnd, and writeEnd.

I'm fairly sure I missed a step porting to 64 bit but I can't figure out what. Everything I read says that fopen and fwrite should work on a 64 bit system but I cant seem to make them work with 64 bit pointers.

Any advice?

[Dane.S]

I wanted to make sure you were aware of the following document:

http://zone.ni.com/reference/en-XX/help/370051AG-01/cvi/programmerref/porting32bitto64bit/

This includes most of the consideration for porting your code to 64bit applications. Can you post a screenshot of what the code looks like?

[rolfk]

This document only describes the differences between 32-bit and 64-bit applications. It says nothing about support for 64-bit file offsets (_LARGEFILE64_SOURCE in the Posix standard) which is entirely independent of the fact if an application itself is 32 bit or 64 bit. If Large File Support is available even a 32-bit app can perfectly write to files up to 2^63 bytes (since the file offset on all relevant platforms including Windows is a signed (large) integer, you loose one bit).

For Windows the relevant APIs such as SetFilePointer(), and GetFileSize(), need to be called with additional parameters to handle the upper 32-bit of a file offset in order to support large file access, or you need to use the newer Ex() variants of those APIs with a 64 bit offset parameter. And the C runtime library for the compiler needs to handle that properly in the file object wrappers it creates. The Windows API WriteFile() and ReadFile() functions itself handle >2GB files transparently. The problem only occurs when the C runtime library tries to handle the file pointer explicitly in its fwrite() or fread() or similar functions.

According to some post here in the fora, you had to use functions hidden in lowlvlio.h to allow support for handling files larger than 2GB. But in my CVI 2013 install on this machine there seems nothing contained in there that would hint at 64 bit file offset support in any way.

The current recommended solution still seems to be to use directly the Windows API functions SetFilePointer(Ex)(), GetFileSize(Ex)() and SetEndOfFile() to access files that can get bigger than 2GB.

[LSadava]

Thanks for that Rolf. I have tried with WriteFile and ReadFile but they are so much slower then fread and fwrite that I was hoping to find a way to make it work using them, maybe it will not be possible though. I'll try running it with WriteFile.

[rolfk]

That makes no sense. WriteFile and ReadFile are the platform APIs. all the C runtime libraries I know about resolve to use them for their file related functions.

[LSadava]

Ok well maybe it's something to do with me casting the file pointer int as a FILE * to pass around so I don't have to change all my function inputs

[rolfk]

Typecast in C is a cheap operation, so I'm sure it must be something else. Maybe an involved caching in the C runtime file functions. Or a particular way of how you use the functions. But are you saying that you typecast the LabWindows CVI File* pointer to a Windows handle for the WriteFile functions or what?

[LSadava]

Yes just to test if WriteFile would work.

...

frange = (FILE *)OpenFile(rangeFileName,VAL_WRITE_ONLY,VAL_TRUNCATE,VAL_BINARY);

...

Then writing 32 bit numbers to the file with this:

void writeInt32(FILE* fd, int32_t val){
           //fwrite( &val, sizeof(int32_t), 1, fd);
           WriteFile((int)fd, (char *)&val, sizeof(int32_t));
}

...

[LSadava]

The files are writing out properly but about half the speed.

[rolfk · Accepted solution]

---

@LSadava wrote:

Yes just to test if WriteFile would work.

...

frange = (FILE *)OpenFile(rangeFileName,VAL_WRITE_ONLY,VAL_TRUNCATE,VAL_BINARY);

...

Then writing 32 bit numbers to the file with this:

void writeInt32(FILE* fd, int32_t val){
           //fwrite( &val, sizeof(int32_t), 1, fd);
           WriteFile((int)fd, (char *)&val, sizeof(int32_t));
} 

---

Now you had me stumped for a moment but I see what happened.

First, typecasting like this is highly unsafe. Most C runtime libraries know two different file IO function types. One uses the FILE * pointer which is usually a pointer to an opaque structure containing all the information necessary to manage the file access. The other is an integer filedescriptor which often is implemented as some sort of index in an internal array that may contain FILE* pointers or something else. Some C runtime libraries provide non-standard conversion routines to convert from one to the other.

Here seem to be one of a few situations present.

- The FileWrite function which is a LabWindows CVI high level function either has a wrong signature that seems to use an integer fd instead of a FILE* pointer despite being not so.

- In LabWindows CVI an integer fd is really the same as a FILE* pointer.

- The FileWrite() function internally has some automagic backwards compatibility operation that detects what type of "file descriptor" is passed in and does the conversion when needed.

I suspect it is the second one and such nasty things are one of the reason why porting of LabWindows CVI to 64 bit seems such a difficult project. Actually if this is the case it is really impossible without backwards incompatibilities.

Anyways, I was not referring to this LabWindows CVI WriteFile() function but to the Windows API WriteFile() function. And yes this one uses a HANDLE as "file descriptor" and you should definitely not assume that the LabWINDOWS CVI FILE* is the same. So you will also need to use the Windows API CreateFile() to open the file handle, and CloseHandle() when you are done.

Use of the LabWindows CVI FileWrite() function is almost certainly no solution for your problem as it is a high level CVI library function that internally uses the lower level file IO functions which seem to not work for files >2GB.