Memory management when converting a set of PNG images into a large 3D array

[Yamaeda]

"The absolute upper voxel limit (taking into account the build extents of the 3D printer) is 76000x1280x800 = 7.7824E10"

I assumed reducing this by 8 was a rather decent start, i do hope the SGL calculations are done 1 layer at a time ... 🙂

/Y

[wiebe@CARYA]

>As I said, a smart ordering of operations and only operating on a single 2D output array by doing the summing in place is probably the solution.

Agreed. But I'm not getting the exact problem of these calculations. I'm not sure if OP is making this much to hard, or that the problem actually is hard. Either way, TDMS seems out of place to me... But if it "works"...

[altenbach]

---

@Yamaeda wrote:

"The absolute upper voxel limit (taking into account the build extents of the 3D printer) is 76000x1280x800 = 7.7824E10"

I assumed reducing this by 8 was a rather decent start, i do hope the SGL calculations are done 1 layer at a time ... 🙂

/Y

---

Agreed. Converting to a compressed boolean array is only part of the story. Ideally, the data should be saved in the compressed format (1bit/pixel) from the beginning.

In order to efficiently do the SGL calculations, we also need fast random access (read, write) to any particular voxel element, which is relatively trivial, of course. Personally, I would probably compress the bits into U32 or U64, keeping the indices smaller. (same overall memory as using U8).

(There are also edge effects if the array size is not divisible by the #bits/element and that might need to be handled).

[paulygon]

---

wiebe@CARYA wrote:

>As I said, a smart ordering of operations and only operating on a single 2D output array by doing the summing in place is probably the solution.

Agreed. But I'm not getting the exact problem of these calculations. I'm not sure if OP is making this much to hard, or that the problem actually is hard. Either way, TDMS seems out of place to me... But if it "works"...

---

Yup TDMS was just an idea I didn't end up implementing it - I was just experimenting with methods of offloading from RAM to disk and the normal binary read/write methods weren't working well for "streaming" (I guess that's why TDM streaming was better ).

@altenbach hits the nail on the head here - the elephant in the room is the 3D SGL array that I generate using the 3D U8 array. So I think the solution here might be as follows:

1. @Load the images into memory as a 3D array using the packing method described by @Yamaeda and wiebe@CARYA
2. Re-slice the 3D array into 2D slices looking from one of the horizontal directions (X or Y) and save to disk.
3. @Perform my calculations on the 2D arrays to generate 2D SGL arrays (instead of the current 3D array) as suggested by @altenbach
4. Finally, my output from all of this will be a set of altered PNG slices (depending on results from the calculation) so I will have to stitch these all back together.

If anyone has any handy tips for part no. 2 (re-slicing the array and saving to disk) in a memory efficient manner I'm all ears.

I will update when I have had a chance to implement some of these ideas but it may take me a little while...

~P

[wiebe@CARYA]

I'm still thinking you want a 2D projection of the cube, and that's why you need the 3D array of doubles? I'll probably have to print out this entire thread and study it for a few hours...

So with a two pass strategy, you can make a 2D buffer to hold the projection, and process each individual PNG to it.

Then, use a second pass to process each individual PNG with the projection results, and stream to disk.

No need at all to hold any 3D  array.