Reading multiple files simultaneously or in parallel

[id]

I have an array of filenames that need to be read and fast.  Is there a way to parallel this IO process and speed it up or am I left with one at a time being read in?

[TiTou]

what kind of files are you reading? how big are they? how many files will you have to read at the same time? be carefull to not blow-up you system's memory.

If the files to read are in an array you can them in a for loop and set the for loop to parallelize each instance of the file array.

something like this :

and then :

[Don_Phillips]

Well...

You could parallelize this but I have the feeling you might be shooting yourself in the foot.  All the hard drives I am familiar with only have one read/ write head so the fastest possible scenario would be to read one file at a time consecutively.  If you did try and parallelize you might have the read/ write head moving back and forth between files wasting time.

[id]

I suspect that you are correct from what I am seeing.  Would it behoove me to read all of these files in and then store them as one file later for faster retrieving due to this?  Maybe to a datalog file of an array of data.  Right now the thing takes 11 secs to read 19 files of png type.  Should I read them all in, pack them into an array and save them as a datalog file for faster retrieval later?  Are tehre any techniques like that that could be used?

[Don_Phillips]

One file or many should not make any real difference.  What are the PNGs for?  Depending on what they are used for, your best option would be to switch to JPG or another lossy format and compress it down to reduce file size.  If that is not an option I would need to think about it...

[MStewart]

You could store that array in a shift register, as long as you have enough available memory.  That way it would already be loaded into RAM, for quick access.

[Joseph_Loo]

You should note png is a lossless compression system

[id]

The PNG File is 73 KB. The JPG File as created in photoshop and exported is 498 KB.

[Don_Phillips]

Can you post it?  Your compression settings obviously leave much to be desired...

[TiTou]

---

@id wrote:

The PNG File is 73 KB. The JPG File as created in photoshop and exported is 498 KB.

---

It takes 11 seconds to read 19 png file and your png files are all around 73 kb?