According to the lz4 description, it should handle >500 MB/core, so running it on 8 cores would solve my problem. However, this didn't work for me. And, for comparison, the 500MB/s benchmark was done on data that is much more densely packed with information than my regular data. So this is odd and we are troubleshooting. I guess now this is a topic for a lz4 forum, not LV anymore. However, in case someone in the LV forum is looking for implementing a streaming compression like I do, I will keep posting updates here in case I make progress. Thank you all for now!
I assume you've tried different compression levels with the ZStd library? They'll trade off compression and time, though not linearly, and possibly not even monotonically. lz4 certainly should perform better than ZStd, although all parameters seem to be very data-dependent.
Thanks Greg, I did. For Zstd, compression "-5" was used since this should give the fastest performance. For lz4 we didn't find an explicit value to use for the acceleration. Acc. of 1 is default and each additional level should give roughly a 3% increase in speed. However, we tested until acc. level 100 and didn't see an increase in speed. Will try some more.