Linux 64 thread limitation In library.

[n2new]

Granted my situation is a bit unusual, perhaps there is a work around but I can't find one. I am using  a read from pipe in a for loop
after scratching my head for days and hours. I have determined that in deed the linux threading library's with labview are limited to 64.  I have 80 threads available  but they can't be used because the remaining 16 processes just go into pipe wait. the library in labview seems to be at fault here. 
I even tried using 4 separate loops to divide the the 80 threads into 20 each but it doesn't matter. the Pipe read will not read any of the processes after 64 threads. Not expecting a fix but just want confirm that my findings are True.  Is this limitation and why? Or perhaps it is a bug and unintended limitation?

[wiebe@CARYA]

Maybe something like this: linux - Is there a limit on how many pipes I can use? - Stack Overflow:

there is a limit on opened file descriptors, but on current systems it is significantly big (several thousands) in practice. AFAIU, POSIX guarantee a small limit (perhaps only 20). Your system has probably a file descriptor limit per process, and another file descriptor limit system wide....

Your system has probably a file descriptor limit per process might be relevant...

No idea if your pipes are file descriptors...

[n2new]

All ready looked into his.  I have verified the assigned descriptors and they are there. For what ever reason the pipe read limit is 64 but a peak at the threading library so file is named threads 0 through 63 which is very telling. It also does create 80 pipes, just won't read more than 64 at a time. There really doesn't seem to be a work around unfortunately.  If I am using my read pipe incorrectly I am certainly open to that. They are named pipes and use descriptors. 

[TonyStone]

I think you are the same person from a few months back where we tried to figure out how to read a pipe line by line.  Not sure how you ever made out with that.  But I had a terrible performing solution and tried to run it on an old machine with 100 plus cores.  I ran into the same issue where no matter what I couldn't run more than 64 threads.  Sadly I had to give up on LabView for my projects backend.  I wrote the back end in Lazarus.  The front end is a LabView program still but I am gonna rewrite it in Pascal using Lazarus!  Not sure if this is an acceptable solution for you but I can tell you that in my experience since moving over to Pascal it is night and day the difference in power of the language itself.  I run 150 plus threads on that machine reading LINE BY LINE from 150 pipes.  Maybe you can do what I did and write your application in part in a more versatile and powerful language like Pascal.  Real easy to learn!  I even tried messing with LabViews config files and got no where....  If its just more than 64 pipes you need to read I think your only choice sadly is to use a more capable programming language and write that part as a separate project.  Python is capable and you can probably learn to read pipes in that quick but unlike a Lazarus program it will not be a native application that can run on any OS without tons of dependencies etc.  Just so you know the labview config file is in your home directory in nationinst/.config  <---- took me for ever to find it.  but like i said the changes i made didnt help me.  Good luck man! 

[wiebe@CARYA]

---

@n2new wrote:

a peak at the threading library so file is named threads 0 through 63 which is very telling. It also does create 80 pipes, just won't read more than 64 at a time. There really doesn't seem to be a work around unfortunately.  

---

You could try to skip the threading library and try to call the OS API directly. If the threading library so is an OS library, I doubt calling it from another language will help. 

That threading library could be made 20 years ago...

[TonyStone]

I was under the impression LV is automatically threading my loops because it has some sort of a "smart compiler".  My issue may not have been the same as OPs but I think we are running into the same limitation of LabView.  I am quite sure LV on Linux is just using the systems threading library.  Likely pthread.  I can confidently say the threading library on my linux distro is not putting a measly 64 thread limit on my application.  It doesn't in other languages.  I would love to see a solution to this so I don't have to rewrite my LabView front end application in a real programming language.  I'll keep following this forum thread.

[wiebe@CARYA]

LabVIEW was multithreading it's code when the OSs (well, Windows 3.11) only had 1. Even the CPU only had 1 thread on 1 core...

LabVIEW threads are not OS threads.

If you're using for loops set up for parallelization, then LabVIEW has limits.

Maybe it's time to put some code on the table. I don't have much knowledge of Linux or pipes on Linux, but I do know my LabVIEW.

---

@TonyStone wrote:

 I would love to see a solution to this so I don't have to rewrite my LabView front end application in a real programming language.

---

Not in the mod for a flame war..

[wiebe@CARYA]

---

wiebe@CARYA wrote:

If you're using for loops set up for parallelization, then LabVIEW has limits.

---

LabVIEW's ini (LabVIEW.ini on Windows) file key "ParallelLoop.MaxNumLoopInstances" is quite low by default. Not sure if it's 64, but it could be.

You can safely set it to 10000, or higher. Somehow this limit for parallel for loops got linked to the supposed nr of cores or CPU threads...

[TonyStone]

In my code I am using a while loop.  I did find the setting for maxnumloopinstances in my config file.  Actually I think i had to add it.  It however did not make a difference.  I will open my project up again this weekend and see if I can post some screenshots...  I may have to make another reproduceable example as I already modified my LV frontend application and have the backend doing all the threading in something other than LV.  Wasn't trying to start a "flame war" by the way.  I am never in the mood for that nonsense.  I am just becoming frustrated with what should be quite simple according to the LV documentation i have gone through.  Is there anything special that needs to be done after modifying the configuration file?  Just restarting LV should be enough, correct?

Thanks

[n2new]

I do not have parallelization enabled, not needed.  Each iteration is a file descriptor from a named pipe read. The thread limitation could be caused by the read pipe library.