Build a $100,000 labview supercomputer?

[Root_Canal]

I've been testing experimental computer vision algorithms using labview. We plan to eventually convert the code into assembly language or VHDL or embed it in a processor or whatever to make it really fast. However, my insane boss is willing to spend alot of money to speed up development. He wants to buy or build a multiprocessor system that will speed up the execution of parallel labview code. Alot of our code can be made super parallel, so whoopie hooray. Thing is, we already run on a dual core xeon 5140, so it's not like we can just buy a quad core chip and get a couple orders of magnitude performance increase. I see hints that labview can be made to take advantage of a large number of parallel processors, so my question is: how do I run labview on forty processors in tandum? What hardware should I buy? What OS do I use? And will it be faster than running on an 8 core processor or will it be just a glorified space heater? Let's assume I have a $100,000 dollar budget. please help me spend my crazy boss's money. Thanks.

[altenbach]

Sorry, I don't have any specific suggestions, but have a look at the following NI-Week 2008 Keynote clip. It discusses some impressive computing hardware. 🙂

(OTOH, My old LabVIEW 4.0 code still controls an instrument running on a 120MHz Pentium I) 😮

[muks]

Hi Root canal,
                        Can you share us for what application u need to go this far.What is the processing speed requirement?

[Ben]

I agree that you should check with NI now that NI WEek is behind us to see what they want to sell us today.

Another idea:

DIstributed processing - Curtis-Wright offers a product called SCRAMNet that is shared memory linked using fiber. The throughput absolutely blows away anything else I have used, but I digress. If you split the work of your app across multiple high-end processors, and pass the analysis works to seperate machines, the processings cpabilities are limited only by your budget. In your case you may want to skip using the fiber switch they sell (unless you need the redundency) since the switch alone will set you back about $20K.

Ben

[Root_Canal]

The application is a flexible computer vision testbed. We have a generic problem of figuring out how to get a computer to recognize and properly identify objects in a video. We are presently looking for a small set of objects in a small set of scenes, but we will want to adapt to other objects and other scenes in the future so we want to be able to test different algorithms using a variety of software configurations. We can do that right now, but more complex algorithms take much longer to process than simple algorithms.

Alot of the image processing functions that we use can be optimized to run quickly in a parallel computing architecture because either the image can be broken into overlapping segments which are then processed independently or because we are solving large systems of linear equations. However, we can not simply divide the entire computer vision process up into a bunch of completely seperate parallel calculations. There has to be a controller that goes along and dynamically assigns computing tasks to different processors and then pulls the results back together, thinks about it for a fraction of a second, and then distributes new tasks.

Because this will be a testbed for computer vision research and development, and because many problems in computer vision are as yet unsolved, the  processing speed requirement is simply: " as fast as we can make it while maintaining programmatic flexibility." We are not trying to achieve a certain speed benchmark with a known algorithm, we are trying to keep from falling asleep while waiting for it to compute. You can only sit next to a grinding computer playing guitar hero for so many hours before that too begins to feel like work.

[RavensFan]

You might be able to do this with multiple desktop computers.  Create an application as an executable that you can deploy on numerous computers that does the algorithm processing.  Use VI server working across the network to start up or command those individual applications as necessary.  Have your master computer break up the process and determine which PC is doing what.  Distribute data to each of the slave PC's by way of TCP/IP and command it to do whatever task by way of VI server (or perhaps another command channel opened on another TCP/IP port.)  When they complete their task, they send the processed data back by TCP/IP and wait for their next command.

It sounds like an interesting application, but would be complicated.  But I think most of the complication would be in handling the processing algorithms, how to break up data and merge it back together.  But that seems like it would be your area of expertise.

Good Luck!

[Root_Canal]

I think I understand what you are saying, but the thing is that I've read all this hype about labview supporting multicore and multithread processing. Labview is an inherently parallel language, ideal for parallel programming. It can already automatically break up computation between cores on a quad core processor (or an eight core processor if you're on that crazy eight core mac). It does so without the programmer having to specifically delegatae which task goes to what and worry about timing issues between processor cores!

That's really neat! From the literature I've read from NI, all you have to do is make sure you wire up your block diagram in such a way as to not force linear execution (for instance by running everything in series through a flat sequence) and then labview dynamically handles the load balancing between cores. So this begs the question: If labview can do automatically do this with one eight core processor, then can it do this with eight single core processors? Can it utilize sixteen single core processors? Four quad core processors? A hundred quad core processors!?!

It seems like it should be able to allow a user to build a scalable supercomputer without having to go through the pain of creating some complicated control hub program to handle task distribution and timing because labview already does that for you.

Does anyone know if the multithreading capability in labview can handle multiple processors (not just multiple cores) and if so, how many? How do I build a computer with thirty or forty processors running in tandem that would be well suited to parallel processing in labview?

What OS would I use? what kind of architecture should the computing platform have? Should I use special labview software or applications? What kind of data transfer constraints are there? how much memory do I need and how should it be shared by the processors? Can I use multiple multicore processors? Would it be better to use all single core or all multicore?

I don't know what I am doing here; all I know is that NI says that labview 8.6 can run super efficiently on a multicore system and sure makes it sound like labview can run super efficiently on a multi processor system. Cool! Please give me a solid  example of how to buy or build such a system!

Anybody know?

Please Help!

[Ben]

Hi Root,

I have managed to load up 8 CPU with LV apps but it takes a little extra effort to harnes the power of all of them. In a nut-shell, you want to have a lot of parallel code in your number crunching routines. In my case I broke up arrays into groups of eight and called all of my sub-VIs as reentrant.

LV will break up the code into threads and then use the OS to get the work done.

Ben

Message Edited by Ben on 08-08-2008 01:01 PM

[altenbach]

---

Root Canal wrote:

Does anyone know if the multithreading capability in labview can handle multiple processors (not just multiple cores) and if so, how many? How do I build a computer with thirty or forty processors running in tandem that would be well suited to parallel processing in labview?

---

Here is some related discussion, but it is a bit old (LV 7.1). Have a look at vi.lib\Utility\sysinfo.llb\threadconfig.vi in your labview installation.

It seems that the just released LabVIEW 8.6 has a few new enhancements that provide even better support for multiCPU systems (There was apparently some hype at NI week). So that's what you probably should use.

[Ben]

Sorry! 3Meg image is bad bad bad!

Ben

Message Edited by Ben on 08-08-2008 03:11 PM

Message Edited by Ben on 08-08-2008 03:11 PM