Reentrant Execution gets slower.... (DATABASE TOOLKIT)

[ktian]

Maybe MySQL database do not support 1000 connections on your computer, look at this reference manual.

http://dev.mysql.com/doc/refman/5.1/en/too-many-connections.html

[D.S]

HI Yamaeda,

 I am a little bit confused with the calculations you presented.

What I can interpret:

If each clone is writing 1 row/sec, then Ideally 1000 clones must write 60,000 Rows/min.

And if 1 row consumes 600 bytes, then in 1 min 60,000 rows will consume (60,000 X 600) bytes/min.

=36000 KB/min

=35 Mb/min (Approx.)

=0.58 Mb/sec (Approx.)

So, may be 0.58 MBPS network speed could be sufficient to insert required amount of data across the network.

[D.S]

No, database and application machines are different. Network speed is approx. 100 Mbps. CPU & memory usage is moderate. About 4 Gb RAM with Quad core processor (Application Machine).

VI's given by you had 2 column, we expand the code for 52 columns. We avoided sub vi's to max the executions speed. Hence code snapshot is large. I am attaching the VI.

[Yamaeda]

---

@D.S wrote:

HI Yamaeda,

 I am a little bit confused with the calculations you presented.

What I can interpret:

If each clone is writing 1 row/sec, then Ideally 1000 clones must write 60,000 Rows/min.

And if 1 row consumes 600 bytes, then in 1 min 60,000 rows will consume (60,000 X 600) bytes/min.

=36000 KB/min

=35 Mb/min (Approx.)

=0.58 Mb/sec (Approx.)

So, may be 0.58 MBPS network speed could be sufficient to insert required amount of data across the network.

---

My calculation seems to assume 8560 lines transmitted from each client. Thus 8560*600kb total data/min. I realize now that's not what happens ...

Your numbers looks about right with a couple of exceptions. I assume network traffic like rs232 and similar has some overhead, so 9-10 bits/byte is realistic, thus 36Mb/min (approx, minor change)

Networks measure in bits, not bytes, so 0,58Mb/sec requires 6 Mbit/sec. Depending If it's a hub network instead of switches you probably cant reach much more than 4 Mbit in a 10 Mbit network (but close to 10 with switches).

It might be hard to get it to work on a 10 Mbit network but it should be possible.

How's the database setup? Does it lock the whole table or just the row when inserting? Does it allow parallell inserts?

We'd be helped by some more information, as how it performs if you run both clients and server on the same computer and such.

/Y

[Yamaeda]

Hah!

From the linked document above (http://dev.mysql.com/doc/refman/5.1/en/too-many-co​nnections.html) the default is 151 max connections.

151*60 rows/minute = 9060 rows/minute ... sounds about right?

/Y

[ktian]

Can you please also attach your main VI that calls 1000 clones? Thanks.

[D.S]

Hi ktian,

Kindly find the attached CALLER.VI...

[ktian]

OK, I figured it out now.

I calculated the execute time of each while loop and wrote it into database too. The result is, when 1000 clones running together, a single loop will take about 6 seconds to run.

The reason is that LabVIEW controls mutilthread switching, when 1000 clones running together, LabVIEW needs to control switching between these clones. If there are too many codes in the loop, it can not be finished in the time LabVIEW gives to it, then it needs to wait for next time LabVIEW activate it. In your VI, the big loop maybe need this kind of switching more than two times.

I make some change to your code, if there are so many columns, it's a bad idea to use ADO parameters which will make the VI huge.

Look the VI I attached, it writes about 60000 rows with 1000 clones in one minute. You just need to modify the format string to get the data type you need.

[Ben]

---

@ktian wrote:

OK, I figured it out now.

I calculated the execute time of each while loop and wrote it into database too. The result is, when 1000 clones running together, a single loop will take about 6 seconds to run.

The reason is that LabVIEW controls mutilthread switching, when 1000 clones running together, LabVIEW needs to control switching between these clones. If there are too many codes in the loop, it can not be finished in the time LabVIEW gives to it, then it needs to wait for next time LabVIEW activate it. In your VI, the big loop maybe need this kind of switching more than two times.

I make some change to your code, if there are so many columns, it's a bad idea to use ADO parameters which will make the VI huge.

Look the VI I attached, it writes about 60000 rows with 1000 clones in one minute. You just need to modify the format string to get the data type you need.

---

"Threadconf" (is it under platfrom maybe ???) lets us control how many threads LV uses.

Ben