Actor Framework Discussions

cancel
Showing results for 
Search instead for 
Did you mean: 

Will I get extra buffer allocations sending large data sets through messages?

I can't think of a great title for this, but here's the basic gist of it.

 

I have a DAQ actor that runs and builds up a whole lot of data over 32 different channels. The different channels are used by different parts of the code; there will be at least 2 actors that receive the data, though they each receive different channels of data.

 

Right now, my DAQ actor acquires all of the data and presents it as an array of Waveforms. In my DAQ actor, I use abstract messages to send a subset of the main array to each other subscribed actor.

 

I'm wondering if I'll be creating a bunch of different buffers, and if I should use override VI's to do my processing before sending the data through the AF network. For example, say I wanted to average all of the data to return just one point instead of the whole waveform. The current plan is to send the whole waveform to the subscribed actor, then have that actor do the averaging.

 

Will this be needlessly duplicating my data? I assume I'll be creating new buffers for the data when I split the main DAQ stream into subwaveforms, but I'd also assume the original buffer would be destroyed (or at least reused for the next data set for the next loop) so I don't end up with a bunch of extra *copies* of my data, so I think I'd be fine, right?

 

I could theorize having my "Send new data.vi" call an overrideable VI called "Preprocess" that each subscribed actor's message class would override, that would do the filtering before it gets sent. The abstract message would therefore be attaching my "work" data to the child private data, rather than the parent abstract data.

 

That seems a bit overly complex (not to mention hard to read for other programmers) if I'm not gaining anything. I basically want to avoid having extra copies of a ton of data being sent around within the AF message handling system.

 

Thanks for any tips, and please let me know if I can explain anything a bit better.

0 Kudos
Message 1 of 9
(3,645 Views)

I would recommend a DAQ actor sends the whole data to a Processor actor that does all of the calculations that are required and then the Processor actor publishes the results to specific users. The users could in this case be callee actors in the communication hierarchy under the Processor.
I always try to do the heavy processing in a centralized Processor actor. Hint. The Processor actor can still spawn sub-processors for more modularized processing.

Does this make sense?

Piotr Kruczkowski
Certified TestStand Architect
Certified LabVIEW Architect
0 Kudos
Message 2 of 9
(3,616 Views)

@BertMcMahan wrote:

I could theorize having my "Send new data.vi" call an overrideable VI called "Preprocess" that each subscribed actor's message class would override, that would do the filtering before it gets sent. The abstract message would therefore be attaching my "work" data to the child private data, rather than the parent abstract data.


Just override "Send New Data".  I wouldn't expect this to be too complicated for any programmer that is working with the Actor Framework.

0 Kudos
Message 3 of 9
(3,606 Views)

@Piotr, yes that makes sense, but could you elaborate on why having a Processor actor is better than having the Listener do the calculations? My original worry was about processing/reducing the data *before* it left the DAQ actor, and whether that would benefit me from a performance standpoint or not. I suppose what I'm worried about is the moving all of that data around; do the internals of the AF result in copying of data? I'm wondering if I need to send a DVR (or similar) to intentionally prevent duplicating things.

 

@drjdpowell: Well now that you say it it seems rather obvious 🙂 I suppose I was overthinking it yesterday! My above question still remains though, is there any risk of data duplication or extra buffers when sending the data through the framework? In looking through the AF code I didn't see anything particular that I'd be worried about, but I'm still very new to this, and I'm concerned there may be nuances to the dynamically called code that may result in more data copies than I'd expect.

 

At the moment I don't know if there will be any issues with this amount of data, but as I'm building the foundations of it at the moment I'd like to make it capable of handling data the best way possible rather than try to refactor it later.

 

Thanks for the help all.

0 Kudos
Message 4 of 9
(3,595 Views)

You can use DVR between daq and processor. Processor would minimize the data and forward only subsets. Optimizing data manipulation in processor would decouple complexity from daq actor.

Piotr Kruczkowski
Certified TestStand Architect
Certified LabVIEW Architect
0 Kudos
Message 5 of 9
(3,589 Views)

That makes sense, thanks. Is it required to use a DVR to avoid duplication?

0 Kudos
Message 6 of 9
(3,587 Views)

Remember that a central idea of the Actor Model is to interact only via messages and not share references.  One should avoid breaking the model with a shared DVR unless one has a good reason.  Give messages a chance.  You can use the Desktop Execution Trace tool to investigate buffer copies if you like.

0 Kudos
Message 7 of 9
(3,581 Views)

DVR gives the chance to control the lifetime of data in one thread, so if there is a big chunk of data I wouldn't use messages. That being said the compiler is very smart and in many cases it will understand what you are doing and optimize. I would not use a message only if you find that its to slow or causes memory problems. Like Mr. Powell mentioned DETT is best for this.

Piotr Kruczkowski
Certified TestStand Architect
Certified LabVIEW Architect
0 Kudos
Message 8 of 9
(3,554 Views)

@PrimaryKey wrote:

DVR gives the chance to control the lifetime of data in one thread, so if there is a big chunk of data I wouldn't use messages. 


In the Actor Model, the actor is the thread.  You control lifetime and keep things in one thread by keeping things in one actor.   Is there any actual reason that both the DAQ actor and the Processor needs to build up a big array?  Why not have the DAQ actor just forward new data (without making any copies) and have the Processor accumulate a large amount of data?

 

It's OK to break the rules sometimes, but always know the reasoning behind the rule and be sure you have a good reason to break it.  In fact I break this very rule in more than one app in that I share an SQLite database file reference among multiple actors.  My reasons are the major advantages of an ACID-compliant full-SQL-queriable database as one's datastore.  However, I still have to be more careful because the by-reference sharing has the potential for a whole class of bugs that aren't their with by-value messages.

0 Kudos
Message 9 of 9
(3,547 Views)