String shared variable max length?

[guybrush_threepwood]

Hi!

I'm experimenting with sending data via shared variables. I'm planning on sending some data in json format (meaning string type of variable size) through a shared variable. Is there a limit to how long of a string I can send over a shared variable?

Thank you!

[crossrulz]

Let me just say that this is likely a bad decision.  You probably want a messaging communication scheme, not a tag (single value, stays there until overwritten).  If going over a network, Network Streams or TCPI/IP are good options.  If inside of an application, use a queue.

[guybrush_threepwood]

Hello, and thank you for your advice. Would you mind expanding on why this is a bad choice?

I am not a LabVIEW programmer per sé (I use Labwindows/CVI mainly, and is pretty new at it to boot), I just need to communicate with a colleague's code - mainly a measuring system with a few controls - and thought that shared variables would be a nice choice for displaying collected data in real-time (that's what it's for isn't it?).

My aim with the JSON strings was to ensure that we could easily add or remove entries in the data that's sent without having to sync an upgrade of the two programs or "reserve" spots in arrays for possible future entries and such. To make the construction more flexible, so to speak.

I'm very interested in hearing why my approach would be bad and how you would go about making a flexible data transfer!

[crossrulz]

Generally speaking (not just LabVIEW here), there are 3 main types of communication: Tag, Message, Stream.

Tags are for holding the latest value.  They are persistent (stay there until overwritten) and do not hold history.  So you have no clue when the tag was written or if another write was performed between your last read and the current read.  I will use these for "constants" (ie settings that are set at the beginning of the application based on a config file or an actual constant) within an application.  Inside of an application, I typically use Global Variables.  I do not typically use these between applications.

Streams are a very fast, usually lossless (you should get all of the data points), single data type, typically to a single reader.  Classic example here being a Producer/Consumer.  Within an application, use a Queue.  Between applications people tend to use a Network Stream.

Messages are sort of a mix between a stream and a tag.  They could be lossy or not.  They could be sent to multiple receivers.  The data types are dynamic.  But I use these all the time.  The biggest difference between a message and a tag is that the message is gone once the receiver reads it.  Inside of an application, I typically use Queues or User Events.  Between applications, I use TCP but I know a lot of people like to use Network Streams.

So in your case, you want to send a message to the other application stating "Here is some data and here is what the data is".  The other application can then take that message and process it.  If you used a tag, it would have to constantly process that data to verify it did not change.

To take it a step further, Network Published Shared Variables have all kinds of issues and I actually state them as "evil".  They are slow (data isn't written to the server until enough data needs to be written or a timeout), introduce all kinds of weird race conditions (mostly due to how the server is updated), and worst of all they are extremely insecure (ANYBODY on the network can write to it and possibly cause all kinds of issues).

[guybrush_threepwood]

Thank you very much for taking your time to expand on your stance!

I'm sorry to say I'm still not convinced that using shared variables in my case is a bad idea. Your arguments against them, if I understand it correctly:

1. SV's are persistent and do not hold history
2. I cannot know when a new value has been written
3. SV's are slow
4. They are insecure

Regarding (1), this is not a problem for me. The data will be sent at regular intervals and I only want the latest value.

Regarding (2), I'm not convinced this is true. Each shared variable has a timestamp when it was last modified (that can be seen in the NI Distributed System Manager), and even if it did not, I can create callbacks in both CVI and LabVIEW that are called when a shared variable is changed. I have myself experimented and written code for this following examples. If I update a shared variable too often, I might lose data, but since my application will at most send new data at 10 Hz, I do not currently consider this a problem.

I need several readers, so unfortunately network streams are not an option, otherwise I might have used that.

(3) and (4) I completely buy, but at the moment I favor the slow shared variables over a fast TCP messaging system, only because of the simplicity of shared variables. As both me and my colleague are pretty new to LabVIEW and CVI, I think setting up a well-functioning lossless protocol with possible handshake functionality may not be worth the time, as well as none of us has any experience doing so. The insecurity issue is true, but if we isolate this network from the internet it should be fine, right?

Sorry for debating, I know I'm much less knowledgeble in this field, but I would like to understand why I should adopt another person's stance before I do so.

[guybrush_threepwood]

I'm still interested in getting an answer to the question if anyone knows:

Is there a limit to how long of a string I can send over a shared variable?

Thanks!

[RavensFan]

I have yet to find any documentation on this.

I'd suggest your run an experiment to see what the limit might be.  Send a string of a certain length.  See if it gets through.  Sent a longer one.

Try increasing lengths until you find a point where it errors out or truncates the message.  Or at least as high as the longest message you ever expect to send.

[crossrulz]

I'm not sure about an actual limit on the string size.  But I will refer to the LabVIEW Help (Sending Shared Variable Data over the Network Immediately and Flush Shared Variable Data VI).  Both articles mention the 8kB buffer in the Shared Variable Engine that has to be filled up before data is sent over the network (or a 10ms timeout).  I am not sure what will happen if you try to send a string larger than 8188 characters (a string data has an I32 to state the size at the beginning, so 8*1024-4).  I refuse to use Network Published Shared Variables partly because of this interaction (almost always using the 10ms timeout, making communication slow).

[guybrush_threepwood]

Thanks for all the suggestions. The links were also very informative. The 8k buffer is what I was looking for, but I also did some tests myself.

I set up a small LabVIEW program which reads a file into a string and sends that string to a shared variable. On the other end I put a CVI program that receives the string through the shared variable, checked how many times I need to get the data (for instance, if the buffer was filled I would expect having to get data several times), and the resulting length of the string I got. The result was that I could get at least 100 000 chars with a single "get", and neither program complained about sizes or buffer problems. So as far as I can see, at least 100kB is no trouble. Not sure how the 8kB buffer plays into this, but it doesn't seem like you have to worry about it as a user.

Not sure how the time required to send data scales with the string length, that I will leave for someone else to experiment with.

EDIT: Tests done with LabVIEW 2020