vi.lib's NI_Bytestream library documentation?

[BertMcMahan]

I stumbled across someone in another thread mentioning vi.lib\Array\NI_Bytestream.lvlib. It's a class library that, well, appears to operate on bytestreams. I've got an application that needs to ingest a bytestream, so this looks promising, but I can't find any documentation on this library.

I can't find anything online or in the example finder. None of the VI's have comments or VI descriptions.

I could stumble my way through it to see if I could use it, but figured I'd ask the community before I started a wild goose chase.

[avogadro5]

vi.lib\net\SSH\NI_SSH.lvlib uses it, haven't dug in that deep though my issues have been in the SSH lib itself.

[thomas-allibe]

---

@BertMcMahan wrote:

I stumbled across someone in another thread mentioning vi.lib\Array\NI_Bytestream.lvlib.

---

Me too, so I gave it a try! (I'm not aware of any documentation...)

Fortunately the API is pretty simple. It seems to be an "unlimited" size U8 buffer optimized for concatenating and cutting data from start.

[rtollert · Accepted solution]

NI_Bytestream was a mercilessly rushed attempt to minimize the number of buffer reallocations and copies when doing protocol-type stuff. To be honest, I never got around to confirming that it's the right approach, Exhibit A being, well… SFTP's performance. The most salient alternatives would probably be things like “single giant ring buffer tied to each stream and eat the I/O copies” or “huck Data Value Refnums holding page-aligned buffers around and run scatter-gather I/O on those”.

Nevertheless, yes, NI_Bytestream's API ought to be simple enough to stumble through successfully, and it's almost certainly faster than manipuating byte arrays directly. As an undocumented/unsupported API, NI has the right to yadda yadda yadda you know the rest.

[BertMcMahan]

Much appreciated! I have a need to ingest a continuous bytestream, and was dreading making a way to do that when I stumbled onto this. I couldn't quite figure out how it worked internally, but it's bound to be better than something I'd come up with on the fly!

[rtollert]

Basic instructions; all of these VIs/VIMs are contained in the top level of the lvclass:

• Concatenate.vim accepts two of any of the following: Bytestream, array of Bytestream, or array of U8. It returns a Bytestream which concatenates all inputs in order.
• Concatenate2.vi accepts two Bytestreams and returns two Bytestreams; the first is the concatenation, the second is empty.
• Size.vi and Is Empty.vi are basically what you'd expect, as alluded to by the very array-like icons.
• Cut from start.vi chops off the first N bytes from a Bytestream and returns them as an array of U8.
• Copy from start.vi is the same thing except the Bytestream is unmodified.
• The normal ways you'd create a Bytestream are to either drop down a default lvclass constant or to wire two U8 arrays to Concatenate.vim.

The principle of operation is to represent the data as a list of arrays of bytes, instead of a single array of bytes. As you consume bytes from each individual array (“datum”) using Cut from start.vi, rather than doing Delete From Array every time you read bytes out of the start of the array, it increments an index that points to the first “actual” element of the array. Likewise, Bytestream maintains an index to the first datum that contains unconsumed data. Once the number of datums grows large enough (currently 200, a number I picked out of a hat), the Bytestream is compacted back into a single datum.

[BertMcMahan]

Much appreciated! Do you have any guidance on sizing/frequency of additions?

In other words, if I'm reading a continuous stream of data, how often should I Concatenate? Would, say, 10x a second be a good starting point, or would it depend more on the volume of incoming data?

[rtollert]

I can't offer any guidance about that, sorry. Sight unseen... for fewer than 8-12 bytes, I would stick with normal byte arrays for concatenation.

[BertMcMahan]

Gotcha, thanks. My data's coming in at around 15-50 kB/sec. I'll try to play around with it some and maybe do some benchmarks. If I come up with anything I'll post the results here.

One question to clarify- when pulling data out of the Bytestream, it'll automatically span Datums, right? I don't have to worry about that on the "extraction" end?

(LabVIEW's running a long update and I can't play with it yet)

[PinguX]

Hello,

I randomly found those two GDevCon presentations :

A sublime dive into Arrays and Strings - Wiebe Walstra - GDevCon#5

GDevCon N.A. - 5x7 Subarrays and Substrings by Greg Richardson

I thought this might interest you. 😀

I've done some exercises with subarrays to explore their possibilities.

Although I'm not yet able to explain everything (and I prefer not to try, for fear of making mistakes), I've created a small VI to test different configurations: array, subarray, and bytestream. If it might be useful to you, it's attached.

(LV 2020)