XNET Frame Sniffer question - best way?

[CoastalMaineBird]

LV 2013.  LVRT 2013.  XNET 13

I need to monitor an XNET port (or legacy NI-CAN port) to find all unique identifiers within a given time period (say... 1100 mSec), while not disturbing any actual signal-level DAQ going on the same port.  The purpose is to detect what signals are actually being published.  The lowest update rate I need would be 1000 mSec.

This code seems to do exactly that:

But an easier way would seem to be to just ask the READ CAN FRAMES vi to read 999999 frames, with a timeout of 1100 mSec and then search the list for duplicates.

But doesn't that require the driver to hold 1100 mSec worth of messages in a buffer somewhere, until I get around to reading them?  I have no idea how many messages might come in.

I'm on an RT box, and while I'm not a slave to determinism, I don't want a series of memory allocations to clog up the works.

But I don't want a huge memory block reserved for an operation that might get used only once a day.

I also have a hundred other things going on, so I don't want to eat up the CPU.

What's the best way to balance those concerns?

[nathand · Accepted solution]

---

CoastalMaineBird wrote:

But doesn't that require the driver to hold 1100 mSec worth of messages in a buffer somewhere, until I get around to reading them?  I have no idea how many messages might come in.

---

You actually do have some idea how many messages might come in. Your code shows a baud rate of 125kbits/s. A standard (11-bit-identifier) CAN frame is at least 44+8N bits, where N is the number of bytes of data (no more than 8), and a few more bits might be needed if the data requires bit stuffing. Also, there's 3 bits of interframe space between packets. So, if your CAN network is absolutely flooded with the smallest possible packets, you'll have 125000/(44+3) messages per second = 2660 messages. The NI-CAN representation of a CAN frame, including timestamp and padding that aren't part of the underlying frame, is 22 bytes. 22 bytes/message x 2660 messages = 58,520 bytes, or about 58K. 58kbytes of memory is not a large allocation, even on an RT system with limited RAM.

So, you won't need a large buffer to read all the frames that could arrive in 1100ms. Set the number of frames to read to just more than you expect will actually arrive (or use the above worst-case estimate) and you'll be fine.

[CoastalMaineBird]

OK, that's where I was heading - figuring out the message size and dividing that into the baud rate.  I didn't know any of the overhead details (44 bits) so that's why I said "I have no idea".

This definitely seems like the simplest, as far as CPU goes.

I wonder how the allocation goes, though.  Does the driver figure out BaudRate / (9999 * (44+64) * 1.1 sec) * 22 and then allocate that much at once?  Or does it make a guess (1000 bytes) and allocate that, and then allocate AGAIN if it gets full?

Or maybe I just do it and see if anything complains, much as I don't like that approach.

Thanks, nathand!

[nathand]

---

CoastalMaineBird wrote:

I wonder how the allocation goes, though.  Does the driver figure out BaudRate / (9999 * (44+64) * 1.1 sec) * 22 and then allocate that much at once?  Or does it make a guess (1000 bytes) and allocate that, and then allocate AGAIN if it gets full?

---

None of the above. The documentation http://zone.ni.com/reference/en-XX/help/372841M-01/nixnet/propertysessionqueuesize/ indicates that the XNET driver allocates a buffer when you open the session, and XNET Read retrieves frames directly out of that buffer. If you don't read fast enough and the buffer fills, you probably get an error. It appears the default buffer size in your configuration is 1000 frames so you probably want to make it larger - see the documentation. That buffer allows the CAN device to do DMA transfers of frames from the device to the host with minimal involvement from the host CPU.

[CoastalMaineBird]

OK, thanks.  I'm going to go with a variation of the code i posted, which is a compromise between simplicity + lots of mem, and CPU usage.  If I read 50 mSec worth, there are only 21 iterations required, and I don't have to do any allocations.

Thanks!

[nathand]

---

@CoastalMaineBird wrote:

OK, thanks.  I'm going to go with a variation of the code i posted, which is a compromise between simplicity + lots of mem, and CPU usage.  If I read 50 mSec worth, there are only 21 iterations required, and I don't have to do any allocations.

---

I'm curious - what are you considering "lots of mem"? A buffer to hold all the frames that could possible arrive in 1100msec (just over 1s) is well under 100K. Your approach of rerunning Search 1D Array and Build Array in a loop isn't very CPU-efficient. I think you'd be better off reading the entire block of frames at once, then running Sort 1D Array, then looping through the array once to count (ignoring duplicates).

If you are using only a small range of possible IDs (for example if you're only using 11-bit identifiers), then you might consider a solution using an array large enough to have every possible ID as an index.

[CoastalMaineBird]

Well, I'm considering anything over 5000 or so to be "lots" for this purpose.

The SORT idea doesn't seem to me to improve anything, perhaps I don't get it.

And searching through 21 buffers of 100 is faster than searching through one buffer of 2100, I think.

Using a preallocated (2^11) array of IDs might be a tad faster, but I don't know if they will be using extended or not, and that's a big chunk of memory for a small benefit, I think. 

[nathand]

---

CoastalMaineBird wrote:

And searching through 21 buffers of 100 is faster than searching through one buffer of 2100, I think.

---

Why would it be any faster? The actual comparisons are identical. It doesn't take any longer to retrieve element 20 than element 2000. There might be some speed advantage to doing it once if there's any once-per-execution overhead, or if by doing it in the same place you keep more of the array cached. The bigger issue, though, is using build array in a loop. Each call to build array is an additional allocation, which usually means copying the entire array to a new, larger location in memory (copying is the expensive part of a buffer allocation - the actual allocation is fast).

---

@CoastalMaineBird wrote:

Well, I'm considering anything over 5000 or so to be "lots" for this purpose.

The SORT idea doesn't seem to me to improve anything, perhaps I don't get it.

---

Sorting is reasonably efficient and can be done in-place (which I think the LabVIEW primitive does). It's easy to remove duplicates from a sorted array, and you can pre-allocate the array to the full size (assume no duplicates), then shrink it afterwards to the size without duplicates.

See what your memory usage is on your RT target. Obviously I don't know what the rest of your code is doing, but if you're not moving huge amounts of data around, I'd be surprised if you didn't have well over 100K free memory available to buffer the CAN frames.

[CoastalMaineBird]

You're right - my bad - I'm not searching through the array of 2100, I'm searching through the array of unique-IDs-so-far, which is a lot less.

[CoastalMaineBird]

Sorting is reasonably efficient and can be done in-place

True, but I can't sort on the FRAME I get from XNET READ (the first element in the cluster is TIMESTAMP).  So I have to unbundle ALL the frames into an array of IDs, sort that, and then remove duplicates.

That removes the simplicity advantage over the chunk-at-a-time approach, it seems to me.