Creating Large USB Instrument Networks with Persistent VISA Namespace

[FCGoat]

Intro:

Our team transistioned to the linux based cRIOs (two 9068s) roughly 6 months ago with dreams of everything being significantly easier, faster, better, and full of hope, wonder and joy.  Turns out we were a little optimistic, and it's been a pretty bumpy path towards making a fully operational transition from our previous cRIO units.  However, that said, it's opened up some powerful opportunities for some really cool new things, this document outlines one such item with all the steps needed to complete it with some miscellaneous nuggets along the way to help you tweak it to your needs. 

Background:

We work in the fuel cell industry doing system development on large complicated systems with dozens of different instruments with an assortment of various protocols.  This has led to some interesting choices with repsect to just how much instrumentation we use, and how we get all of the signals into a computer reliably for analysis later.  In particular we use lots of serial instruments; mass flow meters, pressure regulators, homegrown control and daq boards, motor drivers, arduinos, etc, which leads to a problem with respect to just how you can get all of these signals into a computer.  The 9068 comes with two RS-232 ports and one RS-485 port, however as a lab environment with lots of users we've seen these ports get damaged with collateral damage to the main control board so we like to include some isolation.  You can expand the number of these ports with the 9870 (which just got 9068 driver support with NI-Serial 4.1) but at >$500 it starts looking expensive to connect very many serial devices.  With our intention of hooking up 15-20 devices we started looking for alternative solutions which led us to start thinking about inexpensive and robust methods to increase our instrumentation.  It was the combination of realizing that we had a full blown RT linux machine in our RIO, and that it had a USB port, and it supported USB hubs, and that we used COTS USB-Serial adapters all the time for debug that crystallized into the following idea: we could use the built in linux kernal USB drivers to expand our number of connected instruments (mostly) indefinitely. Then we got to work. 

Overview of the Process:

In order to achieve this extendable network of instruments we'll need to collate a bunch of information, apply a bit of creativity and learn to read forums/linux technical guides.  Our basic process will be this:

1: Ensure we have control over our cRIO

2: Get Linux to recognize our device(s) like we want

3: Get Labview to recognize our device(s) like we want

Step 1: Ensure we have control over our cRIO

All of this process relies on the cRIO doing what we want, if you don't have at least firmware version 2.1.0f0, and likely NI-RIO 14, these tricks aren't going to work.  So, first things first, run NI-Update service, and update everything you can, this will save a multitude of headaches later on. 

Next, you'll want to spend a little time coming to terms with SSH'ing into your device and the command line interface.  First open MAX, select your cRIO, and in the System Settings check the "Enable Secure Shell Server (sshd)" option, this lets the cRIO accept incoming connections, save, and restart.  You'll need to install a good "terminal emulator" program, PuTTY or Tera Term are good options on windows, and then check out the start up guide found here to get connected. Once you've connected to the device, if you've never used a linux machine before you should probably take a bit of time to come to grips with command line basics, this is a pretty good tutorial. 

Up until now there has been pretty good documentation on how to get started, from now on we're moving into more foggy territory

2: Get Linux to recognize our device(s) like we want

One of the downsides to using USB devices is that they are dynamically assigned a connection ID, and this means you can't always count on a device to end up on the same COM port.  This is further complicated by power cycling a device, usually things come up in the same order, but not always.  Obviously this isn't ideal for use of instruments programatically in Labview as we'd like to know their address with assurance that they haven't moved.  However, as expected, this isn't an unknown problem (1, 2, 3), and luckily there are a number of ways around it.

With cRIO firmware 2.x came a beautiful, linuxy thing: udev.  Udev is the device manager for the linux kernal and it keeps track of all the miscellaneous devices you hook up to a linux machine.  It has a very nice built in protocol for applying rules to the process of adding hardware to a system which we'll be leveraging to manage our USB devices. 

First things first, we'll need to be able to uniquely identify each instrument.  To do this we'll use a linux utility called "usbutils," to get it, here are the steps (commands are in bold, notes are in italics):

opkg update                    get the current list of available packages to install

opkg install usbutils        install usbutils

Now, we'll want to use the "udevadm" program to learn about the device(s) in question.  To do this, you'll want to use to the following commands:

cd /dev/                            change directory into the device directory

ls | grep USB                    list all the devices, pipe the response to the search program "grep" and output only the items with USB

This gives us all of the miscellaneous USB devices attached to our system, now we can query udev about each one:

udevadm info -a -n ttyUSBx               This queries udev about everything it knows about this device results should look like this (truncated):

    looking at device '/devices/amba.0/e0002000.usb/xusbps-ehci.0/usb1/1-1/1-1.1/1-1.1:1.0/ttyUSB0/tty/ttyUSB0':

    KERNEL=="ttyUSB0"

    SUBSYSTEM=="tty"

    DRIVER==""

  looking at parent device '/devices/amba.0/e0002000.usb/xusbps-ehci.0/usb1/1-1/1-1.1/1-1.1:1.0/ttyUSB0':

    KERNELS=="ttyUSB0"

    SUBSYSTEMS=="usb-serial"

    DRIVERS=="ftdi_sio"

    ATTRS{latency_timer}=="1"

    ATTRS{port_number}=="0"

  looking at parent device '/devices/amba.0/e0002000.usb/xusbps-ehci.0/usb1/1-1/1-1.1/1-1.1:1.0':

    KERNELS=="1-1.1:1.0"

    SUBSYSTEMS=="usb"

    DRIVERS=="ftdi_sio"

    ATTRS{bInterfaceNumber}=="00"

    ATTRS{bAlternateSetting}==" 0"

    ATTRS{bNumEndpoints}=="02"

    ATTRS{bInterfaceClass}=="ff"

    ATTRS{bInterfaceSubClass}=="ff"

    ATTRS{bInterfaceProtocol}=="ff"

    ATTRS{supports_autosuspend}=="1"

    ATTRS{interface}=="FT232R USB UART"

  looking at parent device '/devices/amba.0/e0002000.usb/xusbps-ehci.0/usb1/1-1/1-1.1':

    KERNELS=="1-1.1"

    SUBSYSTEMS=="usb"

    DRIVERS=="usb"

    ATTRS{configuration}==""

    ATTRS{bNumInterfaces}==" 1"

    ATTRS{bConfigurationValue}=="1"

    ATTRS{bmAttributes}=="a0"

    ATTRS{bMaxPower}==" 90mA"

    ATTRS{urbnum}=="83"

    ATTRS{idVendor}=="0403"

    ATTRS{idProduct}=="6001"

    ATTRS{bcdDevice}=="0600"

    ATTRS{bDeviceClass}=="00"

    ATTRS{bDeviceSubClass}=="00"

    ATTRS{bDeviceProtocol}=="00"

    ATTRS{bNumConfigurations}=="1"

    ATTRS{bMaxPacketSize0}=="8"

    ATTRS{speed}=="12"

    ATTRS{busnum}=="1"

    ATTRS{devnum}=="9"

    ATTRS{devpath}=="1.1"

    ATTRS{version}==" 2.00"

    ATTRS{maxchild}=="0"

    ATTRS{quirks}=="0x0"

    ATTRS{avoid_reset_quirk}=="0"

    ATTRS{authorized}=="1"

    ATTRS{manufacturer}=="FTDI"

    ATTRS{product}=="FT232R USB UART"

    ATTRS{serial}=="FC1211"

This particular device is a USB to serial converter, and normally we'd look for something unique in this configuration data compared to our other devices, but in our case we went one step further.  Some devices (like this FTDI chip) can be programed to have unique identifiers.  We used the FTDI chip programmer found here, to assign a distinct serial name to each of our devices which corresponded to their actual name in the system (FC1211 in this case).  This is a fantastic way to absolutely, positively identify your device.  If you end up doing this, make sure you don't change the vendor ID or product ID, as they'll likely break your drivers (the system won't recognize which drivers go with it).  If you are unable to do this step, you can just look for a distinct entry, product, vendor, kernal number etc.

Next we'll be making a udev rule.  There are a huge number of udev tutorials out there, but in this case we need something very simple.  Here is the basic work flow: write a rule, place it in a readable file, place that file where udev looks for rules.  Read through a tutorial for a better overview but here is an example simple rule that we use:

ATTRS{manufacturer}=="FTDI", ATTRS{serial}=="FC1211", SYMLINK+="FC1211", MODE="0666"

This rule looks for devices that match the two keys of manufacturer (ATTRS{manufacturer}=="FTDI") and serial (ATTRS{serial}=="FC1211") and adds a symlink (SYMLINK+="FC1211") and sets the permission of this device (MODE="0666") so that any user can use it (note: linux treats ports like files so the permissions work like they do for files).  This allows us to first positively identify a device, give it a known address, and set it's permissions so it is accesible by VISA in one step. We can add as many rules as we'd like into one file, and even rules that allow different hardware to create the same symlink.  I've attached my full rule file for perusal, make sure you keep yours in source control so you can easily keep track of changes to the hardware.

This rule is saved in a file with the following format "number <50"-"Label".rules  for example, ours is "49-USBFix.rules"

Next we place this file on the cRIO here: "/etc/udev/rules.d/"  You can do this most easily with the file transfer function in MAX.  Once it's there, we'll want to set the permissions (chmod 0644 49-USBFix.rules) and trigger udevadm so that it knows it has some new rules (udevadm trigger).  Once the rule is loaded udev automatically runs it anytime a new device is loaded or on system restarts.  We can check to see if our rule worked by looking at the contents of "/dev/."  Look for your new symlink, it should be at the top of the list, the command is "ls". 

That's it, now the linux side of the cRIO knows all about our new device and has a standard place for us to find it whenever we need it.  Awesome, onto Labview.

3: Get Labview to recognize our device(s) like we want

Labview is similarly straight forward to get working (if a bit tedious).  What we want is to link this new static symlink on the linux side to VISA so we can always know what the VISA address is when we plug in our device.  To do this, we'll add some static links to our visaconfig.ini file.  On the cRIO, the visa config file is located here: "/usr/local/vxipnp/linux/NIvisa/."  Copy it over to your favorite text editor and get editing. 

The first thing you'll want to do is move the line "NumOfResources=x" directly under the heading "[ASRL-RSRC-ALIAS]," this tells VISA how many static references to expect, and if the number is set lower than the number that you've set up, you won't get all of them.  This is a very frustrating debug since this line sometimes gets buried deep into the file. 

Next, find the last chassis port, in my case (the 9068) this is ASRL3::INSTR, the RS-485 port.  We'll copy this entire section and make our modifications, it should start out like this:

Name2="ASRL3::INSTR"

Enabled2=1

Static2=0

SystemName2="/dev/ttyS2"

BaudRate2=9600

DataBits2=8

Parity2=0

StopBits2=10

FlowCtrl2=0

We'll add a new entry for each of our usb devices, change the ASRL number to something higher than will normally be expected, iterate the number on each field, set the static field to 1, change the connection info, and point the system name to our new symlink.  Here is an example:

Name3="ASRL30::INSTR"

Enabled3=1

Static3=1

SystemName3="/dev/FC1211"

BaudRate3=115200

Parity3=0

StopBits3=10

DataBits3=8

FlowCtrl3=0

This provides a static reference  (Static3=1) in VISA that points to our symlink with the appropriate name (SystemName3="/dev/FC1211"), baud rate (BaudRate3=115200), and VISA address (Name3="ASRL30::INSTR").  Again, we can add as many of these as we'd like (I've attached my current visaconfig.ini for reference), and they create static points (that may not even be present in the system) which we can use as our addresses.  Make sure when you finish adding all of your devices you update the "NumOfResources" field, otherwise this work will be for nothing.

We're not quite done with this ini file yet, rather than trying to remember esoteric ASRL numbers, we can make semantically useful aliases.  A little higher in the file you'll find the ALIASES heading, underneath this you'll add your aliases, they'll have the following format:

Alias0="'FC1211','ASRL30::INSTR'"

Alias1="'FC1212','ASRL31::INSTR'"

NumAliases=2

Again, this is fairly straightforward, iterate on the field name, place your desired reference name and link it to the appropriate ASRL number. At the end, note how many aliases you have.  Like before with the rules file, you'll want to place this file under source control so that you can keep track of your hardware changes.

Back in the cRIO, change the name of the old "visaconfig.ini" to "visaconfig.ini.old" and upload the new and improved "visaconfig.ini" and refresh the device list in MAX.  If all went to plan you should be looking at a device list like the one in the uploaded picture.  You'll see some devices are listed as not present (little red x's) this is because I have two cRIO's and use the same rules file and visaconfig for both.  As each instrument is only on one cRIO it shows up as present or not present depending on which device you're looking at.  This is also a nice feature to figure out if all of your devices are up and running where they should be.

Conclusions

Hopefully this is helpful to some people, it was a bit of a long and tedious trial and error process to get this all up and running, but I think it really demonstrate what is capable when you integrate the tremendous linux ecosystem with NI hardware and software.  Hopefully this is a sign of great things to come on the linux/labview front. 

I'm sure there are things I missed, steps that are incomplete, or flat out errors, please let me know and I can make the corrections as necessary and I'd love to hear any comments you may have. 

Acknowledgements are hard in this case... because we've been working hard to make a deadline here and I'm not sure what all I read and where and who it was from etc.  So... if you think you helped me write this.. then thanks to you.  🙂 

Cheers  ~Micah

[gil.smith]

I want to similarly force USB-to-serial port names on Raspberry Pi running Raspian.

As I am new to Unix (let alone RPi) I 'think' I have installed Visa, but I do not have a directory on the RPi like "/usr/local/vxipnp/linux/NIvisa/" to edit visaconfig.ini file.

I guess I need to check Visa is installed correctly, then find the correct directory and file.

Can anyone point me in the right direction?