Distributed Control & Automation Framework (DCAF)

cancel
Showing results for 
Search instead for 
Did you mean: 

DCAF Event Logging with Syslog

One of the most common needs when debugging embedded systems is a log file that contains a sequential series of timestamped events that occurred on the system.  Attempting to isolate an intermittent issue in a deployed system in  is nearly impossible without a set of reliable log files to tell you what happened.  This is especially important when attempting to debug a system that has multiple parallel processes - having a single timestamped log fed by a first in first out buffer ensures a clean view of the order of events leading to a particular behavior.  

 

Since recording system events is a very common need, a standard format for the messages is useful to ensure interoperability.  Syslog is a standard logging solution on Unix-like systems, and is very widely used.

 

Syslog-ng is a background process that is installed by default on all National Instruments Linux RT systems.  It provides a very flexible architecture that supports acquiring messages from many different source types, content-based filtering of messages, and distribution to many destination types.  It can acquire messages from files, local applications, or over the network.  It can send messages to local files, over the network using the syslog protocol, to a remote collector over the network using a variety of protocols (including TLS encryption), or log directly into a database.  Excellent documentation for syslog-ng may be found on the publisher's syslog-ng administration guide.

 

Since most CompactRIO and CompactDAQ developers are not aware of the built-in capabilities of syslog on NI targets, the Distributed Control and Automation Framework includes syslog features built into the framework.  By default, the framework will log the following events to the system log:

  • Initialization and shutdown of all software modules
  • Any error reported to the engine
  • Any module trace messages added via the API

The framework also includes a simple API function for custom modules to log to syslog.  It can be found on the Module Interface>>Module Runtime palette

 

syslog api.png

 

Messages in the log will have the format DCAF::Module::<module name>:<message>, will have the local timestamp of the time the message was generated, and can also have configurable priority level associated with the message.  The log can be easily filtered based on the module name, the priority, or other criteria.

 

Logs are stored by default at /var/log/messages.  You can modify the configuration of syslog-ng (stored at /etc/syslog-ng/syslog-ng.conf) to filter messages based on module name, priority, or other criteria, as well as set custom destinations for messages in addition to the default.  For example, you could set syslog-ng to log all messages to disk, log any messages from your custom module to a different file, and also broadcast any messages of Warning, Error, or Critical to a remote collector system and to a local database.  See the syslog-ng documentation for more details.

 

National Instruments Linux RT systems also include a web interface that can be accessed by a web browser without needing to install LabVIEW.  You will need a browser that allows you to run Microsoft Silverlight.  You can view the statically defined system logs via the built-in web interface on Linux RT targets by navigating to the IP address of the system, then clicking the System Log Viewer button on the left.  You will need to be logged in as a user with permission to view the system logs before the web interface will display them to you.  If you are not logged in, click the Login button on the upper right.  The system log viewer will display the last several messages written to each of the logs that are statically defined in the syslog-ng.conf file.

 

wif.png

 

DCAF messages by default are stored in the messages log.  Any error that occurs in a DCAF module will be reported here.  In the example below, a DCAF configuration with the LED, TDMS, and Scan Engine modules was deployed.  One of the C-series hardware modules in the chassis was removed while the system was running, which generated an error.  The system then shut down.  This sequence of events is shown in the log snippet below.  All of this log information is provided by default with DCAF, no special configuration necessary, and no custom error logging logic required.  To see the error description like in the example below for a deployed RT executable, you must enable the 'Copy error code files' option on the Advanced page of the RTEXE build specification in LabVIEW.

 

error.png

 

If your module calls the API function to write messages to the log on a repeated basis, make sure to configure log rotation on the system such that your message logs do not grow without bound.  You can configure log rotation for new log files by adding a new file to the /etc/logrotate.d/ directory.  To modify the log rotation settings for the default messages log, edit /etc/logrotate.d/messages.conf.  You can find help on the configuration options on the logrotate manual page.

Cheers,

Matt Pollock
National Instruments
Message 1 of 6
(7,677 Views)

Hi Matt Pollock

i have cRIO-9075 and system log viewer doesn't appear, i check permissions

 

0 Kudos
Message 2 of 6
(5,987 Views)

Hi

I have a new cRIO-9056 and it doesn't appear to have liblv_syslog_x86_64.* as called by the call Library function at /usr/local/lib/

So I get Error 7 occurred at Call Library Function Node in write syslog.vi

LabVIEW: (Hex 0x7) File not found.

 

So I have done a modification - see attached. (called from C:\Program Files (x86)\National Instruments\LabVIEW 2018\vi.lib\NI\Tag Bus Module Framework\runtime\write module trace msg.vi )

 

 

0 Kudos
Message 3 of 6
(5,239 Views)

Please note: this library is not present on cRIO units by default.  You'll need to use the DCAF deployment utility at least once on that target to push the library over.  Have you already used the deployment utility and still seen this error?

Cheers,

Matt Pollock
National Instruments
Message 4 of 6
(5,232 Views)

From the DCAF Editor, go to Tools >> Deploy Tool ...

 

From there, push to your cRIO the .so files along with your .pcfg. This process of pushing the .so files has to be done only once, unless you reformat the cRIO or get a new one.

Message 5 of 6
(5,225 Views)

Thanks guys - I didn't realize that is what the deploy tool did. I have just been using the web interface to copy the pcfg file across. 

 

Error definitions will be useful too. 

deploy.png

 

0 Kudos
Message 6 of 6
(5,219 Views)