FlexLogger & SystemLink File Ingestion Service

[jyoung8711]

I'm working with FlexLogger 2021 / SystemLink 2021.

I have SystemLink running on Computer A

FlexLogger running on Computer B

Computer B is set up as a SystemLink Client for A

According to this announcement:

https://forums.ni.com/t5/FlexLogger/Announcing-FlexLogger-2020-R3/m-p/4069378

and the option in the "Project Settings" I *thought* that FlexLogger could automatically submit files to the SystemLink File Ingestion Service.

When I check the box "Publish channels and files remotely" the channels do indeed show up as tags in SystemLink (great!)

However the files are not being ingested in SystemLink.

Are there additional settings I have to toggle here? or am I missing something else? or was this feature removed?

Thanks in advance

[Brad]

Once the FlexLogger system correctly has the SystemLink Client connected to the Server:

I would expect the files to be copied to the server as well. You may want to try checking all the options in FlexLogger related to publishing tags (i.e. publish them locally as well). I think this may have been something got fixed in a later version. You also need to make sure you press the "Run" button in FlexLogger because this is what starts logging data to a file. Once you stop logging data, the file will be created and then copied to SystemLink if you have the option selected. The channel values and tags values update without having to log data, but the file isn't created until you run and stop the log.

If this still doesn't help, you can use an ETW Trace tool to record internal messages that might be helpful to figure out what is wrong. I attached an ETW Viewer and you can run the ETW Viewer.exe when launching FlexLogger and let it run while you create a log file in FlexLogger. Go to the All Events tab in ETW Viewer and press the update button in the upper right. Then press the Save button in the lower left and attach that log file in this discussion thread so we can see if this has any helpful info about why the file isn't saved to SystemLink.

Thanks,

Brad
You can also run an ETW Trace tool when launching FlexLogger and trying to log data and get files copied to

[jyoung8711]

Brad --

Thanks again for the quick responses.

I've confirmed everything in your email:

1. SystemLink client is connected

2. Settings are enabled for project

3. FlexLogger is posting tags to the correct server -- though I should note that this only worked on one of my "projects" for some reason... the other, which was calling a custom plugin, did not post to SystemLink.... which I thought was weird... The test that worked was using a simulated device created in MAX

4. Started Logging

5. Waited ~20 sec.

6. Stopped Logging.

No file.

I did try uploading a file from the browser, and that DID work.  it showed up in the browser, and on disk in the expected location, but no dice with the flexLogger file.

I ran the tool you provided, and attached is the log generated during steps 4-6.

Nothing jumped out at me, but the log is pretty cryptic... I zipped it up so that I could upload it here.

[Brad]

Thanks for posting the trace. I didn't see the info I was expecting. Could you please open ETW Viewer and in the Settings tab, and in the Providers column, make sure you enable NI-Lumberjack, NI-LumberjackPefEngine, and NI-PEF (you can disable everything else in the providers column to reduce the amount of noise). Then launch FlexLogger and do your test again. also what is the exact version of FlexLogger you are you using...this will allow me to send you debug versions of some of the code if needed.

Thanks and sorry for the confusion,

Brad

[jyoung8711]

See the attached log.

FlexLogger 2021Rs

Build 9.0.0.50430

DevKit v1.3

[Brad]

Thanks for sending the trace. It looks like you started ETW after launching FlexLogger. There is an important message I am looking for when FlexLogger is launching to ensure it is using the right systemLink settings. Would mind getting another trace and starting ETW before launching FlexLogger. Sorry for the confusion and thanks for your pateince!

Brad

[jyoung8711]

Ah.  Sure, no worries

Here I started ETW, started FlexLogger, Loaded Project, Ran Test, Shutdown SystemLink, and then captured ETC log.

I see that it did catch a systemLink related error (-251251) when "Setting secondary skyline config from command line".

Also -- Another (related questions)... assuming we are able to get this to work, is it possible to capture the DataFiles WITHOUT posting tags? or will the two always be linked?

As I mentioned in my other post, I wanted data retention with the tags, so I ended up building in that to my SDK Plugin I was working on... However, I'm logging 1000+ tags... so I don't really want them to show up twice... however I *WOULD* still like to get the data file automation.

Thanks!

[Brad]

Thanks for the updated ETW trace. As you mentioned it appears error -251251 occurs when initializing the Skyline connection. It appears this error is related to the config file:

kErrorCodeConfigInvalid = -251251

If you browse to "C:\ProgramData\National Instruments\Skyline\SkylineConfigurations" are there any crash/error logs or information in the skyline_master.json that might be helpful?

We currently don't support enabling file copying to SystemLink server without also updating the tags, but this is definitely something we can consider, so I will add this to the backlog. It's also worth mentioning that newer versions of FlexLogger enable the data retention by default when logging data to remote SystemLink servers, I think it's in FlexLogger 2021 R1 or R2.

Hope this helps and let me know if you find any helpful info in the SkylineConfigurations folder. I attached a simpe LabVIEW VI so you can work with SystemLink support if you continue to have problems...I am surprised the tags are updating since we use the same session for updating tags and copying TDMS files to the remote SystemLink server.

I also included a sample of what my skyline_master.json looks like (in a zip since the web page didn't allow me to attach files with json extension) so you can see if there's any important differences with yours.

[jyoung8711]

Thanks for the additional info, and pointing me in a couple of directions here...

1. No errors in the skyline configuration folder

2. Nothing obviously wrong with the skyline_master.json file

3. When I run the LV VI (out of box) I got the same error number : -251251, but the LV error gave me some additional context:

"Unable to open configuration. 'secure only' was specified but the server is configured for non-local access over 'http'."

When I change the boolean in the "HTTPS only" to "FALSE" I can run w/out error.

This makes sense since the SystemLink server is NOT using https... only http on this current example, so I'm going to guess that FlexLogger is trying to use HTTPS and is erroring out.

4.  I changed the SystemLink server to use HTTPS -- and everything magically works! however --

5.  I can't validate this for sure... but it seems that the tags updating 2x as frequently... The server tag "count" value went from update 3x every web page refresh, to 6x every web page refresh... Data is only coming in @ 1 Hz though, so I'm not sure what is going on here...

6.  W/out retention turned on, I can't tell for sure on (5)... but the "count" is definitely going up faster than data is coming in.  I tried to turn on "retention" manually to test this, but that didn't seem to work? So with/out a packet sniffer like fiddler installed, I'm not sure how to verify this 100%

So in summary, progress (I think):

- It seems like FlexLogger (in it's current configuration) requires that SystemLink be running in "HTTPS" mode for the connection to work

- My tags for this project seem to be updating 2x per second... (potentially once through http & once through https??)

--I've tried disabling re-enabling 

Two other notes:

1. Ok, i'll look out for the next flexlogger update for the date retention being on.

2. Yes, I would advocate very strongly for being able to turn on/off tags & file ingestion independently... I can understand the value of keeping things "simple" given the nature of flexLogger, but as a partner who's looking at integrating FlexLogger into our Solution Platform (w/SystemLink) for customers, having the ability to tweak FlexLogger configuration is critical for such efforts.

[Brad]

Glad to hear changing to HTTPS allowed you to get the files copied to SystemLink. I setup a similar system and ran into the same experience...I wasn't able to get tags to show up or files to get copied to SystemLink until I enabled HTTPS, so this may be a new requirement, so I will make sure we document this.

I tried with 200 channels and noticed I got about 10 updates/minute. When I changed to 16 channels, I got about 60 updates/second. I was also getting the min/max/mean aggregates enabled by default. Just to be clear, by default the Aggregate Values is enabled for remote SystemLink Server, but the Retention will be set to None:

Are you wanting retention set to something else? In the short term, would adding INI tokens to enable/disable SystemLink Tag updates vs copying files or setting Retention to something else address your needs?

Thanks for your patience and feedback,

Brad