Example Code

Read and Write BLF Files

Products and Environment

This section reflects the products and operating system used to create the example.

To download NI software, including the products shown below, visit ni.com/downloads.

    Programming Language

  • LabVIEW G

Code and Documents

Attachment

Description

Overview

Read and write BLF binary files using the Vector provided DLL, or a more Raw API using the zLib Inflate/Deflate. 

Description

Vector has a custom binary file format used primarily for logging raw CAN data.  When using a Vector product such as CANalyzer, a log can be done in ASCII which baloons the file size but has the most compatibility, or you can log in BLF which is a small binary file, which also has some level of object compression built in.  With this API you can now read these BLF files and get the raw CAN frames in a form that is similar to the XNet frame structure, allowing for easy conversion using the XNet API.  You can also use this API to perform the reverse function.  Where you may want to log into a BLF file, so that Vector tools like CANalyzer can analyze and view your data.

 

All the core functions are implemented by calling the Vector provided DLL, which is documented online and has a shipped example installed with CANalyzer.  There is an alternate API that also allows for reading and writing BLF files by not using the Vector DLL and instead uses the OpenG zip utility.

Hardware and Software Requirements

Back saved into LabVIEW 2014 (version 2) or LabVIEW 2017 for version 3, and requires the Windows platform for executing the binlog.dll.  Version 8 is saved in 2018 and has an example of both the DLL version and the Raw version which requires the OpenG zip package.  The OpenG zip method works on all platforms that API supports, and is much faster than the DLL method.

Steps to Implement or Execute Code

The external dependencies are referenced with the Package Dependency.vipc file.  Once those packages are installed through VIPM, open either of the examples and run them to write and read a file.

 

Untitled.png

Additional Information or References

Version History

 

Version 8

  • Raw Open has Overwrite input
  • Raw added Polymorphic Write CAN Data, that can write a single frame, multiple frames to the same channel, or multiple frames to different channels.
  • Raw bug fix with ID mask of standard CAN frames when reading.

Version 7

  • Bug fix with uncommon object types
  • Added IDs To Return filtering on the Raw Read function

Version 6

  • Made reading more robust supporting other objects like System Variables in the BLF.

Version 5

  • Added a secondary API for reading and writing using the OpenG zip utilities.
  • Added reading and writing CAN FD frames

Version 4

  • When writing to a BLF the day light savings flag is used and will offset the write by an hour if necessary
  • Object Offset was removed and instead before every read the Object Offset is read from the session.  This is used instead of tracking the current offset since it had issues with newer CANalyzer files

Version 3

  • Bug fix with the Read CAN Frames.vi which now accepts reading CAN_MESSAGE2 which newer Vector tools may write to
  • Fixed Read CAN Frame.vi to return an ID masking out the 29 bits for extended, not 27

Version 2

  • Back saved to 2014
  • Added CAN Channel input and output to the Read and Write functions

 

 

 

Example code from the Example Code Exchange in the NI Community is licensed with the MIT license.

Comments
Dhakkan
Member
Member
on

Thanks for sharing!

IvanAlegria
Member
Member
on

Hi Hooovah! would it be possible a version for earlier labview versions? I am running 2014 SP1 and this VIs would be very useful.

Thanks

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

New version added that is saved in 2014.  In using this API I realize that the CAN Channel is an input and output that I found useful so I added this.  With this input you can write a CAN frame as from CAN 1 or CAN 2 and then in CANalyzer you can associate two different DBCs to each CAN Channel.  When reading you will also get an array of Channels, along with the array of frames.

IvanAlegria
Member
Member
on

Thank you so much!

IvanAlegria
Member
Member
on

It does not work for me, it seems like it doesn´t work in a 64 bit platform. Could that be the problem?

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

What doesn't work?  Give some more details on the result of running the included example program.  I developed it and tested it on Windows 7 x64 using LabVIEW 2015 SP1 32-bit.

IvanAlegria
Member
Member
on

It is not able to load the dll. My labview version is 2014 SP1 64 bit. I guess the problem is that I would need a 32 bit version of labview.

Thanks for the help.

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

It seems you are right.

 

http://digital.ni.com/public.nsf/allkb/6059E812DFC60ED486257640007B5DAF

 

The DLL provided by Vector is 32-bit only.  We can either beg Vector to release a 64 bit version (unlikely), you can look into WOW emulation to run 32 bit binaries in a 64 bit environment (no clue the difficulty), you can use 32 bit LabVIEW, or you can look into re-writing the API using pure G removing the DLL call.  Owning LabVIEW means you have access to both 32 and 64 bit environments and that would probably be the easiest.

 

When I wrote this API I did start down the path of reading and writing BLF files using pure G and not calling their DLL.  The file format is somewhat documented, and I figured I could reverse it enough to come up with something.  But I figured whatever I would come up with probably wouldn't work in all situations and would have a fair bit more bugs so I just resorted to calling their DLL.

alii001
Member
Member
on

hi,boy,It seems you lost an important parameter when wrapper the DLL, "Channel Index", as its result, my app can not disguinish the Channels when the messages were received ,the are all "Channel 1".  I suggest you release the access to set the channel when wrapper ithhhhh.png

labviewman
Member
Member
on

Hooovahh has done a fantastic job here (and on his CAN blog!), but I found a few issues:

1) Using the offset when loading from a blf file didn't work for me

2) CAN frames read from a blf file created by newer versions of CANalyzer were not recognized

3) Extended CAN frames read from a blf file returned the wrong ArbID (applied a 27-bit mask instead of a 29-bit mask if the object indicated it was an extended frame)

4) CAN channel for the frame was not indicated

 

See attached snapshots for the suggested changes (seems to work for me).

 

 c1.PNGc2.PNG

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

For some reason I'm not getting email notifications on this so sorry I didn't see the older message.

 

@alii001 - You are looking at the older version.  Version 2 adds CAN Channel to be written and read.  To avoid confusion I'll remove some older versions.

 

@labviewman - Similarly your issue 4 was fixed in Version 2 with reading and writing CAN Channels.  Issues 2 and 3 are totally valid, thanks for catching it I'll be updating it here and on my blog. 

 

Issue 1 I don't fully understand and I feel like it needs more investigating.  Maybe it is something to do with a change to the BLF format over the years, and a newly written one handles object offsets differently.  I won't be changing this in my release yet because I don't understand it and it seems to be working on the test files I create.  I hope you can understand that the code reads the current object position, then adds to that value the number of objects read and sets that to the new read position.  When reading a large file it is often useful to read in chunks like 1000 frames at a time and between reads to update a progress bar, or poll for a cancel.  If we need to read an object at an offset, we only have two ways to change the object position.  We can set the current object to 0 by closing and reopening, or we can skip the current object until we are at the offset we want to read.  So I have code to keep track of the current object position between reads, so that a read of the next 1000 frames, won't require setting offset to 0 and then skipping until we get to the new offset.  It is possible you have a problem with your code, or with the BLF, or it is some change I'm unfamiliar with in the BLF file.  The change you made I believe will have it always invoke the skip.  So if you read 1000 frames, then read the next 1000, it would reset to 0, and invoke the skip 1000 times.  I'm glad it works for you but it can be inefficient for large files to have to keep resetting.

labviewman
Member
Member
on

Hey Hooovahh!

 

Regarding #1 (Using the offset when loading from a blf file didn't work for me), I spent most of the day yesterday working the issue...I certainly could have made a mistake and will be the first to admit it Smiley Happy

 

I ran a test this morning with a blf file created with CANalyzer 11.047 SP1 (64 bit) where I collected 10,377 frames. When I used rev 2 of the blf vis w/o the offset mod I suggested, I was only able to read 1,746 frames. Using the suggested offset change, I read all 10,377 frames.  Also, if you insert "Get File Statistics.vi" after "Read CAN Frames_File.vi" that does not have the suggested offset mod, I read the same frames as the suggested offset mod (w/o "Get File Statistics").

 

I also modified the code to output the offset after each read, which is then plotted. Should be a linear line.

 

*New issue/question: I wrote the same frames I read with the offset fix then read them back with the offset fix...I get close to the same timestamps, but they are off by 1 hour and several 10s of microseconds. I didn't spend much time looking into this but changing the value passed into Header.Object Flags and modifying the divisor for resolution in "Write CAN Frame.vi" had no affect.

111.PNG222.PNG

 

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

There is something wrong for sure, but resetting the offset to 0 each time doesn't seem like the right solution, because as I mentioned before, reading 1000 frames at a time means, skipping an addition 1000 frames each time, instead of picking up where it left off at the last read.  I'd expect read times for the second method to be much longer than the other two.  It is also odd that by just adding the read position offset, the code works as expected.  I only have access to CANalyzer 8 at the moment. 

 

As for the hour off it is a problem with day light savings time.  6 months out of the year it would look fine.  The fix is in the Set Measurement Start Time.vi.  Unbundle the DST output and subtract the hour if it is a 1.  Update will be posted soon, but I think there could be a bug in reading a file written by another application.  The Read File Statistics.vi should be setting the DST flag for the start time and last object and I'm unsure what issues will come up with not setting it.

 

I've decided to just do away with the object position since it is having issues on some file types.  Instead it will just read its offset on a read each time, and determine what to do based on that.  The update will remove that data type from the class and all references to it.

labviewman
Member
Member
on

Are you referring to the offset getting reset to zero, close the file, re-open...the section of code below? If so, that doesn't happen...the number I'm passing-in is an offset based off of the start of the file, thus, the (ObjectReadPosition<=Offset) check is always true because the offset I'm passing in is always larger than the current object offset.

 

I can e-mail you a log file from CANalyzer 11 if that helps (I have your e-mail address).

 

I'll take a look at your suggestion for DST.

 

Regarding writing a blf file, I suggest changing the timestamp resolution to 1 nano-second resolution...maybe the 10 usec resolution was there for a reason?  See attached.

 

As always, thanks for your support in all things CAN Hooovahhh!

 

BTW, NI isn't notifying me about post updates either...

aaa.PNGas.PNG

 

 

JJSung
Member
Member
on

Thanks for sharing.

I couldn't find the version 5.

I'd appreciate it if you could send it by email.// (sjjp7738@naver.com)

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

Sorry it somehow didn't get attached I've updated the post.

ThomasMuller
Member
Member
on

Thanks for the update, I use the version 5, and everythink works for me.

It would be great to add a function to read only an ID.

I tried to modify code but it doesn't work.

I have some data and very big  file (800Mb), and it's too long to convert all signal from the blf file. I need to select only ID that I would like to analyse.

Could you help me to add this function ?

 

Thnaks in advance.

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

With help from @ThomasMuller I've uploaded version 7 which improves the raw BLF reading features to the point where I think the DLL method might not be needed soon.  It adds robustness and bug fixes to the reading of objects, but also adds an IDs to Return input which will only return CAN IDs that match those IDs.  Keep in mind that this filtering takes place after the Search Offset and Length.  This means that returning no Frames does not necessarily mean the end of the file has been reached.  Use the End Of File output for that.

Yoost
Member
Member
on

Does this also work with a LIN frame, or are further adjustments necessary?

A-T-R
Member
Member
on

Hi hooovahh,

nice work, and your blog is quite as good! Thanks for your enthusiasm on the topic!

 

I have a question though...

 

I would like to perform a CAN logging, which is commonly known as a "trace". It would be great if I could do this in the BLF format, because the files then could be loaded and replayed with VECTOR tools. You mentioned in the toolkit comments, that appending data to an existing file is not supported by the API.

 

What would you recommend to get the tracing task done? The continuous "open (including read)-write-close" of the file is not suitable for a long-term test situation. Even the logging of only three messages A, B, C (cylce time A = 10 ms, cycle time B = 200 ms, cycle time C = 200 ms) for only one minute results in ever increasing writing times up to 1500 ms in the end.

 

Is there some fast method to stream the frames to disk in BLF format?

 

Any idea is appreciated!

 

Achim

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

Look at the newest version I posted.  It has two APIs, one using the Vector provided DLL and the other using the raw method which I made that exposes the low level file structure.  This Raw type is much faster, and has other improvements like reading and writing with the same session. 

 

I'm not familiar with the Trace that you are talking about.  I've only ever used the BLF to record or playback the raw frames, which can then be converted into signals using a DBC.

labviewman
Member
Member
on

Kudos to Hooovahh!  The Read/Write .blf RAW vis are amazingly fast...about 11x the write performance of the .dll version (in a virtual machine)

A-T-R
Member
Member
on

Hi hooovahh,

I downloaded this version 8 and played a little bit with it.

 

It is really fast! I think one of the reasons is, that no actual writing to disk takes place until the given 65536 frames (specified in the "Open File.vi") have been collected. Hmm...I'm wondering what's the reason for this particular number of frames might be? For me, "holding back" of the frames seems quite unsafe regarding a (continuous) observation of a long lasting e.n.d.u.r.a.n.c.e. test (several days or even weeks), with "logging ON" and "logging OFF" phases.

(I wrote this with dots in between, because the forum software wouldn't allow this word without them...I assume some very clever algorithm which suspects some erotic reference 🙄)

 

I tested this with my experimental "setup", which means I pushed dummy frames to a Queue in one loop, and in the other loop I read them out and handed them over to your "Write CAN frames.vi". (Again: Three messages A, B, C (cylce time A = 10 ms, cycle time B = 200 ms, cycle time C = 200 ms) for one minute.

 

This resulted in ~6588 frames, which haven been written to file eventually, and this resulted in a file size of 2 kB.

I then changed the "frames to flush" parameter to just 100 frames, and started the same test...which resulted in a file size of  7 kB.

Can you explain what happens?

 

One other thing: The two APIs (DLL version and "Re-engineered") are using nearly the same VI icons. Could you change that, so that they can be easily distinquished from each other, e.g. by different icon BG colors or different "banner" BG color?

 

Will this version of your APIs be available via VIPM in the near future? There is a "Package dependencies.vipc" file included in the downloaded zip file, which cannot be opened with VIPM, and I just don't know what to do with it...

 

Achim

A-T-R
Member
Member
on

Hi hooovahh,

(I wrote some elaborate comment, but after editing/correcting something, the whole thing went somewhere...so I have to write again...grrrr)

 

I downloaded this version 8 and played a little bit around.

 

It is fast! I guess it is because no actual disk writing takes place until the specified number of frames (65536) have been collected. I'm wondering what the reason for this particular number might be? For me, "holding" back the frames doesn't seem appropriate for a long lasting test (days or even weeks), with "logging ON" and "logging OFF" phases.

 

I tested this with my dummy setup: Loop 1 pushes dummy messages A, B, C  (cylce time A = 10 ms, cycle time B = 200 ms, cycle time C = 200 ms) in a Queue for one minute, and Loop 2 reads them out and hand it over to your Writing-VI.

This results in ~6588 frames wich will be flushed to disk eventually, and this resulted in a file size of 2 kB

 

I then changed the "frames to flush" to just 100, and this resulted in a file size of 7 kB. Can you explain what happens?

 

One other thing: Your APIs (DLL and "re-engineered" or "self-engineered" raw version) use nearly the same VI icons. Could you change this, so that the VIs can be easily distinguishable? Maybe by using different BG colors or banner colors?

 

Will your version 8 be available for VIPM in the near future?

 

Regards

Achim

 

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

Every time we write some frames to disk, we do so by compressing those sets of frames into a block.  Sorta like making a single file in a zip.  Compression techniques work best when there is lots of cyclic patterns.  CAN is full of patterns since most frames won't change from one read to the next so it is often the case that a set of raw frames compresses very well.  If I have 64k of frames to compress there is lots of room for reducing the file size since there should be lots of repeated patterns.  If I wrote to disk after every frame I would be trying to compress that one frame on it's own.  Now that frame may contain a bunch of 0s and be compressible, but long strings of similar data is more useful.  Back to that example of a file in a zip.  I could make one file with 64k frames in it.  Or I could make 64k files each with one frame in it.  The compression will be much larger if we can get more frames before flushing to disk.

 

Now there is a trade off obviously.  If the bus only gets a new frame once a second, and we are writing to a BLF as they come in, then 64k of frames means over 17 hours before we would flush to disk.  If the power was removed in that time and we shutdown we would lose up to 17 hours of CAN data.  However with CAN our transmission rates are typically much faster.  Our bus load and baud rate can vary greatly, but it isn't uncommon to see a 500k baud bus loaded to 20% or more.  With this rate we have 400 extended frames per second at 8 byte payloads. This is roughly 3 minutes between flushes.  At 80% this is under a minute.

 

So my value of 64k was somewhat arbitrary, but I wanted it to be large.  Which is why I have it as a single constant on the block diagram of the Open File function.  It could easily be made into an input, but I figured most people don't really care.  My use case wasn't to log data as it comes in, but to convert from one file type to another.  In this case I will read potentially millions of frames from one file type then write them to a BLF.  In this case you really do want your chunk size to be as large as it can be.  In addition to being able to change this in the Open, there is the Flush File Buffers, which you can call in your own program as often as you'd like which (should) start the buffer over again.  If you have long periods of bus off, you might want to consider calling that function just after turning things off, to ensure all has been written.

 

Regarding icons.  I agree that it can be confusing having the same icon.  My intent is that the Vector DLL based method will go away, probably in the next release.  I wasn't sure if there was stability, or compatibility issues with my raw method.  Since it's initial release I've gotten feedback and used it enough that I believe it is stable.

 

Regarding package.  I want to make this a package and update everything.  One feature I want rolled into everything is CAN-FD.  You may have noticed that this supports writing and reading CAN-FD frames with payloads up to 64 bytes.  My frame signal conversion library needs to be updated to support this, and my XNet tools may need updating, and my TDMS CAN logging needs to be updated.  I've already done lots of work on these things, but I don't want to package this up, until everything else is ready to be packaged up too.  I do worry that in the next release compatibility will be broken and old functions will be missing that need to be replaced with new ones.

ikc
Member
Member
on

Hi Hooovahh,

 

Hope you are doing well.

We are trying to use your API in a 64 bit version software and we found the "binlog.dll" is built in 32 bit library.

Could you please let me know which version of Vector software (CANoe, CANalyzer) you were using to get the "binlog.dll"?

 

We could build 64 bite software with the latest 64 bit "binlog.dll" file from CANalyze 10.0 but it doesn't create any file.

Any thoughts?

 

Thank you in advance.

A-T-R
Member
Member
on

Hi hooovahh,

thanks for the good explanation of when and why and how!

 

When I have my HW ready, I will do further testing! Let's see which settings are best for our application and use case...

 

Regards

Achim

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

@ikc - I'm not exactly sure what version it was pulled from.  The DLL I use lists version 4.6.2.0 with a date of 2016.  I'd suspect this was around CANalyzer 10.  Sorry you are having trouble with 64-bit LabVIEW.  I don't use 64-bit LabVIEW and am unsure what compatibility issues there are.  Is there a reason you don't use LabVIEW 32-bit?  Also if you do need to stick with 64-bit LabVIEW I'd suggest checking out the raw version (version 5 or newer) which can read and write BLF files by using the compression algorithm built into the OpenG Zip API, and doesn't need the Vector DLL.  I believe version 4.2 of the OpenG zip library linked to here supports 64-bit LabVIEW.

ikc
Member
Member
on

@Hooovahh - Thank you for your prompt reply. That raw version works for us and the 64bit software can create, write the CAN data log file in BLF format. Our team will explore using the 64 bit DLL file option in parallel but that would be in a lower priority at the moment. We do appreciate your time and help!

alii001
Member
Member
on

Hi,Sir,I am using 2014 version2,it seems your code is not working efficiently, it can only read 2000 frames per seconds . That is too slow , even slower than online replay. 

Hooovahh
Proven Zealot Proven Zealot
Proven Zealot
on

The older version of the code posted calls the Vector provided DLL to read and write to the file.  It was painfully slow and I eventually re-wrote the code to use more G code, and have less dependency on external libraries.  The newer code that is on the order of 1000 times faster relies on the OpenG implementation of the inflate/deflate functions of zlib.  If you can test that code out I'd highly suggest it.  If that version still gives you issues with performance please post a set of examples.  Also the code is open source other than the external libraries I mentioned.  If you can improve it please share those improvements. 

HNCW
Member
Member
on

Hi Hooovahh,

 

I have been using your API to create blf files from my tests.

Recently I am having trouble opening some blf files on CANoe 10 SP7, the issue is pretty erratic. No major software change has been done to the software or the unit being tested.

When I compared the blf files of a readable unit vs a corrupt file, I am able to find out that the corrupt file had the header info missing. 
Could you help me read these corrupt files and also solve this issue from happening the future.

 

 

Thank you.

Contributors