DIAdem

cancel
Showing results for 
Search instead for 
Did you mean: 

I would like to get only specific channels from several .csv files and concatenate into one group.

Hello,

 

I am working with other groups and getting the data in daily .csv files.  When I use the "concatenate groups" script along with a script on importing files, I end up getting a huge file that takes about an hour to concatenate.  In order to reduce the amount of time and memory that this takes, I was hoping that someone could help me modify the script so that I could just list the channel name that I am interested in and concatinating only those channels rather than all of them. 

 

For example, if voltage, temperature, pressure and time data are taken daily for 30 days, I would like to import only the temperature and time data (from .csv format) and concatenate into one group.

 

I have attached the .vbs files that I use.

 

Thanks in advance,

 

Alan

0 Kudos
Message 1 of 13
(5,086 Views)

Hi Alan,

 

Actually, the feature you're asking for is already in the code of mine that you sent back. Look on line 11 of the main VBScript:

   

ChannelSet = "" ' "" or "1-" (DataPlugin) or "Sheet1" (EXCEL Wizard)

 

If this "ChannelSet" parameter is set to something other than "" or "1-" then it is used in line 78:

 

Call DataFileLoadSel(FilePaths(i), DataPlugin, ChannelSet)

 

Yo can specify the channel indices to load with an expression like this:

 

ChannelSet = "[1]/[1],[3]"

 

Let me know if you have further questions,

Brad Turpin

DIAdem Product Support Engineer
National Instruments

0 Kudos
Message 2 of 13
(5,072 Views)

Thanks Brad.

 

What you say makes sence but I am not getting the desired results.

 

 The files that I have are about 200 channels and only 1 group.  I went ahead and changed the "ChannelSet" to the way you have it there ([1]/[1],[3]) and it did put the channels into the same file but they were still under different groups.  I also tried the exact channel name rather than the channel number but it did not work.  I am thinking that the error I got "Cannot load the file <file location>\<file name>.csv with the loader <my plugin>." was because the channel did not exist throughout all the logs (sometimes channels are added at different intervals).  In this case, I think having a number for a channel reference would then throw out a different results if the channel locations "shifts" over. 

 

Is there any way that I can concatenate the files into only one group by using the channel name?  Also, is there any way that if a certain channel does not exist in certain logs, that it will treat the channel as if it was virtually there and consider that timestamp null?  Normally, I go into each individual file and add the channel name with no values to all the log files.  In this case, I would have to do it to over 100 files...

 

Thank you for your support,

 

Alan

0 Kudos
Message 3 of 13
(5,060 Views)

H Alan,

 

DIAdem 11.1 and earlier versions can only programmatically load selected channels from a file by channel index.  I am hopeful that in future versions DIAdem will be able to programmatically load selected channels by channel name, but that doesn't help you right now.  Could you send me the DataPlugins and a set of data files which exihibit the full range of channel order variations and missing channels?  There should be a way to work through this in your current version of DIAdem.

 

As for the imported channels showing up in separate Groups in the Data Portal, that is the desired first step.  The other VBScript file you sent back to me, "Concatenate Groups.VBS", is designed to concatenate all the channels together from all those separate Groups, resulting in 1 Group of very long channels-- and this it does based on channel name, by default.  This VBScript is usually called automatically by the main script, so I'm unclear why this is not occurring for you. Again, if you send over the DataPlugin and example files, I'll be much better able to help you.

 

Brad Turpin
DIAdem Product Support Engineer
National Instruments

brad.turpin@ni.com

0 Kudos
Message 4 of 13
(5,042 Views)

Hi Brad, as you can probably tell by now, I am not fluent in scripting.  I have too many files to send all of them in one zip file but I have weeded out some files that I have had problems with...

 

I have tried importing the files in different combinations and get different results every time.  Sometimes I get an error stating that "the operation does not work with write protected channels" referencing the concatenate groups script line 381, column 29.  Other times I get an error that the file cannot be loaded with the dataplugin that I have on the script (though when I open the file alone with the dataplugin it opens just fine) and other times it gives me an error in the custom imports script line 74, column 15, error executing "DATAFILELOAD" command.

 

Attached are a few files that don't concatenate and the dataplugin I use.  I also included a screenshot of the error I get when I try to import and concatenate the entire folder I am trying to combine (a total of 110 files).  You can also see that the screenshot shows a different error as well.

 

Thanks again!

 

 

 

 

 

 

0 Kudos
Message 5 of 13
(5,007 Views)
On another note, since I can import several files already, how can I concatenate them into just one group?  This might be a simpler solution rather than trying to find the problem with the files themselves...
0 Kudos
Message 6 of 13
(4,994 Views)

Hey Alan,

 

I will look into hopefully fixing the fully automated script that loads and concatenates, but I like your suggestion of a temporary solution where you load interactively and concatenate via script.  The VBScript application you sent back to me contains a main VBScript file and a second VBScript file that is called by the main VBScript.  This second "Concatenate Groups.VBS" script file does exactly what you're requesting-- it concatenates channels of the same name located in different Groups into one long channel of that name, and it does this for all channels which have matchers in other Groups.  Try loading a few data files into the DataPortal and running this second VBScript in directly in DIAdem SCRIPT (without using the "Intertek Custom Imports.VBS").

 

More soon, hopefully,

Brad Turpin
DIAdem Product Support Engineer
National Instruments

0 Kudos
Message 7 of 13
(4,992 Views)

H Alan,

 

You sent over a "Calterm III" DataPlugin, but the "Intertek Custom Imports.VBS" was configured on line 19 to use the "Intertek HALT Plugin" DataPlugin. When I tried to use your "Calterm III" DataPlugin, I just got a bunch of properties and no data channels in the Data Portal.  I happened to have a "Calterm II" DataPlugin handy, and that seemed to work pretty well, though it loads the first Time channel as a string channel, which would bug me.  Anyway, after changing line 19 to use the "Calterm II" DataPlugin, it looks to me like everything is working with the files you sent.  I also changed line 18 to look for *.csv files in the file dialog.

 

Brad Turpin
DIAdem Product Support Engineer
National Instruments

0 Kudos
Message 8 of 13
(4,989 Views)

Hi,

 

I am trying to load some channels from a .csv file which contains around 600 channels. The command DataLoadSel works perfect when I write the number of the channel I want to load, but what I really want to do is load the channels that have the same name as a DefaultChannelsNames which are in an other .csv file (contains the names of the channels required for my analysis) Is it possible?

 

Thank you for your help

 

Maria

0 Kudos
Message 9 of 13
(3,635 Views)

Hi Maria,

 

This forum is over five years old and you might do better to create a new thread. You'll get more traffic that way and help keep your questions separate from the original post. You can reference this post if you need to.

 

Thanks!

Kelsey W.
National Instruments
Applications Engineer
0 Kudos
Message 10 of 13
(3,617 Views)