CSV Script

[Bilby42]

I'm new to DIAdem so please excuse the basic nature of this request. I'm attempting to create a plugin for my csv data structure. I've attempted using the Wizard, but I've since found in the help that if the number of columns in the csv is > 50 you need to generate the plugin directly with the script editor. My initial attempts are not pretty, hence this question!

The csv data structure is unusual and I've attached a mockup csv representing a simplified dataset. Basically first line is header information. Subsequent lines are all data but mixed in an unusual way. "Date", "Time", "Position X", "Position Y" are as expected and represent a specific location on an XY grid. The bit I'm struggling with is the DataSample1, DataSample2 .... DataSampleX etc. 

These columns are the results of multiple timed data samples while at each relevant XY position, i.e. for each XY position there is an array of DataSample values, with a common timing increment. I'm having trouble creating a script to import this "DataSampleX" information into a channel and linking this to the positional data. Additionally, in the actual files there are about 150 DataSampleX points per PositionX/PositionY location (hence the wizard doesn't work).

Suggestions on a start to the script would be extremely helpful.

[Bilby42]

Further to my previous post, I've been considering how my data should be structured within the Data Portal. I've attached a screen shot of a possible structure I think may work. 

Basically, I'll use the channel called "DataScanIndex" to hold integers which reference each set of DataSamples for each position, i.e. DataScanIndex for the first set of X/Y data will contain "2", which links to the channel "DataScan1".  Assuming any groups created during further processing are added to the end of the Data Portal, my indexes should allow me to index the correct DataSample set.

[AndreasK]

I did not really understand how the data should be imported but your data can be imported using the default "TextFile autodetect *.csv" plugin.

DataFileLoad(filePath, "CSV")

which allows you to do preprocessing in DIAdem.

Format:

Date, Time, Position X, Position Y, Misc Heading1, Misc Heading2, DataSample1, DataSample2, DataSample3
22/1/17, 14:22:00, 407, 900, blah, blah, 45, 46, 47
22/1/17, 14:22:00, 408, 901, blah, blah, 25, 26, 28
22/1/17, 14:22:00, 409, 902, blah, blah, 90, 91, 92

An example how preprocessing might look like is the attached code which

• merges the time channels
• sorts data by x,y,time

I am not sure what your use case is but maybe this might help you.

A lot of methods are reachable by ANALYSIS Tab and can be teached using the macro recored (Teach in).

Option Explicit

if "IDOk" = FileDlgShow(DataReadPath, "CSV file (*.csv),*.csv", "Select a csv file", false) then 
  call LoadFile(FileDlgNameList(0))
end if

function LoadFile(filepath)

  ' use default CSV plugin that can be used for this format
  dim elems : set elems = DataFileLoad(filePath, "CSV")
  dim grpO : set grpO = elems(1)
  if grpO.IsKindOf(eDataRoot) then
    ' if portal is empty root is returned
    set grpO = grpO.channelgroups(1)
  end if
  
  ' address is used a lot of times so we define it here e.g. [1]
  dim grpAddress : grpAddress = "[" & grpO.Properties("index").Value & "]"
  
  ' merge date and time to abstime
  call ChnAdd(grpAddress & "/Date", grpAddress & "/Time", grpAddress & "/AbsTime")
  call grpO.Channels.Remove(2)
  call grpO.Channels.Remove(1)
  call data.Move(grpO.Channels("AbsTime"), grpO.channels, 1)
  
  ' sort all channels by x,y,time
  call ChnMultipleSortExt(_
    "'" & grpAddress & "/Position X', '" & grpAddress & "/Position Y', '" & grpAddress & "/AbsTime'",_
    "'" & grpAddress & "/[4]' - '" & grpAddress & "/["& grpO.channels.count & "]'",_
    "Up,Up,Up",_
    "UpperCase,UpperCase,UpperCase",1)

  set LoadFile = grpO
end function

[Bilby42]

Hi AndreasK,

Thank you for the fast reply.

Re my data format: I'm probably not explaining it clearly. Each line in the csv file refers to an X/Y location in my test system's grid. At each system grid location an array of data is measured. So every line also has a corresponding measurement array which is simply inserted into the remaining columns (DataSample1...DataSampleX in my example).

It would be clearer if a TDMS channel could hold a pointer to an array of data. So for instance I would then have a channel called "WaveformPointers", which would in turn hold a waveform for each X/Y value. I've tried to emulate this by having a channel that holds channel indexes (Called DataScanIndex in my example).

I have been attempting to write a plugin, as I thought that was the best way to structure everything. It appears you can simply place the csv decoding in your main script.

Thanks for your code sample. I'll experiment further.

Regards,

Tom