I need to import data from a ASCII csv file.
The problem is the format of the date in this particular data file (.csv)
The date format for Days < 10 and Month from Jan to Sep is:
YYYY/_M/_Dthe character "_" is a simple blank space, not a literally underscore
The date format for Days >= 10 and Months from Jan to Sep is:
The date format for Days >= 10 and Months from Oct to Dec is:
The date format for Days < 10 and Months from Oct to Dec is:
I create already a routine that recognize the date in the tittle and use one of the 4 filter (.stp) to process de data. It's works.
The days are separeted in 2 Files, one file go from 9:00 to 21:00 of one day and the other one from 21:00 to 9:00 of the next day. That's means that i lost data when the day change from 9th to 10th each month and the month change from Sept to Oct, and Dec to Jan.
I just need to delete the unconfortable blankspace before the dates changes to 2 digits, but i don't know how to process the CSV File before imported to Diadem (10.2).
My other idea is recognize this files and imported double (once with each stp) but i will need to positioned the import exactly to dont have problem with the novalue data. (Until know i found to complicated for a simple blankspace error).
I hope you have some ideas ...
Thanks in advance...
Solved! Go to Solution.
I would think the most efficient way to read the data is a DataPlugin. Given your format, its probably necessary to create this plugin manually (=requires a little scripting).
You could either look into the documentation describing how to create a Dataplugin or you would provide a few example data sets so that we can provide a DataPlugin to help you get started.
Thanks for the fast answer. Yes it is the most efficient way and i did before with normal data format (with the wizard), but with this one have been a nightmare
Exactly that is what i want, create the data plugin by hand (or minimun just the filter), some short examples could help me a lot. Here i attach the small version of the typical data file in the point where the format change.
I will start to look info about scripting in a Data plugin
Thanks a lot.
the attached dataplugin does a little processing for the date/time columns to handle the format. Besides that, everything else is standard processing. I hope this works for you..
Excelent, Thanks a lot,
Investing a little bit time i understand now how is working, (of course work well) but this syntaxis is not the same that i use in the Script in Diadem, so i habe some problems now to integrate my last scripts with this data plugin.
I will explain my self better:
1st - I used before (with my 4 .stp filters) the code for "Importing Data From Multiple ASCII Files" and "Concatenate Groups", and i adappted fine except for the filter problem. It is possible to open multiple files with a Data Plugin ?
2nd and more simple - i used in the same script the funtion DataSetSave to save after open in TDM format, i don´t find the right command to do it in the data plugin (so far). Eg: Call DataSetSave(FileDlgFile & ".TDM","")
Unfortunatly i´m not so involve with this syntax from the data plugin, or better i should say that the script syntax is more faster and easy to understand and replicate based in examples. the syntax "[Group]/[Channel]" it is easy to follow.
Exist tutorials for Data Plugin programming ? (i didn´t found so far) or some recomendation.
Thanks a lot for the help, really usefull already.
I am not sure whether I understand what steps are unclear. Here is what I would think you want to do :
1) Load data from different files into DIAdem
To do so, you would use "Call DataFileLoad("E:\Customer_Requests\caracasnet\log(111231).csv","caracasnet_log","Load") "
You would call DataFileLoad for each of the files.
2) You want to concatenate the groups.
This should be no different from what you have done so far
3) You want to store data to a (TDM) file.
To do so, you would call DataFileSave(....)
Let me know if you have further questions...
I found the solution in the forum
To make clear i have been working just with scripts, until i found the problem with the data date format.
With the Dataplugin the format problem is solved. But i i was looking the way to integrate the data plugin to my normal scripts or viceversa.
Finally reading the forum for 3 time (my fault) i discover how i could just integrate EASY the data plugin into my normal Script.
Thanks a lot for the fast answer and the excellent help...
Until my next problem
I have a new problem with the date format... different logfile, different work but the same post tittle...
I have a CSV file to read with the date/time column in days (some version of time format design originally for excel ), i am able to process it by SCRIPT to change to a normal format + 693958)*24*3600 is working fine.
Call ChnCalculate("Ch(""/Zeit"") = (Ch(""/date\time"")+693958)*24*3600)
The problem come when i try to do it directly with the dataplugin, until now i am able to order data by groups, channels, change names and thanks to the forum some different date format. But i never worked with normal operations like in the scripts and i should be missing something because is not working.
My plan is:
- load all data like eR64 including the first column that is date/time (done)
- create a new column where the new time format will be (done)
- make the operation between both (2 days of forum and nothing)
- delete the old column date/time (not even try)
Any suggestion ?
Here i attached both files.
Thanks in advance
Actually DateTime channels in DIAdem are just R64 (Double) numeric arrays in DIAdem, scaled to contain the number of fractional seconds since 0 AD. This means that you can use the DirectAccessChannel you already have that contains the number of days since about 1900 (Excel standard) and apply the following scaling and display property:
Set oDAChn = oBlock.Channels.Add("DateTime", eR64) oDAChn.Factor = 3600*24 ' Excel oDAChn.Offset = 3600*24*693958 ' Excel oDAChn.Properties.Add "DisplayType", "Time" Call oGroup.Channels.AddDirectAccessChannel(oDAChn)
DIAdem Product Support Engineer
I figure it out yesterday, but i wrote different (amateur code) so i rewrite it like you say and is working well...!!
Can i ask you fast: How i change the name of the group in the dataplugin ?
I need to change the name of the group but based in the data inside (the header doesnt have the info, it is in a channel) and because i create the group and later the channels inside i dont know how rename it. If i take the first line of data with getnextline and proccess it, i'm able to do it from the beginning (not renamming) but i miss that first line in my data.
Any suggestion ? should i open other topic with the question ?