Reading Large Text file

[Munna232]

Hi GerdW,

As per my 1^st post, if I want to know start & End then 1^st I need to read text file.

After that again I need to read Text file for Data Conversion.

[GerdW]

Hi Munna,

so you generate the left most table by reading the middle CSV table?

How do you know a signal has "ended" when the value doesn't change anymore? What is the difference between a signal with stable value and a signal, that isn't received anymore?

[Munna232]

Hi GerdW,

I created Left most Table from Text file.

I’m checking for Signal ID. Where it is Ending & Starting. It’s not related to Value. It’s about CAN ID.

Doesn’t matter if Signal value is stable or different. If CAN ID is Stopped coming after certain time then I need to display NaN.

In my snippet, Table2 is related to Old CSV file. I’m writing Data as per Table2. But I want it like Table3.

[GerdW]

Hi Munna,

so you know the "end of message" by it's missing msgID. Again: what's the problem of selecting between last value and NaN depending of that information?

You know the ID and you know it's send interval. So you know when to expect a new message and when the message is lost…

No need to scan the whole file twice. Just keep information on msgID, it's interval and expected new msg data in a shift register…

[Munna232]

Hi GerdW,

---

GerdW Wrote:

so you know the "end of message" by it's missing msgID. Again: what's the problem of selecting between last value and NaN depending of that information?

You know the ID and you know it's send interval. So you know when to expect a new message and when the message is lost…

No need to scan the whole file twice. Just keep information on msgID, it's interval and expected new msg data in a shift register…

---

Without reading File, how can I get the End of message?

Here, I can't consider Signal Send Interval. Because CAN Signals have specific Interval + Event. Event means, if user performing some operation then Signal will come so, Line number will change.

if User is not performing any operations then it will come in same interval.

[GerdW]

Hi Munna,

in your 1st post you clearly stated a "periodicity" of your CAN signals. Now you tell me, you get values of this interval!

Please get all your information sorted. Make clear conditions on when a signal is dead and when it is still alive…

[Munna232]

Hi GerdW,

Sorry,

In my 1^st Post, in some cases Signal will come periodically. But sometime it will come before that interval (Because of user Event).

So, we can’t consider this parameter to check CAN is dead and when it is still alive (all the time).

By reading Text file, I will get where signal will start & Ends.

Suppose Signal is coming at line no. 100,120,130,145,489,500,609 (as example) and if I need to check the value in between then I need to take previous value.

After 609 I need to display NaN.

Here, Start Line 100 & End Line 609.

I hope now it’s clear for you.

[RavensFan]

And you can still do that reading through the file the first time.  Keep track of your previous line in a shift register.  When you get to those points where you realize the signal is dead, then you use the value in the shift register rather than the newly acquired values.

[Munna232]

Hi OriolesFan,

For Example My file have 30000 lines.

Signal A, 1^st time came at row 10, after that 20000 then stopped.

So while data conversion after 10^th Line signal is not coming (up to 20000) so keep on taking previous value. After 20000 again I signal is not coming (up to 30000) so I need to display NaN. It should to not take previous value.

If I want to confirm signal is stopped at line no. 20000 then I should read up to 30000 lines.

[GerdW]

Hi Munna,

you can perform the very same check of "CAN signal dead" in the loop, where you decode the data file.

Why do you think you need to decode the file one time for determining signal start and stop and then start to decode the very same file again to actually parse the data? Include both steps in just one loop!