Storing and recovering data types

[George Haldeman]

I am looking for a mechanism to save the data type (typically a cluster
of arrays and strings) which will represent the basic record for a datalog
file and which can be recovered easily, hopefully within a labview
program itself. I am planning on using the datalog records as the primary
data storage for an acquisition system, but am worried about losing the
ability to read and process data if I modify the structure (to store more
or
different data) at some time in the future.

It seems like one way to do this is to use type definitions and to just keep
copies of the critical type definitions whenever I change the program. This
seems to be innefficient and not necessarily failsafe.
Does anybody have a better idea?

[Gilbert Hangartner]

Hi George!

I have solved the problem exactely as you suggest, but I've still having a bad
feeling thinking about future changes ...
Seeing your question, I though of the following solution:

writing:
1. Flat the data to string.
2. Write the type string in a file header
3. Write the flattended data in the rest of the file.

when you read from the file, you first read the type, and then use unflatten
from string using this type.

What do you think about that? (I did not actually test it yet, but think it
should work...)

Gilbert


George Haldeman wrote:

> I am looking for a mechanism to save the data type (typically a cluster
> of arrays and strings) which will represent the basic record for a datalog
> file and which can be recovered easily, hopefully within
a labview
> program itself. I am planning on using the datalog records as the primary
> data storage for an acquisition system, but am worried about losing the
> ability to read and process data if I modify the structure (to store more
> or
> different data) at some time in the future.
>
> It seems like one way to do this is to use type definitions and to just keep
> copies of the critical type definitions whenever I change the program. This
> seems to be innefficient and not necessarily failsafe.
> Does anybody have a better idea?

[George Haldeman]

Hi Gilbert,

Thanks for your reply. Your idea is a good one, but I cannot figure out
how to
unflatten the string data without having a constructed data type. It seems
like
the unflatten function wants an actual constructed cluster to wire in instead
of
the type descriptor that you get out of the flatten function. I kept thinking
there
must be some way to do it, but I cannot find anything short of using the
manual
and writing my own routines to decode it.

I haven't been doing this for long, am I missing something obvious?

George

Gilbert Hangartner wrote:
>Hi George!>>I have solved the problem exactely as you suggest, but I've
still having a bad>feeling thinking about future changes ...>Seeing your
question, I though of the fol
lowing solution:>>writing:>1. Flat the data
to string.>2. Write the type string in a file header>3. Write the flattended
data in the rest of the file.>>when you read from the file, you first read
the type, and then use unflatten>from string using this type.>>What do you
think about that? (I did not actually test it yet, but think it>should work...)>>Gilbert>>>George
Haldeman wrote:>>> I am looking for a mechanism to save the data type (typically
a cluster>> of arrays and strings) which will represent the basic record
for a datalog>> file and which can be recovered easily, hopefully within
a labview>> program itself. I am planning on using the datalog records as
the primary>> data storage for an acquisition system, but am worried about
losing the>> ability to read and process data if I modify the structure (to
store more>> or>> different data) at some time in the future.>>>> It seems
like one way to do this is to use type definitions and to just keep>> copies
of the critical type defin
itions whenever I change the program. This>> seems
to be innefficient and not necessarily failsafe.>> Does anybody have a better
idea?>

[Kevin_work]

George Haldeman wrote:

> Hi Gilbert,
>
> Thanks for your reply. Your idea is a good one, but I cannot figure out
> how to
> unflatten the string data without having a constructed data type. It seems
> like
> the unflatten function wants an actual constructed cluster to wire in instead
> of
> the type descriptor that you get out of the flatten function.

This is correct and is reflected in the help description of the Unflatten function.
The type descriptor from the flatten is a 1D of decimal. If you wire that to the
unflatten you will get a 1D of decimal and the data is meaningless (at least to me).

> I kept thinking
> there
> must be some way to do it, but I cannot find anything short of using the
> manual
> and writing my own routines to decode it.

This is basically the only way to do this.

>
>
> I haven't been doing this for long, am I missing something obvious?
>

In my opinion this is an unfinished feature. This is one of my pet peeves about LV.
The string that is created from the flatten function contains the data
AS WELL AS the type description.
I see no reason that this could not be used by the unflatten to recover the data.
These two functions would seem to be the inverse of one another but they really arent.
There must be some way of doing this easily since the Dataloggiing feature has no
problems.


>

Back to the original question, I would use a strict type definition. If you are
worried about losing it then simply make a backup copy of it. Also as far
as I remember you can not mix datalog types in a file. If you change the type
you will have to recover all the old data, reformat it to the new type and rewrite the file.
So changeing the format is not a simple undertaking.

Good luck
Kevin Kent

>
>
> Gilbert Hangartner wrote:
> >Hi George!>>I have solved the problem exactely as you suggest, but I've
> still having a bad>feeling thinking about future changes ...>Seeing your
> question, I though of the following solution:>>writing:>1. Flat the data
> to string.>2. Write the type string in a file header>3. Write the flattended
> data in the rest of the file.>>when you read from the file, you first read
> the type, and then use unflatten>from string using this type.>>What do you
> think about that? (I did not actually test it yet, but think it>should work...)>>Gilbert>>>George
> Haldeman wrote:>>> I am looking for a mechanism to save the data type (typically
> a cluster>> of arrays and strings) which will represent the basic record
> for a datalog>> file and which can be recovered easily, hopefully within
> a labview>> program itself. I am planning on using the datalog records as
> the primary>> data storage for an acquisition system, but am worried about
> losing the>> ability to read and process data if I modify the structure (to
> store more>> or>> different data) at some time in the future.>>>> It seems
> like one way to do this is to use type definitions and to just keep>> copies
> of the critical type definitions whenever I change the program. This>> seems
> to be innefficient and not necessarily failsafe.>> Does anybody have a better
> idea?>

[Gilbert Hangartner]

Hello Kevin, Hello George!

Sorry, I was confused. You can't you use "Unflaten from String" as Kevin pointed out, but I was
thinking of the "Invoke Node", "Set Control Value" method, which you can also apply locally, and which
accepts the "type string" (strange terminology to call an array of numbers "string" ...) as it comes
out of the "Unflaten to String" function.

Anyway, when I tried to implement it, I realized that also this will not help: LabView does not have
controls which automatically adapt to source (what would also be great to write polymorphic SubVIs),
so you can't actually do anything with your recovered data, if you are not knowing to what kind of
control you should wire it ...

So, sorry for confusion, and let's stick to the "Strict type definition" ...

Greetings, Gilbert

"Kevin B. Kent" wrote:

> George Haldeman wrote:
>
> > Hi Gilbert,
> >
> > Thanks for your reply. Your idea is a good one, but I cannot figure out
> > how to
> > unflatten the string data without having a constructed data type. It seems
> > like
> > the unflatten function wants an actual constructed cluster to wire in instead
> > of
> > the type descriptor that you get out of the flatten function.
>
> This is correct and is reflected in the help description of the Unflatten function.
> The type descriptor from the flatten is a 1D of decimal. If you wire that to the
> unflatten you will get a 1D of decimal and the data is meaningless (at least to me).
>
> > I kept thinking
> > there
> > must be some way to do it, but I cannot find anything short of using the
> > manual
> > and writing my own routines to decode it.
>
> This is basically the only way to do this.
>
> >
> >
> > I haven't been doing this for long, am I missing something obvious?
> >
>
> In my opinion this is an unfinished feature. This is one of my pet peeves about LV.
> The string that is created from the flatten function contains the data
> AS WELL AS the type description.
> I see no reason that this could not be used by the unflatten to recover the data.
> These two functions would seem to be the inverse of one another but they really arent.
> There must be some way of doing this easily since the Dataloggiing feature has no
> problems.
>
> >
>
> Back to the original question, I would use a strict type definition. If you are
> worried about losing it then simply make a backup copy of it. Also as far
> as I remember you can not mix datalog types in a file. If you change the type
> you will have to recover all the old data, reformat it to the new type and rewrite the file.
> So changeing the format is not a simple undertaking.
>
> Good luck
> Kevin Kent
>
> >
> >
> > Gilbert Hangartner wrote:
> > >Hi George!>>I have solved the problem exactely as you suggest, but I've
> > still having a bad>feeling thinking about future changes ...>Seeing your
> > question, I though of the following solution:>>writing:>1. Flat the data
> > to string.>2. Write the type string in a file header>3. Write the flattended
> > data in the rest of the file.>>when you read from the file, you first read
> > the type, and then use unflatten>from string using this type.>>What do you
> > think about that? (I did not actually test it yet, but think it>should work...)>>Gilbert>>>George
> > Haldeman wrote:>>> I am looking for a mechanism to save the data type (typically
> > a cluster>> of arrays and strings) which will represent the basic record
> > for a datalog>> file and which can be recovered easily, hopefully within
> > a labview>> program itself. I am planning on using the datalog records as
> > the primary>> data storage for an acquisition system, but am worried about
> > losing the>> ability to read and process data if I modify the structure (to
> > store more>> or>> different data) at some time in the future.>>>> It seems
> > like one way to do this is to use type definitions and to just keep>> copies
> > of the critical type definitions whenever I change the program. This>> seems
> > to be innefficient and not necessarily failsafe.>> Does anybody have a better
> > idea?>

[Dave Trickey]

If the number of records is not huge (and nowadays with 8-gig hard
drives going for $150, its usually no problem to waste a few
megabytes) why not define a single typedef with more entries than
you'll ever need to "add" new data just use one of the dummy entries?

On 14 Apr 2000 13:17:30 -0500, "George Haldeman"
wrote:

>
>I am looking for a mechanism to save the data type (typically a cluster
>of arrays and strings) which will represent the basic record for a datalog
>file and which can be recovered easily, hopefully within a labview
>program itself. I am planning on using the datalog records as the primary
>data storage for an acquisition system, but am worried about losing the
>ability to read and process data if I modify the structure (to
store more
>or
>different data) at some time in the future.
>
>It seems like one way to do this is to use type definitions and to just keep
>copies of the critical type definitions whenever I change the program. This
>seems to be innefficient and not necessarily failsafe.
>Does anybody have a better idea?
>

[George Haldeman]

Hi Dave,
Thanks for the idea. I am not sure that this will work since you still
have to define
the data types that you add for future upgrades, and if they themselves are
clusters, then the complexity
you need to cover is considerable. You could force yourself not to use clusters,
but then you have arbitrarily cut yourself off from one of the more powerful
data features in labview. What do you think?
George
dtrickey@snet.net (Dave Trickey) wrote:
>If the number of records is not huge (and nowadays with 8-gig hard>drives
going for $150, its usually no problem to waste a few>megabytes) why not
define a single typedef with more entries than>you'll ever need to "add"
new data just use one of the dummy entries?>>On 14 Apr 2000 13:17:30 -0500,
"George Hal
deman" >wrote:>>

[Darren Harrison]

Hi George,
I had to do the same thing (I think!) for a project, i.e. save a number of
clusters etc.

I wanted something fairly fault tolerant & easily upgradeable. ie. not datalogs.
I came up with a function that converts a given control to text that can
be written to an INI file.
To make it a little more general, I use the Application Control -> Invoke
Node, GetAllCtrlVals & Set Control Value. This means I can drop a bunch of
controls onto a form, use the vi I created which reads all control values
from it's Parent vi, & returns them as text (complete with control names)
all ready for an INI file.
For reading, I find out all the control names from the Parent vi & look for
the INI entries, then I format the result using the Type Descriptor again,
& it's all ready for "Set Control Value"
Because Type Descriptors aren't necessarily stable between LabVIEW releases
it may require changes for LabVIEW 6 (???) or whatever.

I can't really claim any credit for this - I think I got ideas from the 'Cluster
to String' vi (See thread "Re: Cluster to string") & maybe others.

It all seems to work quite well, it's fault tolerant, allows me to upgrade
structures without having to upgrade huge numbers of datalog files & should
survive a lot better through LabVIEW versions than datalogs (guess who got
caught with the LV4 boolean change!!! :-).
All the best, let me know if you need sample VI's.

Darren.

"George Haldeman" wrote:
>>I am looking for a mechanism to save the data type (typically a cluster>of
arrays and strings) which will represent the basic record for a datalog>file
and which can be recovered easily, hopefully within a labview>program itself.
I am planning on using the datalog records as the primary>data storage for
an acquisition system, but am worried about losing the>ability to read and
process data if I modify the structure (to store more>or>different data)
at some time in the future.>>It seems like one way to do this is to use type
definitions and to just keep>copies of the critical type definitions whenever
I change the program. This>seems to be innefficient and not necessarily
failsafe.>Does anybody have a better idea?