LabVIEW
Limitation of data value references (DVR) in system
@RavensFan wrote:
Who knows what the exact reason is you get an out of memory message. --> thanks, helpful
I linked a lava post, which contains a simple vi, but the code is trivial:
I know, the easiest to say, your architecture is wrongly designed, but:
- Currently we are using much less than 1M, but as I said, we build that "model" from an input file, which may be in the future much larger
- You say, 2^20 reference (to an object) is ridiculously large number, but 2^20 object is not?
And what if I have a lot of measurement data and I would like to reach those by reference? I know your answer: bad architecture. - Independently from the programming language, if I try to do the same for example in Java or C, what should you say? (please dont say that: "than why dont implement your application in Java?")
Actually im waiting for just a definite (_reasoned_) answer.
I don't have any answer about possible limitations on the number of DVRs, but I still don't understand what you are trying to do. How does dealing with millions of different DVRs simplify anything. What is the datatype of your DVR? How are you planning to decide which DVR to access (i.e. where does the index come from?) Instead of indexing into an array of DVRs, you might just index into an array of integers (same input (index), same result (value). To only have the bulk of the data stored in one location (array, variant attributes, tree, etc.), you could design an "action engine" containing the data with modes to read, insert, define/return relationships, etc.
I agree with Altenbach, either variant attribute (it might also have the same limits btw) or a simple array of numbers/clusters (in the case of a family tree it'd be the cluster of person information), this cluster would include a Parent index and an array of child indices and then the data itself.
/Y
@ViltBalint wrote:
A few comments for your answers:
- is there any chance, it will be changed in the future Maybe with a change in OS like going to 64-bit. Otherwise, I doubt it.
Not likely in the current form of LabVIEW. Refnums are 32 bit entities (also on 64 bit) where 20 bits or so are reserved for the index into the refnum container (or cookie jar as they call it internally in LabVIEW) for the specific refnum type and the rest is used to give a possibility to have some uniquely identifiable magic value that allows to verify that the refnum is actually a valid refnum for the actual refnum type. This magic value is dynamically created on initialization of the first refnum of a specific type.
They could of course increase the 20 bits for the index to lets say 24 or so but that would not really solve your problem. A refnum in itself is a resource that takes up some memory. For a DVR it is basically a 32 bit reference to a memory area that takes up probably something like 40 or more bytes since LabVIEW needs to maintain not only the actual data pointer itself but also the type description of the DVR, at least a semaphore for protecting access to the DVR and possibly other things to help protect it from multiple concurrent access and some bookkeeping information. And then there is of course the data itself which can vary from one byte to many MBs if you wrap an array with a DVR.
You basically made the error to use DVRs to manage a huge amount of small objects. This can work if you program in C and use more or less bare pointers to manage that but even there most would think twice about using a linked list of elements to manage millions of tiny data blobs. It's simply not very efficient if the management structure takes up significantly more memory than the data element it is supposed to manage.
And in your case it seems you will end up with a signficantly more complicated data structure than a single integer for each of your data elements. And then you will soon run into a real out of memory situation if you intend to create a million of those.
DVRs are great to manage large data chunks. Their reference character allows to get around the problem of dataflow that mandates that in order to maintain always consistent data you have to often create temporary copies of the data, which can get very expensive for large data objects (millions of data samples for instance). But in order to maintain data consistency, DVRs have to implement concurrent access protection which makes them more expensive to manage and also slower to access. They are the wrong element to implement a highly relational object hierarchy model.
@RavensFan wrote:
Who knows what the exact reason is you get an out of memory message. You haven't posted any code for anyone to experiment with to see if they get the same results. Although, I realize it might be difficult to replicate.
It's pretty simple. When implementing code you have the choice to do some error handling or none. If you do the later, the system simply crashes. If you do the first you have to find some error code or something to pass on to indicate the failure to the caller. Now you can go and invent a new error code every time you run into such a situation. This results in a huge, unmaintainable and likely sooner or later conflicting list of error codes, as two people invent the same error code at the same time for wildly different errors. So you go and look in the existing list of error codes and grab the first that somehow feels like describing the situation.
Is this an out of memory situation? In a sense it is, though it would be better described as an out of resources error. Would I punish the programmer who choose to use this error code in this situation? Absolutely not! I, and I would very much assume everyone else who writes software, has been guilty of using an error code that after more careful review may not have been the absolutely best code to choose to describe the actual situation. That also assumes that someone even does any error handling at all. I see regularly LabVIEW programs were error handling is not even wrong but simply completely absent!
