Access to a remote database

[JCNT]

Hi everyone,

I made an application which stores data using the storage APIs. I use these data to show them in a waveform graph and make some calculations. This data are stored in the same computer (server) where my application is installed.

Now I am trying to make a client program which has the same characteristics in showing and using data but not storing. I could get the data (for my client PC) from the data base of my server computer, but there is a problem  if the server computer write at the same time when my client wants to read the data.

Is there any way to make this just using LabVIEW and avoid this problem? By the way, I am using LabVIEW 8.2

I would appreciate any help. Thanks.

JC

[Herbert Engels]

If you use the functions on the TDM Streaming palette (and hence the TDMS file format), you should be fine. The writing application allows other applications to open the file for reading. You need to make sure that the reading application opens the file "read-only".

Herbert

[JCNT]

Hi Herbet,

Thank you so much for reply. The TDM streaming palette is the one I am using and I exactly did what you said and I didn't have any problem. However, it could happen while the server is writing the data (just for a moment) a client wants to read the database at the same time. In that case, the client won't be able to read the data (a error file is created). On the other hand, if the client is reading the data and the server wants to write the data, it won't be able to write. 

I attached the test that I did and the error file. In order to coincide the client and the server at the same time, I put the Write Data TDM in a while loop.

Thanks again and I hope you can help me.

[Herbert Engels]

You're actually using the Storage VIs with the TDM file format. You might want to try using the TDMS file format instead. If you cannot select TDMS in "Open Storage", you will need to install the TDMS DataPlugin. The plugin is available as a free download from www.ni.com/dataplugins.

The difference here is that the meta data (properties etc.) in a TDM file is stored to a separate, XML based file. This file needs to be completely rewritten when you add any data. So, when the reading VI tries to access it, it might happen that the file is gone at this moment. That's where this error is coming from. TDMS works different from that and shouldn't have this problem.

The task you want to accomplish can probably be solved equally well with the Storage VIs and the TDM Streaming API. However, the functions on the TDM Streaming palette are designed for working with TDMS files, so you might find them easier to use than the Storage VIs. But - unless you run into any performance issues - using the Storage VIs is perfectly fine.

Hope that helps,
Herbert

[JCNT]

Hi Herbert,

I was confused with the Storage VIs with the TDM file format and the TDM Streaming files but I already understood. I did the changes that you suggested me (select TDMS in Open Storage) and I did the test again. I got these results:

- If the server is writing data and the client wants to read data at the same time, there is not any error (great!!!)

-If the client is reading data and the server wants to write data at the same time, a error file is created and the data are not stored ()

I would like to know how I can avoid this last error because it could happen.

Thank you so much for your help which is really useful and I will wait for your advice in order to solve this problem that I know will be soon.

My regards,

JC

Note: I attached the error file and the test.

[mikeporter]

What sort of database are you saving your data in? Any DBMS worth the name should be able to insert and select data at the same time. How big are your datasets?

Mike...

[JCNT]

Hi Mike, thanks for reply

I'm just using the storage VIs to save data in the TDMS format.  Also, it is about 3MB per day my database size.

I solved the problem that I had before (I changed the "open" option for the "open or create" in the open storage function terminal); however I have another big problem, in the TDMS format I got an error if I create 2 groups and for each group four channels (I attached this error file).

I would like to know if it is possible to do this or if it is only possible with the TDMS Streaming APIs.

Thanks so much in advance Mike.

JC

[mikeporter]

<personal opinion>

My main input is that in my experience proprietary file formats end up being a huge pain in the neck. Better to store data in a platform and application independent form like in a real database. There are many good products available (note: Microsoft products like SQL Server and Jet should NOT be included in this list) including several execllent industrial-strength open-source products like Firebird. With a real database the problems of concurrent access, back up, and archiving simply goes away.

In terms of storing the data itself, the data would reside in a field with a datatype of BLOB (which stands for Binary Large OBject - one of my all-time favorite computer acromyms). To convert your data into this format all you do is flatten it to a string, convert the string into an array of U8s and you can stuff it into the database. When reading the data out, the blob will come back as a string the you simply unflatten and **TADA* you have your data back.

Interestingly, this format is also easy for applications other than LV to access. In my current work we are storing a lot of data in LV data structures in BLOBs. Another department wanted to access this data for web-based report generation. The documentation of LV datatypes that NI provides is so good it took the web developers only about 24 hours to create a Java applet to convert the data for their web pages. In addition NI has an example that shows how to access LV from Excel using ActiveX. The example demonstrates reading an X-Y plot into the spreadsheet. Over the course of a couple days, I took this example and modified the LV-based ActiveX control to read our BLOBs from the database and make them available to other applications. The end result is our users can now call up all their LV-generated test data from Excel - including the tables and graphs.

Finally, a lot of people are nervious about interfacing to a database, but it has been my experience that using a database is easier than writing GPIB drivers (especially drivers for HP boxes). One warning though, after you have developed your first application with a database you will be so spoiled by the efficiency it produces you won't want to go back to your old techniques...

Mike...

</personal opinion>

[Herbert Engels]

The TDMS file format is documented. TDMS files can be read into Excel with an add-in that NI provides free of charge, plus there's a DLL available on ni.com that lets you read TDMS in many other environments, e.g. MatLab. TDMS is owned by NI, but it is open to anyone.

Storing flattened data to SQL BLOBs comes with some significant drawbacks. One issue is that LabVIEW flattened data can be read by LabVIEW only, unless you use our documentation and write your own code in a different programming language. Another issue is version control - sooner or later you might want to add something to your data structure and all your client applications will have to maintain backwards compatibility to all earlier versions you have stored.

Using a DBMS is worthwhile if you need access for multiple users [writing to the database at the same time] and/or high-speed lookups in large numbers of datasets. This thread is about multi-user access, but with only one user writing. This can be done with a simple file. It can also be done using a database, but that means a lot of extra work.

With all that said, I'm currently investigating the problems that JC has described. It appears that there is an issue with concurrent access to TDMS files. This issue is specific to TDMS and does not occur with other LabVIEW File I/O. I'll let you know when I have figured out the details.

Herbert

[JCNT]

Thank you very much my friends for all your help which is useful, I will be waiting for your reply.

JC