File format choice

[crossrulz]

---

@JÞB wrote:

JSON also has good support but suffers from the DOM standard and W3C won't / can't access it in a theradsafe manner.

---

DOM is for XML, not JSON.  JSON is very much accepted and there are libraries all over the place for every language.

I think I would struggle with JSON in this case is the streaming aspect since I'm left to assume you will be writing to several "groups" at the same time, which really does not bode well for text-based files.  TDMS handles this in the binary by using another file to mark the "index" of the data.  You can then use a function to defrag the TDMS file and make it smaller by cleaning up the file indexes.

So if JSON or the like is the desired, maybe a solution would be to write to a TMDS file and then take that data and convert it to the JSON format, being able to write an entire "group" at once

[lucian.grec]

I will have all the data available at the time of the write, e.g. after a certain test step finished executing. The only case I see where I would need an index is if I want to append these files instead of always creating a new one. In that case I suppose I'll have to decide what's constant(some sort of header like equivalent for a json tree) and what needs appending but I haven't given this scenario too much thought yet.

[johntrich1971]

---

@crossrulz wrote:

---

@JÞB wrote:

JSON also has good support but suffers from the DOM standard and W3C won't / can't access it in a theradsafe manner.

---

DOM is for XML, not JSON.  JSON is very much accepted and there are libraries all over the place for every language.

I think I would struggle with JSON in this case is the streaming aspect since I'm left to assume you will be writing to several "groups" at the same time, which really does not bode well for text-based files.  TDMS handles this in the binary by using another file to mark the "index" of the data.  You can then use a function to defrag the TDMS file and make it smaller by cleaning up the file indexes.

So if JSON or the like is the desired, maybe a solution would be to write to a TMDS file and then take that data and convert it to the JSON format, being able to write an entire "group" at once

---

I like this approach. I combines the speed and ease of TDMS with the widespread use of JSON. When I first read the OP I had TDMS in mind, but then changed my mind as I considered the need for compatibility with other software. JSON was my next choice, but as you noted the streaming text made the writing complicated. Your approach makes the streaming and the writing of JSON easy. Kudos

[Mark_Yedinak]

The OP also said the data was stored in a DB. Another option is to create a simple utility to pull the data from the DB, format the data and save to file. This utility could support multiple file types such as TDMS or JSON. Once it is in the DB I am not sure I se the need to also save all of the data to file as well. Unless every file will be used, simply save the data which is required for further analysis to file rather than all data.

[lucian.grec]

There is an option to save to DB as well but it's up to the user

[LLindenbauer]

WATS has published standards for XML and JSON that they can import. Both seem to be excellent choices for the requirements.

If it turns out that data interchange with machines is the dominant focus, I would consider going with XML and offering a schema for data validation. This can help in keeping track of different interface versions - your starting post reads to me as if it is subject to future change.

For human-readablity and non-programmer-friendliness it comes down to predicted usage. Prettified JSON is better suited for copy-pasting parts into excellent spreadsheet software (i'm thinking of your time series here) whereas well-chosen XML names can help users make sense of the data structure if they have to dig in without the manual.

While file size was not a primary concern, zipping does wonders to text files. "A zipped bunch of XML" is the current de-facto standard for office documents, after all.

[lucian.grec]

I have already prototyped some code for Wats reporting but Wats is missing some analysis features that we'd like and I'm not sure I want to comply with their JSON Schema because of that. To be honest I haven't tried the JSON report, I've only implemented a crude restAPI POST for the XML report and there were a lot of rules(compared to their client) that I had to keep in mind building the string to get an HTTP 200 OK. I suppose they can change this schema at any time and then we'll have to go and update our code/test/release to still comply. On top of that there is a lot of bloat in the schema that doesn't necessarily make sense to me so it'd make reading the file harder for whoever doesn't know Wats.
Anyway it's still something we're considering.

[lucian.grec]

I'm also thinking what would be a good way to type-wise future-proof my code. E.g. we might want to read old versions of this file into newer versions of the software. I don't want to have to carry over countless object versions and VIs, so maybe there's a way to address smaller chunks of this file using a path system and build my structures this way. I've seen drdjdpowell's jsontext does this. The labview XML primitive does this as well. Not sure if there's a better solution to this than what I'm thinking about.