LabVIEW
[Code review] Reordering an heavy array of cluster
Solved!
It is almost a copy/paste from the example of your library :
I also tried with the "Execute SQL (Poly).vi" directly, but it is nearly 2x slower than the example :
No one expects a short-term solution from me on this matter. It's more of a challenge I set for myself.
So don't trouble yourself during your vacation if you don't want to. 😉
@PinguX wrote:
@PhillipDBrooks a écrit :
if you have large arrays, using external tools to perform the sorting and filtering is going to incur memory copies between LabVIEW and SQLite, Python, etc.
Maybe it would make sense to load the data once into something like the LabVIEW Dot net Data Grid
https://github.com/unipsycho/LabVIEWdotNetDataGrid
Thanks for the suggestion.
It would have made sense if the data destination was a MCLB, and if I was allowed to include .NET in the software.
Unfortunately, it is not the case for both of them.
Great project, though !
Does it have a good support for large table ? I know that natives compenents (MCLB/Tree/etc) have a long delay when we put lot of rows in them.
Bit of an aside but I have a Datagrid control that uses pure LabVIEW:
https://github.com/Dynamic-Structures-and-Materials/D-Datagrid
I haven't tested it with enormous amounts of data as my focus was more on the UI side of things, but it's dynamically organizable (which means you don't have to actually load the whole thing) and it supports custom object type overrides (which means you can have an element store its data in a DVR or even an external database!)
I played a bit with your database using DB Browser for SQLite to isolate the LabVIEW vs SQLite portions. (There does appear to be some difference though, as LabVIEW takes ~650 ms on my machine, and the same query on the DB Browser takes 1500 ms.)
Still, some notes:
Creating an Index on "name" gave me a 3x performance boost in DB Browser, from 1500 ms to 419 ms. However, adding the index nearly doubled my LabVIEW time from 650 ms to 950 ms. Deleting the index dropped my LabVIEW time down to 650 ms again. Perhaps Dr. Powell can elaborate on that one, as it's unexpected to me that 1- the two methods are different and 2- an index slows it down.
Using EXPLAIN QUERY PLAN showed that adding the Index on (name) made the sort operation use the index, but without it, it used a "temp B tree". I would've suspected that this is because the index doesn't sort the output, but it's weird that DBBrowser and LabVIEW are so different in this regard.
@PinguX wrote:
@Andrey_Dmitriev a écrit :If by Large Table you mean a huge multi-column list box, then take a look at this comment: https://forums.ni.com/t5/LabVIEW/ListBox-Color-Change-Runtime-Differences/m-p/4482555#M1324615
Yeah I already use that in some places, loading in front panel only the N rows that are currently "selected" by an "external" scrollbar.
@Yamaeda a écrit :
If you change the handling of the data to DVR things are pretty fast.
Isn't it the same speed without DVR ? ~100/150ms for 512MB and 1'000'000 elements ?
@BertMcMahan a écrit :
Bit of an aside but I have a Datagrid control that uses pure LabVIEW:
https://github.com/Dynamic-Structures-and-Materials/D-Datagrid
I haven't tested it with enormous amounts of data as my focus was more on the UI side of things, but it's dynamically organizable (which means you don't have to actually load the whole thing) and it supports custom object type overrides (which means you can have an element store its data in a DVR or even an external database!)
After testing with "LabVIEWdotNetDataGrid", it turns out that that initial load takes ~4sec for 10k rows.
The features are excellent and practical, but the initialization step is very slow.
If your focus for "D-Datagrid" was about the UI/UX side, I guess it could be the same issue ?
On my machine, a 2-column x 10k row D-Datagrid loaded in about 2 seconds (note, I didn't do a lot of profiling here; all settings are "stock").
If you need something faster, you could easily modify the library to only load subsets at a time. It probably wouldn't gain you a lot if all you're doing is strings, but it might be helpful.
PS: Tested with some other sizes, too, all of which had 2 columns (a U32 and a String). 100 rows took ~110 ms. 1,000 rows took ~310 ms. 10,000 rows took ~2100 ms. 100,000 rows took 18 seconds.
I'm not sure exactly what the bottleneck is on speed, as writing to a plain Table is very quick. I can say that my original goal was to be able to save bit-perfect data within the table, regardless of how it's displayed, so it keeps a copy of all data behind the scenes, PLUS whatever's shown in the table itself. So, it's likely that it's copying data back and forth a bunch.
If I were to optimize for speed, I'd probably let it handle all of the data internally the same way it does now, but only show a subset in the real table. Add a fake scrollbar and display the subset of data you actually want to SHOW the user. It could be handled by the library itself, which would mean the user doesn't have to know about hiding/showing data.
