[Code review] Reordering an heavy array of cluster

[Andrey_Dmitriev]

Another possible way to improve sorting performance is to use parallelism on multi‑core CPUs.

Not every sorting algorithm can be parallelized well, but merge sort is good‑suited for it. In purely sequential form, merge sort is typically slower than quicksort, but with enough CPU cores it becomes faster once the input size is large enough.

Sorting 10,000,000 integers on my i7‑4800MQ (2.7 GHz, 4 cores):

Optimization using SIMD (AVX2) instructions is also theoretically possible here.

[PinguX]

---

@Andrey_Dmitriev  a écrit :

Another possible way to improve sorting performance is to use parallelism on multi‑core CPUs.

---

I try to avoid parallelism for isolated operations, as they are not the only ones using the processor at any given time.

For example, I prefer a duration of 600 ms (single core) over 400 ms (two cores).

[PinguX]

---

@BertMcMahan  a écrit :

I'm not sure exactly what the bottleneck is on speed, as writing to a plain Table is very quick. I can say that my original goal was to be able to save bit-perfect data within the table, regardless of how it's displayed, so it keeps a copy of all data behind the scenes, PLUS whatever's shown in the table itself. So, it's likely that it's copying data back and forth a bunch.

If I were to optimize for speed, I'd probably let it handle all of the data internally the same way it does now, but only show a subset in the real table. Add a fake scrollbar and display the subset of data you actually want to SHOW the user. It could be handled by the library itself, which would mean the user doesn't have to know about hiding/showing data.

---

After several more tests, it seems the indexes on database columns are essential for optimizing queries using "LIMIT" and "OFFSET" parameters.

Even though this may reduce the performance of a simple "SELECT ORDER BY" query.

If you create indexes on "name" and "timestamp" from the previous database, you should be able to use the attached code.

It is quite simple and allows for the display of tables of virtually any size. Approx 1ms on my computer.

It doesn't respond to my original needs, but worth keeping in reserve.

[drjdpowell]

Yes, if you really need all the data then SQLite isn't going to help you.  Databases let you rapidly query large amounts of data for much smaller subsets of data that you actually need.  But you don't want a small subset. 

BTW, another technique to get a small subset (in addition to LIMIT and OFFSET) is to use GROUP BY.  This is useful for plotting data.  You might want to plot a signal with a million points, but that would be very slow.  Instead use AVG() and GROUP BY to get average signal in about 3000 bins, which is enough to plot and can be plotted quickly. 

[Andrey_Dmitriev]

---

@PinguX wrote:

---

@Andrey_Dmitriev  a écrit :

Another possible way to improve sorting performance is to use parallelism on multi‑core CPUs.

---

I try to avoid parallelism for isolated operations, as they are not the only ones using the processor at any given time.

For example, I prefer a duration of 600 ms (single core) over 400 ms (two cores).

---

Yes, agree, for short operations, multithreading often provides little benefit because the overhead of thread creation, scheduling, and synchronization can exceed the performance gain. To achieve meaningful speedup, the workload must be large enough with reasonable amount of data. For example, in this particular case when sorting 100,000,000 integer elements on a Xeon W5-2445, the execution time can be reduced from 13.2 seconds down to 2.3 seconds, yielding an approximately 6× speedup. In practice, high-level parallelization, such as sorting multiple arrays concurrently and simultaneously, is often more effective. For lower-level parallelization, SIMD instructions can provide additional and better acceleration, although efficient SIMD implementations of sorting algorithms are considerably more complex.