I am not an expert at motor balancing but I would like to through out a thought.
Can't this be handled like "quadraphonic stero" and the like?
Use the two signals A and B to create to additional signals "A+B" and "A-B".
In perfect balance all 4 would be zero.
If out at A end, the "A", "A+B", and "A-B" would be non-zero, but "B" would be zero.
Similarly for "B" end.
If "A" and "B" where equal and oposite all signals would have magnitude except "A+B" and "A-B" would be largest.
If Equal and in-phase "A+B" would be largest.
So if you give each of teh above sets of values a direction similar to,
A Plus X direction
B Minus X direction
A+B Plus Y direction
A-B Minus Y direction
You could add them together vectorally and the resulting vector wou
ld indicate the magnitude of the imbalance and the phase.
You may have to set up your sensors such that they measure forces that are perpendicular to each other.
Just my thoughts,
Ben