Random Ramblings on LabVIEW Design

Community Browser
Labels
cancel
Showing results for 
Search instead for 
Did you mean: 

Re: Synchronizing Multiple Chassis

swatts
Active Participant

WARNING: This is one boring-ass blog post! but it may just help a couple of people out if anyone could actually find it.

I'm lucky enough to be working on a multiple rack, high channel count oscilloscope project at the moment and I thought I would tell the story of how we got the various chassis synchronised to <1ns.

The requirement is pretty standard IMO, I want all the cards on all the chassis to trigger at the same time. The trigger signal can go to each of the chassis but they need to be synchronised and in phase.

When I saw the hardware list (8x 18 slot PXI chassis, full of oscilloscopes and each having a timing and sync card) I thought this is the equipment for the job.

Right tools for the job and a standard requirement, what could possibly go wrong? This was my 1st mistake. It was really hard!

In the end I had to lean on Sacha Emery, one of the patient and knowledgable System Engineers employed by NI and he sweated blood on this job too, there just didn't seem to be much information out there.

Anyways here is the solution and I hope is saves someone some time.

System Diagram

ChassisWiring.png

To test the system I send a pulse and read it on channels across the backplane of the PXI rack. We need to use the OCXO (Oven controlled crystal oscillator) as our clock and export this clock to all chassis, we need also need ensure the scopes start at the same time and are linked to this clock on the backplane of the rack. Additionally we need to make sure the clock etc go across all of the racks backplanes (18 slotters are split in 3 for flexibility).

Step 1 Route Master Clock

Route the OCXO from the PXIe-6674T on the master chassis to the CLKOut connector on the  PXIe-6674T.

MasterRouteOCXO.png

Gotcha: Dev1 is the 6674T in my system and Oscillator is OCXO, I didn't find any documentation that stated that this is so, but it is!.... apparently.

Step 2 Master Initial NI-Sync Settings

Turn on the PLL circuit and set the frequency to lock. To cope with the splitting of PFI signals we need to disable ClkIn Attenuation and reduce their threshold to 0.5V (or lower).

MasterInitialNISyncSettings.png

Step 3 Master Route Signals

To remove propagation delays due to cabling we bring the clock back in again from the CLKOut terminal via a matching length lead (matching to the slaves). This is routed to the 10MHz backplane clock.

The start trigger is routed out to PFI0, this is the signal to the other chassis to tell them to start logging data (for pre-sampling). This does not need to be matched in length.

We then use the Global Software Trigger as our Sync Pulse. Because this needs to compensate for lead length propagation delays it is output on PFI4 and routed back on PFI2.

MasterRouteClocks.png

Step 4 Set-up Slaves

The slaves should be set-up as follows and should be waiting for the PLL to lock prior to the master.

SlaveSetup.png

Step 5 Master and Slave Wait for PLL Lock

Pause and loop, waiting for PLL to lock on both master and slaves.

WaitForPLL.png

Step 6 Set-up Master Acquire

Card2 in each rack is designated as the Master Card and by this we mean the card that creates the trigger for the rack. The start trigger is routed here to the PXI Trigger line on the PXI backplane and then out of PFI0. The Sync Pulse is also routed from PXI_Trig2 so that NI-Tclk is synchronised across the backplane for all cards in the rack.

MasterAcquireMasterCard.png

For cards that are not the trigger card the following code is run.

MasterAcquireSlaveCard.png

Step 7 Set-up Slave Acquire

Here the Start Trigger is sources from the VAL_RTSI_0, this is yet another name for the PXI backplane (PXI_Trig0). The sync pulse is routed from the backplane PXI_Trig2 as on the master chassis.

SlaveAcquireMasterCard.png

The other cards are set up as follows.

SlaveAcquireSlaveCard.png

Step 8 Master Wait for Send Sync

All the NI-TClk references are collected and configures the properties commonly required for the TClk synchronization of device sessions and prepares the Sync Pulse Sender for synchronization. It's then held up waiting for the Slave to be in a state ready for the next stage.

MasterWaitForSendSync.png

The slave is set in exactly the same way except it has to be done after the Master.

Step 9 Slave Ready for Trigger

SlaveReadyForTrigger.png

Step 10 Master Wait for Send Start Trigger

The first VI sends a pulse on the global software trigger terminal. Then we finalise the NI-TClk synchronising and wait to send the start trigger. When the button is pressed the start trigger is sent and the whole system is ready to acquire data.

MasterWaitForSendStartTrigger.png

Step 11 Acquire

The test was to send a pulse to Chan 0 on the first and last cards in 2 racks. The first channel on each rack will be used as the trigger source for the rack and the racks will be synchronised. The cabling has been carefully balanced to minimise propagation delays in the leads.

The test was done at 1.25GSamples/sec and 125000 samples were taken for each channel.

We then combined the graphs and compared the readings.

Results.png

From this we can see a delay caused by the triggering system (essentially the detecting of the trigger and passing it to the backplane). This is <1ns and only applies to the trigger channel and can be post-processed out if required.

The synchronisation of Master Card 18 Chan 0 and Slave Card 14 Chan 0 is representative of all the other cards and channels in the system and should scale up to 8 racks (the trigger signals will be reduced by splitting them). The results here are 16ps which is astonishing!

The code and drawing is attached.

When I have completed the system I will probably post the code on-line, because I think there should be a ready made LabVIEW system available for this type of application (personally if I was buying $500k of hardware I would expect something to be available)

Lessons learnt: 

  1. There is no way in hell I would have worked this out without the help of system engineering!
  2. I think system engineering need better access to high value systems, because it felt like we were treading virgin territory here.
  3. The support I received has been exceptional and should be something NI is proud of.

I promised I would write all this up as a response to a query on the forums.

From this point I need to package all this up as part of my Client-Server architecture and the jobs should be a good-un.

Hope this helps somebody in a similar situation

lots of love

Steve

Added Backplane routing stuff here

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

Comments
swatts
Active Participant

Oops I forgot that we set this up in MAX to route the backplane PXI trigger lines

Master Routing

MAX TriggerBus Routing Master.png

Slave Routing

MAX TriggerBus Routing Slave.png

I need to work out how to do this programmatically ideally, but the VIs aren't very obvious

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

Thoric
Trusted Enthusiast

HUGE Kudos for posting this Steve! As unlikely as most of us are to be working on such enormous systems, this will most certainly help someone and costly info like this is rarely available for nothing. A huge contribution from you Steve worthy of much appreciation!

Thoric (CLA, CLED, CTD and LabVIEW Champion)


Sash
Member

Hi Steve, nice write up.

For the trigger lines you can find that here : http://digital.ni.com/public.nsf/allkb/A4DA71CA2F68306686256ED8005515ED

I suggest with an RT controller use a visa resource find for backplane to get the correct reference rather than a constant as that changes if you do a copy paste on a full deployment system.

Cheers.

swatts
Active Participant

Thanks Rich I appreciate that.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

swatts
Active Participant

Cheers Mate and thanks for your help and by help I mean doing all of the work.

I've run it with digital triggering now and it all works nicely. I like the idea of routing the trigger lines in software.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

Sash
Member

Always a pleasure, never a chore.

Elijah_K
Active Participant

Thank you for posting this Steve. Very insightful.

Elijah Kerry
NI Director, Software Community
AaronTheAE
NI Employee (retired)

For Step #1: The user manual of the 6674T lets you know its an OCXO (pg.3-10: http://www.ni.com/pdf/manuals/373089c.pdf)

Cheers!

P.S. Thanks for all the help Steve

Aaron L.
Applications Engineer
National Instruments
swatts
Active Participant

psh no-one reads manuals,

I still can't see the explicit reference that "Oscillator"="OCXO" from the routing dropdown in the NI-Sync function.

And it's a pleasure!

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

Sash
Member

You're right Steve - it doesn't explicitly say to choose "oscillator" to pick the ocxo 10MHz clock on the 6674T. At least it's documented here now though!!