WARNING: This is one boring-ass blog post! but it may just help a couple of people out if anyone could actually find it.
I'm lucky enough to be working on a multiple rack, high channel count oscilloscope project at the moment and I thought I would tell the story of how we got the various chassis synchronised to <1ns.
The requirement is pretty standard IMO, I want all the cards on all the chassis to trigger at the same time. The trigger signal can go to each of the chassis but they need to be synchronised and in phase.
When I saw the hardware list (8x 18 slot PXI chassis, full of oscilloscopes and each having a timing and sync card) I thought this is the equipment for the job.
Right tools for the job and a standard requirement, what could possibly go wrong? This was my 1st mistake. It was really hard!
In the end I had to lean on Sacha Emery, one of the patient and knowledgable System Engineers employed by NI and he sweated blood on this job too, there just didn't seem to be much information out there.
Anyways here is the solution and I hope is saves someone some time.
To test the system I send a pulse and read it on channels across the backplane of the PXI rack. We need to use the OCXO (Oven controlled crystal oscillator) as our clock and export this clock to all chassis, we need also need ensure the scopes start at the same time and are linked to this clock on the backplane of the rack. Additionally we need to make sure the clock etc go across all of the racks backplanes (18 slotters are split in 3 for flexibility).
Step 1 Route Master Clock
Route the OCXO from the PXIe-6674T on the master chassis to the CLKOut connector on the PXIe-6674T.
Gotcha: Dev1 is the 6674T in my system and Oscillator is OCXO, I didn't find any documentation that stated that this is so, but it is!.... apparently.
Step 2 Master Initial NI-Sync Settings
Turn on the PLL circuit and set the frequency to lock. To cope with the splitting of PFI signals we need to disable ClkIn Attenuation and reduce their threshold to 0.5V (or lower).
Step 3 Master Route Signals
To remove propagation delays due to cabling we bring the clock back in again from the CLKOut terminal via a matching length lead (matching to the slaves). This is routed to the 10MHz backplane clock.
The start trigger is routed out to PFI0, this is the signal to the other chassis to tell them to start logging data (for pre-sampling). This does not need to be matched in length.
We then use the Global Software Trigger as our Sync Pulse. Because this needs to compensate for lead length propagation delays it is output on PFI4 and routed back on PFI2.
Step 4 Set-up Slaves
The slaves should be set-up as follows and should be waiting for the PLL to lock prior to the master.
Step 5 Master and Slave Wait for PLL Lock
Pause and loop, waiting for PLL to lock on both master and slaves.
Step 6 Set-up Master Acquire
Card2 in each rack is designated as the Master Card and by this we mean the card that creates the trigger for the rack. The start trigger is routed here to the PXI Trigger line on the PXI backplane and then out of PFI0. The Sync Pulse is also routed from PXI_Trig2 so that NI-Tclk is synchronised across the backplane for all cards in the rack.
For cards that are not the trigger card the following code is run.
Step 7 Set-up Slave Acquire
Here the Start Trigger is sources from the VAL_RTSI_0, this is yet another name for the PXI backplane (PXI_Trig0). The sync pulse is routed from the backplane PXI_Trig2 as on the master chassis.
The other cards are set up as follows.
Step 8 Master Wait for Send Sync
All the NI-TClk references are collected and configures the properties commonly required for the TClk synchronization of device sessions and prepares the Sync Pulse Sender for synchronization. It's then held up waiting for the Slave to be in a state ready for the next stage.
The slave is set in exactly the same way except it has to be done after the Master.
Step 9 Slave Ready for Trigger
Step 10 Master Wait for Send Start Trigger
The first VI sends a pulse on the global software trigger terminal. Then we finalise the NI-TClk synchronising and wait to send the start trigger. When the button is pressed the start trigger is sent and the whole system is ready to acquire data.
Step 11 Acquire
The test was to send a pulse to Chan 0 on the first and last cards in 2 racks. The first channel on each rack will be used as the trigger source for the rack and the racks will be synchronised. The cabling has been carefully balanced to minimise propagation delays in the leads.
The test was done at 1.25GSamples/sec and 125000 samples were taken for each channel.
We then combined the graphs and compared the readings.
From this we can see a delay caused by the triggering system (essentially the detecting of the trigger and passing it to the backplane). This is <1ns and only applies to the trigger channel and can be post-processed out if required.
The synchronisation of Master Card 18 Chan 0 and Slave Card 14 Chan 0 is representative of all the other cards and channels in the system and should scale up to 8 racks (the trigger signals will be reduced by splitting them). The results here are 16ps which is astonishing!
The code and drawing is attached.
When I have completed the system I will probably post the code on-line, because I think there should be a ready made LabVIEW system available for this type of application (personally if I was buying $500k of hardware I would expect something to be available)
There is no way in hell I would have worked this out without the help of system engineering!
I think system engineering need better access to high value systems, because it felt like we were treading virgin territory here.
The support I received has been exceptional and should be something NI is proud of.
I promised I would write all this up as a response to a query on the forums.
From this point I need to package all this up as part of my Client-Server architecture and the jobs should be a good-un.