I created a dll by labview, but I don't know how to debug it. The dll will be called by teststand, how can I go into the labview code in debug mode?
There is no way to debug a LV dll called within TestStand.
But I'm wondering why you don't call the LabVIEW code directly via the LabVIEW Adapter of TestStand.
I had the same issue as gavin, until I got it working as follows: This is using TestStand 4.2.1 with Patch 236982 for compatibility with LabVIEW 2010 and LabVIEW 2010 SP1.
Open the TestStand sequence that calls the LabVIEW-compiled DLL with TestStand Sequence Editor. I have that Step Type set to C/C++ DLL because for all effective purposes, from the outside perspective it is just a __stdcall DLL. There may be a better way to do this, I am not a TestStand guru.
Open the .lvproj LabVIEW project for the DLL with LabVIEW Project Explorer. Make sure under Build Specifications -> MyLibrary.dll Target -> Properties you have Advanced -> "Enable debugging" checked. Also I'm not sure if it matters for this approach but I generally also uncheck "Remove block diagram" from the VIs I want to debug under Source File Settings.
Build a fresh debuggable DLL. Make sure that TestStand is also pointed to and using that specific DLL that the project just built. Run the TestStand Sequence Editor up to the point where the DLL is called, setting a breakpoint there first, otherwise the dialog in the next step say "Failed to Connect".
In Project Explorer at the menu bar choose Operate -> Debug Application or Shared Library. I went with the defaults: localhost and SeqEdit.exe - MyLibrary.dll and clicked 'Connect'. If successful, ALL of the VIs for the exported functions in the DLL will open. You can then Show Block Diagram, step through, set breakpoints, etc.. from there while the sequence is suspended on your DLL call.
There are many reasons to build and use code as a DLL, and I won't go into them here. I just hope this helps anyone else in this situation.