Acquire touch screen input as data for plot

[JamesTillotson]

Application Notes: Labview 2018SP1, Windows 10 (Windows 7 upgrade), AR1100 Resistive Touchscreen input, Touch-Base V6 Drivers.

LabVIEW Tools: Initialize Mouse, Initialize Joystick, Acquire Input Data, Query Input Devices, Input Device Control Pallet.

Hello All!

I just started a new job for a university mechanical engineering department and I've run into a bit of a challenge. I've been updating computers from windows 7 to windows 10 and one of the teaching lab LabVIEW controlled demonstrations didn't survive the leap.

The issue seems to be with the enumeration of a resistive touch screen membrane that is used to collect cam path data. This works by rotating a student designed cam that has a pin which contacts the membrane. The cam rotates and the motion trace on the touch screen is pulled in by the labVIEW code and maths happen to compute acceleration velocity, displacement and the sort.

The touch screen shows up in the windows 7 device manager as an HID compliant device and I can use the *initialize joystick vi.* To bring the data into the main VI. This results in a code which only responds to the input from the touchscreen and not the mouse.

In windows 10 it enumerates as both HID compliant touch screen and as a mouse. But it only seems to be using the mouse enumeration no matter what i do in the driver settings or in device management (so far). If I bring it into the main VI through the initialize mouse it works but the VI also tracks mouse movements which would make the data sets unreliable since it would include the cam path and mouse movements to interact with the VI. Also potentially the CAM motion would make it so the mouse is not usable or the pointer would be doing things on screen while the device is working interacting with stuff on the window instead of just providing data. 

I have been on the LabVIEW Forum and a few other sources as well as the driver manufacturer (who has been fantastic to work with so far) and I have not yet found a direction for a to start working out a solution.

My question is several parts.

1)

First has anyone come across a similar problem personally or in the forum? Are there resources I could read to understand the issue of touchscreen enumeration or in grabbing that data in labVIEW? I've been checking the LabVIEW user forums and have found some similar work but no similar problems so far.

2)

Next, I have been trying to understand how windows decides something should enumerate and how might I change this if there is a way.

3)

Finally, is there a better way to access the info coming from the touch screen. I'm going to investigate VISA and some other tools.

Attached are reduced codes that I am currently working with as well as screenshots of the front panel and block diagrams. I removed the other functionality and am focusing on reacquiring the touch screen coordinates.  Another thing I am excited to bring to the department is more maintainable codes. So forgive the silly button size and formatting. I have started cleaning things up and building sub VI's but this is only my 9th day on the job and a fella can only accomplish so much in that time.  The Joystick code is the original code which was working on windows 7 but on windows 10 produces an invalid device index error at the initialize joystick VI. I have incremented the device index from 0 to 100 in a brute force attempt to see if the index had changed in windows 10 with no luck. The mouse version of this code shows functionality that I want in the plot and data streams but has the mouse interaction issues that I described earlier. 

This is my first post here, thanks for all the great problem solving on this sub. I've enjoyed reading here for a long time I have grown a lot professionally and personally as a result of this community. I am looking forward to any advice you might have for me!

JT

[JamesTillotson]

Hello everyone,

Here is an update based on what I have been working on for the last few days. A part of the issue I have been having is related to the way that Windows 7 handled USB device data and enumeration, and the new way that windows 10 handles the same.

I have been working with some gentlemen at Touch-Base who build the windows driver for the touch screen I am using and they said the following which I found to be quite useful towards developing a basic understanding of what is happening between the data output of the touchscreen, and the device in the device manager. 

 ---Start Touch Base Correspondence ---

It sounds like in Windows 7 you have a HID compatible touch screen and this will be enumerated under “hid devices” (or a 3^rd party driver that achieves the same) .

You are apparently opening this hid device and reading data, but I can’t tell from your description if this is the USB data coming in from the hardware or the touch data going out to the OS.

In your W7 scenario this is academic as the data has the some format.

The UPDD architecture separates the input (hardware) and output (touch) data. There is a UPDD Devices device instance for the hardware an HID instance for touch and a mouse instance for the mouse interface. The HID and mouse are root enumerated devices (not directly associated with hardware) and thus permanently present.

You MIGHT be able to access the HID data as previously described, but this depends if you were accessing the USB data or Touch data. In the UPDD configuration the USB data is managed by a USB class device and as such you can access this via an HID interface.

Hope that helps; I’m simplifying quite a complex area of course.

 ---End Touch Base Correspondence ---

This leads me to 3 possible paths to take to make this hardware work.

1) Change the device enumeration through another driver or a setting. 

I am hoping to do here is either find a setting in the current driver which I don't think exists given my communications with touch base, or another driver which can alter the device enumeration so that it can enumerate as a controller device like a joystick rather than as a mouse or HID touch screen. This would enable me to access the the information using the joystick VI. 

2) Snag the data from the touch controller prior to the system level the way the driver does. 

This method can be broken into several sub methods

a) get the output from the device driver (this would be ideal as the data is already scaled and processed by the driver).

b) Write some code in python or another language that can extract the system mouse pointer data but still allow me to disable the touch mouse device in the device manager so it is not controlling the cursor while the cam data is being collected.

c) Grab the raw device data on the USB port prior to the driver, system enumeration, or any of it which I am unsure of how to do. I don't know that I understand the data format well enough yet and this method seems difficult, so this will require some google-foo. Heck I might even need to watch a youtube video or *gasp* read a book. 

3) Identify another hardware device that can track the spacial location of the cam pin.

Because Windows 10 enumerates touch screens as mice or multi touch devices, I will not be able to use an updated but similar hardware.

I am not exactly sure what I am even planning here yet but my initial ideas would include the following. 

1) Optical techniques using a camera or a set of optical sensors and lights.

2) Grab the data from the resistive touch device before it ends up on the USB terminal and fee that into lab view as serial or digital data  using an analog card in the cDAQ if possible. Again I am not sure what the output from the touch screen to the touch controller is, and I don't know what the output of the touch controller at the USB port is. 

That is where I am at so far. ONWARD!

[JamesTillotson]

Alrighty everyone, 

"If it is stupid but it works, its not stupid"

- James Tillotson

- Murphy S. Law (Circa 1908 Colorized)

So to give a quick TLDR of the above issue that quickly became issues (plural). 

1.  I am upgrading a Cam design teaching lab for a university Mechanical Engineering undergrad program. 
2. The old lab ran on windows 7 which permitted a touch screen membrane to be enumerated as a generic HID device without an actual monitor behind it so that LabVIEW can suck up the "cursor" data without actually controlling the cursor on the screen.
3. The Windows 10 upgrade killed the Microchip Driver that I had been using to connect to the Touch Membrane.
4. I got a windows 10 supported driver from the driver designer (Touch-Base, Awesome support, prompt and technically sufficient from those folks so far, just no solution from the driver side) that supports the Microchip AR1100 Touch Screen. The driver now allows the touch screen to enumerate in Windows 10 as a mouse and as a multi-point touch device but not a Generic HID like windows 7, Naturally windows 10 broke this feature.
5. If I use the mouse version of the code posted with the initial post above (or below this, or wherever it goes when I post this reply) my code works fine but it moves the mouse cursor around on the screen which opens, closes, resizes and controls stuff on the display this is naturally sub optimal and of course could result in the device closing the .
6. I got stuck here on Friday, I cried in my car for 30 minuets before leaving early to smoke some meat on the pellet smoker and forget my Windows 10 derived sorrows. 

Okay Here is where we pick up my adventure. I had a fever dream on Sunday night after hitting a few too many of those otter-pop iced popsicles with my 5 year-old that there might be a way to trick windows into thinking there was a third dummy display connected to the computer. I could then assign the touch membrane to this display, set the mouse cursor to return to previous point in the driver (this sets the touch device to return the pointer to the origin once the touch has stopped thus preventing the pointer from becoming lost in lala land of a spoofed display, and voilà, now the mouse can click away on a fake display and not open close or move anything that is important. 

Like I said that Murphy said, "If it is stupid but it works, Its not stupid"

The only sadness that I have bumped into so far is that once again windows 10 kicked me in the shin, resulting in more crying (man I am really getting soft in my old age). Back in the early days of W10, making a spoofed display was easy to do from the control panel It used to work like this (according to some forum in the underbelly of windows support. 

1. First go to Control Panel (not Settings app)
2. Go to Adjust Screen Resolution. You will get a similar window as you did in Windows 7.
3. Click Detect
4. Go to display Drop Down and select "Display Device on VGA"
5. Select Desired Resolution. If windows is unable to save your settings use the software from your video card to adjust the resolution.
6. If you aren't able to output the display to the Fake screen only, Press Windows+P and the select "Second display only"

But of course a windows update between 2013 and 2016 broke this feature (this seems to be a theme of my life at the moment) 

I have been unable to identify a software based workaround thus-far but I found a stupider-er solution that does work mostly flawlessly. But before I spill the beans I am going to start a new post so that someone doesn't need to read all my junk above to get to it. I also need to finish up some code so that it works well for other users in the future. 

Since this is the stuipder-er solution I will still be open to other more mature and elegant solutions. So if grinding your gears on this sort of malarkey is your thing don't feel like you should keep that to yourself. 

VR,

JT

[JamesTillotson · Accepted solution]

Here is my current solution.

1. Install current Touch-Base Drivers on Windows 10 Machine
2. Attach AR1100 Resistave touch device to USB port
3. Verify that the device enumerated as a mouse and as a HDI Touch device
4. Connect headless display dongle to an open display port. If you need to make one here is how I did it. 
   1. I jumped pin 1 to pin 6 with a 150 Ohm resistor on a VGA to HDMI dongle
   2. Plug the dongle into the HDMI port
   3. Profit (by which I mean observe a display in the display manager that is not actually there)
5.  Open the Touch-Base Driver Console and set up the touch device as follows. 
   1. Set the AR1100 Device to Mouse mode (single point touch)
   2. Set the Anchor Mouse option to on. So that the mouse cursor returns to its original position after the touch pointer device has been used. Normally used in multi-monitor configurations where the cursor is to return to another monitor.
6. I have one remaining task but I think it will be easy-ish (famous last words, I know I know). I still have the issue that the device registers all mouse activity, so what I need to do is write code that ignores any mouse inputs that happen in the coordinate range of the actual monitors and only passes inputs to the graph and computations if they occur on the coordinate range that the dummy display occupies. I also need to scale the outputs so that a cam moving the pin in, for example, a 4" circle displays a 4" circle on the graph rather than whatever it is doing now. 

 Once I have this working as a sub vi with all the fun-ness I will post it here as the solution unless i can edit a post that has been flagged as a solution to add the code. I will also post photos of my current and maybe future headless dongle as described above. For now this post is a pseudo solution.

Thanks again for reading this far, while I am bummed that no one else chimed in on the thread even to say "man that sucks", or "I sure am glad I am not you", Or "are you silly!" I have been having a good time working through this problem "out-loud".

At your service!
JT