Random Ramblings on LabVIEW Design

Community Browser
Labels
cancel
Showing results for 
Search instead for 
Did you mean: 

Re: Gateways - A Design Concept

swatts
Active Participant

Hello Error Destroyers

When writing software you often have to take data from a human and convert, parse, transmit or store it on a machine. It often transpires that humans are really incompatible with computers! They are flakey, vague, illiterate, moody, impatient, bored, rushed and/or inaccurate.

 

You need to bear this in mind when taking data from them.

 

This example is how most software I have seen looks. Quite a lot of what I've written too!

 Unsafe.png

What's wrong with that I hear you ask?

Well try entering O'Reilly into an SQL Statement or a SELECT user query with ' OR 1=1 as its recipient.

Or you could offer Users the freedom to set their own channel names and they include control characters, an action that will destroy TDMS files (a bug that took me a LONG time to find and fix!)

 

Here's some example of possible recipients of dangerous data.

  • Storing data in databases and SQL
  • Storing data in formatted files (TDMS, JSON, HTML, XML)
  • Transmitting data (UDP, Serial (XON/XOFF))

 

Here's what we need to consider when dealing with human enter data.

 

Gatewayed.png

 I've tried various methods for policing and making data safe. You can police at the dialog/data entry stage.

PolicingDataEntry.png

The vi is called  "Is Name Multiplatform.vi" and can be found in ..\vi.lib\Utility\libraryn.llb. Check out Darren Nattingers Weekly Nugget.

 

Another way that we often use is to make it safe before it is used, as below.

SafeInAction.png

The innards of which looks like this.

SafeInAction.png

Obviously it will need making unsafe by return, this is just the reverse of the above.

This is especially useful when users like apostrophes and single speech marks, SQL hates them!

The moral of this story therefore is if you have human entry whack in a post-condition check after entry or a pre-condition check prior to entry. It will save you some headaches!

Lots of Love

Steve

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

Comments
joerg.hampel
Active Participant

Spot on as always.

 

We just had a discussion today on which configuration storage mechanisms (.cvs, .json, database, ...) offer which pros and cons. One aspect is how resilient each of the options is to the possibly user-supplied dangerous data...




DSH Pragmatic Software Development Workshops (Fab, Steve, Brian and me)
Release Automation Tools for LabVIEW (CI/CD integration with LabVIEW)
HSE Discord Server (Discuss our free and commercial tools and services)
DQMH® (The Future of Team-Based LabVIEW Development)


swatts
Active Participant

We use SQLite now and have for a few years over lots of projects, never had a single issue and you can easily add Calibration, User Details, Experiments, Display settings if required. Highly recommend it (and James Powells SQLite toolkit too)

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

joerg.hampel
Active Participant

Same here, both SQLite in general and James' tool in particular are very welcome tools to work with in la casa del hampel.




DSH Pragmatic Software Development Workshops (Fab, Steve, Brian and me)
Release Automation Tools for LabVIEW (CI/CD integration with LabVIEW)
HSE Discord Server (Discuss our free and commercial tools and services)
DQMH® (The Future of Team-Based LabVIEW Development)


crossrulz
Knight of NI

Back in high school (learning GW Basic at the time), I was lucky enough to have a teacher who pounded this concept into our brains.  "You expect a number.  What if the user puts in 'A'?"  Modern tools do take care of a lot of this mess for us, but we do need to watch out for our corner cases.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
StefanR
NI Employee (retired)

Well, I know this being the LabVIEW board, but...

... there already exists a fully automated solution for cleansing and enriching incoming data, thus making that data suitable and usable for subsequent analysis tasks: NI's new Data Preprocessor

Data Preprocessing and Standardization

The preprocessing routine standardizes your metadata, units, and statistical calculations so that consistent analysis can be applied regardless of data source and format.

Set up your own preprocessing routine

 

...a part of DataFinder Server.

PhillipBrooks
Active Participant
@swatts wrote:

Hello Error Destroyers

When writing software you often have to take data from a human and convert, parse, transmit or store it on a machine. It often transpires that humans are really incompatible with computers! They are flakey, vague, illiterate, moody, impatient, bored, rushed and/or inaccurate.

 


Obligatory xkcd reference:

 

exploits_of_a_mom


Now is the right time to use %^<%Y-%m-%dT%H:%M:%S%3uZ>T
If you don't hate time zones, you're not a real programmer.

"You are what you don't automate"
Inplaceness is synonymous with insidiousness

swatts
Active Participant

That is just perfect Phillip!

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

CharlieRodway
Active Participant

Reminds me of an old tester joke:

 

A test engineer walks into a bar and orders a beer.


Then he orders 0 beers.

Then he orders 999999999999 beers.

Then he orders a lizard.

Then he orders -1 beers.

Then he orders NULL beers.

Then he orders asnwikfjsdf.

Charlie Rodway | Principal Software Engineer | Certified TestStand Architect (CTA)

Computer Controlled Solutions Ltd | NI Silver Alliance Partner | GDevCon#1 Sponsor

swatts
Active Participant

I have a short article floating around my head titled "Hit any button to continue" that follows a similar path. From the olden days when computers had pause buttons.

http://www.hpmuseum.net/exhibit.php?hwimg=120

HP.png

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

crossrulz
Knight of NI

Where is the "any" button?  I never could find it...


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
joerg.hampel
Active Participant



DSH Pragmatic Software Development Workshops (Fab, Steve, Brian and me)
Release Automation Tools for LabVIEW (CI/CD integration with LabVIEW)
HSE Discord Server (Discuss our free and commercial tools and services)
DQMH® (The Future of Team-Based LabVIEW Development)


swatts
Active Participant

The other issue was calling it a "button" rather than a "key", after several iterations the UI said

 

"Hit Space Bar to Continue"

 

And I told the smart-ass to sod off.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

drjdpowell
Trusted Enthusiast

Someone may have already mentioned this, but this is an issue with constructing a string containing both text and control characters that are later parsed.   You can often avoid much of the problem if there is something in the API you are using that identifies a string as just being text.  So in SQLite, one can “Bind” a string as text, and you can include any characters without issue.  Or in a JSON API one can pass it a string and it will properly quote and escape it for you.

swatts
Active Participant

Exactly right and a good reason to use your SQLite toolkit (and presumably your new JSON one too). One thing about pulling it out separately and making it safe (that I like), is that I like to see the SQL statements in my code (as documentation as much as anything else). 

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

Swinders
Member

I've used the method of replacing 'special' characters in a text input as in your last example and this worked very well when used inside a single application that always used the same 'make safe' and read back functions. As you say people like using the single quote or apostrophe and SQL doesn't understand it's not the end of a string. The main problem I have come across when doing this is trying to provide a nice formatted dump of the data to an excel sheet required some funny replace commands in the SQL query. Messy but it worked.

 

When using SQL I think there is a way to use parameterized queries and pass the string straight through without having to worry about escaping the single quotes but this sometime tricky depending on the SQL library used.

 

Validating user input is always fun but it's not just to prevent SQL injections, what if a sub VI is passed data that is not what is expected. Guess that would be another post.

Steve Swindley (CLA since Feb 2008)
swatts
Active Participant

My make safe VI also has a make unsafe command to convert back again for display purposes (and for comparisons, it's good to compare against the real data and not "safe" data).

Our original LCOD specification had Pre-condition checks on Inputs and Post-condition checks on exit. I've proposed before that you could set these up as part of the VI inputs and Outputs. Not entirely sure it's a good idea, and we jettisoned it from the LCOD definition because we never did it.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

drjdpowell
Trusted Enthusiast

I would strongly recommend people always use parametrized SQL where at all possible.  With a decent library it should be no more work than the equivalent formatting.  Visually, you would just be replacing your various '%s' formatting codes in your SQL with ?n parameter identifiers.

swatts
Active Participant

I concur, I just need to practice what I preach now!

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

PhillipBrooks
Active Participant

If you would rather skip the whole SQL command construction process, use LabVIEW's strong data typing.

 

I shared this LabVIEW scripting vi some time ago to create a cluster from a database table (using the DB Toolkit). Save the cluster as a typedef and then let LabVIEW do the work.  I know that database structures change, but if they were designed and normalized properly in the beginning, it should be mostly added tables, not restructured. 

 

https://forums.ni.com/t5/LabVIEW-APIs-Documents/DB-Tools-Create-Cluster-from-Table-vi/ta-p/3516670

 

 


Now is the right time to use %^<%Y-%m-%dT%H:%M:%S%3uZ>T
If you don't hate time zones, you're not a real programmer.

"You are what you don't automate"
Inplaceness is synonymous with insidiousness