Allow For Special Loop Error Shift Register on Non-Parallel For Loops

[Hooovahh]

On a For Loop that is configured to have parallel iterations there is a nifty feature when using errors on a shift register.  This feature is explained in an article here.  It basically ensures that For Loops that run 0 times, will preserve the incoming error as if the tunnel was a shift register.  However this feature also means that errors from iteration 0, won't be passed into iteration 1.  What is returned if the loop runs 0 times is just the incoming error.  But if it runs for more than that, the errors are all merged and returned will be the first error seen.

This idea is just to allow this already existing feature to work on non-parallel configured For Loops.  When would this be useful?  Say I want to delete an array of files.  I want to attempt to delete all files, even if there is an error in trying to delete some, just keep trying to delete the other files.  Initially you may think this:

But that has the clear problem that if the incoming array is empty, then the error will be cleared.  So we often do something like this.

And honestly this bit of code isn't that big of a deal, but this feature already exists.  However it only works on For Loops that are configured to run iterations in parallel.  In this case I might be attempting to delete files in a specific order due to their placement on disk, and running in parallel isn't what I want.

Since not everyone wants all errors in loops to act this way it would need to be a right click menu to turn the tunnel into a normal one, a shift registered one, or a preserve and merge one.

That is what I want.

[wiebe@CARYA]

It could also be helpful if the array is large. You'd be building a large array of error clusters, and that could be costly.

I've posted this solution somewhere else, in a very similar discussion in the idea exchange (although the idea might be different, I'll have a look):

[robdye]

Interesting idea.

The assumption when we made this feature was that allowing the identical wiring pattern wrt the shift/error register was paramount when configuring a regular For Loop to be a ParFor Loop (or the other direction). The "degraded" functionality of the ParFor's error behavior (i.e., errors in one iteration are independent of other iterations) was regarded as an acceptable trade-off for allowing the identical wiring.

Here you are saying that you'd like to have that "degraded" error handling functionality in both the parallel and sequential For Loop.

I have to admit that allowing that has a nice symmetry.

The editor currently automatically flips between the shift and error register behavior when flipping between sequential and ParFor behavior. With your suggestion, I presume we could keep that same automatic flip, until such time as you explicitly set the shift or error register to retain error register semantics regardless of the parallelism setting of the ParFor.

As always, we'll see how the conversation goes here on the forums. Kudos do get noticed.

[Hooovahh]

Thanks rob for that incite.  You are correct that there is a type of trade off, and that in a ParFor Loop there are less options, which is probably why the conversion on the error terminals is automatic, which I like.  On the normal For Loop I could see places where the developer might want any one of the 3 options. 

Pass in the error on a normal tunnel. This could be used in a retry mechanism where only the last iteration data is returned.  Continue while error could stop it early.

Pass it in on a normal shift register.  Zero iteration loop preserves data. Errors between loops can be handled or not.

Pass it in on the iteration error tunnel.  Continue looping and return the first error on the output.  We don't care about one iteration influencing the action of the next.

wiebe@CARYA Great suggestion I didn't really think about having a huge array of errors to merge.  Here is a modified version which would only return the iterations that had an error and then merge those.

The more I think about it the more I think maybe I'm just being lazy. There are multiple solutions today that can accomplish the same thing.  Just dealing with merging errors, and arrays of errors, isn't as clean as it is on the ParFor Loop.

[paul_cardinale]

I frequently want that.

I also want a way to convince my cow-orkers to actually think about error flow.

[wiebe@CARYA]

Wiring a 1 to P would work:

This gives us the fancy error shift register, but we can't use other shift registers. So it's at best a hack for simple loops.

[wiebe@CARYA]

The (absolutely fantastic) merge errors option "Retain all errors" might complicate implementation:

A tunnel should have this option.