That is quite a job. I don't think I am qualified to recommend a "best" architecture for it.
Producer/consumer is designed to manage parallel functions that have widely different rates of function, as in a UI vs. program functions operating at computer speeds. The UI produces requests to or data for the consumer which may or may not come slow enough that the consumer can keep up, or may be very slow in comparison and the consumer may be performing functions that are transparent to the user, or it may be mixed.
I keep returning to the producer consumer architecture because it can reproduce the functions of a flow chart. If I can describe the decisions to be made and identify when inputs are used, and outputs produced
, I can usually get a good results from a producer consumer.
However, it may not the the best for your application, but it is certainly worth knowing. Producer consumer might be great for a particular module of what you are doing, but it may not make a good overseer program. I am scared of the "highly self-configuring" comment.
Good luck,
Mike