How would you do that?
Simple: I would not use the formula node. It is pointless.
The factorial function is not available directly, but is well defined, so you can roll your own. What is giving you problems? Note that I don't use the factorial function, because we need the intermediary results anyway. No need for these duplicate operations. Try to do a literal translation of my code into text instead.
(You also only need to do the calculation whenever one of the controls changes, and not millions of times per second as you currently do)