I've experienced the Front Assist kicking in unexpectedly a couple of times. Each time, like you, I had noted a potential hazard but concluded there was no risk so carried on, the car clearly thought differently. On each occasion I didn't need to lift off, brake, steer as no 'driver input' was needed.
However, in similar circumstances more recently, I have tweaked the steering slightly &/or lifted off slightly and the car Front Assist has not reacted - I assume those small driver inputs were enough to inform the systems that I've seen it, I'm reacting to it, I know what I'm doing.