Digital Light & Color

Have only tried this with Adjust and Multi Color Balance: Open transformation, start a new amount mask, select mask tool, draw shape (or gradient line...) then minimize the transformation window - mask selection outline disappears, and re-appears when transformation window is restored. (This is prior to Applying the mask)

Update: just tried this with Sharpen - no problems (selection does not disappear...). Also noticed a difference between transformations in how the minimized window headers appear, some or overlapped (and need to be restored in proper sequence, or else the transformation window is overtop of the mask window header...).

Tried with a handful of other transformations (Blur, Brightness, Color Curves, High Contrast, Equalize...) and they are all fine.

Actually, more transformations should work the way Multi Color Balance does - when you click on its title bar to minimize it, it activates its probe and de-activates Mask's probe and overlay. When you click on the Mask title bar, it does the reverse. For transformations without a probe or overlay, the Mask overlay remains in place.

Okay, my bad. It simply requires clicking on the mask dialog window to keep the selection in focus. My mistake was minimizing the transformation window, then trying to continue with the mask operation without re-focusing the mask dialog.