Did more investigation today and I think I’ve figured out where the problem is.
I created a simple Android app and captured all config changes, compared them one by one and found out when my game controller is connected a “navigation” config change is triggered (changed to NAVIGATION_DPAD). Not sure why it’s recognized as a d-pad device, but does it make sense to capture this event so game won’t be reset?