For single-ended, because of the fully differential input stage, L- and R- need to be grounded when using the SE input, and wired normally when using balanced input. You would need some kind of switch to do this. It had been suggested that a 4PDT switch be used, to switch all 4 signals between the RCA/XLR, and to ground the - signals when the RCAs are being used.
I'm not sure if you could get away with just using a DPDT to switch the - either to amp ground when using the RCA or to the XLRs in balanced mode. This would leave the + inputs connected to both the RCA and the XLR at all times.