First of all, if it lights up, then it was encoded. It's determining whether or not to decode HDCD (the same condition that causes the light to light up) directly from the digital audio stream (the bottom-most bit, to be specific). There are a variety of encodings, that might be what is throwing you.
Secondly, are you sure the "doing the opposite" thing isn't an indication of a bug -- specifically, what I interpret that to mean is that when you check 'use 24-bit-audio', you're enabling upsampling, and it might be upsampling the undecoded stream (as many DACs did, back in the day, which is why I always wanted a digital means of decoding the bitstream, like the Pacific Microsonics Model One or Two), hence destroying the encoding bit pattern? In other words, how are you confirming that the decoded stream is decoded when it's supposed to be and not when it's not (and please don't tell me, 'by ear')?