Well, I think things are a bit more complicated than that.
For speaker amps, you generally need a lot of power unless you have very efficient speakers, e.g. horns, which is another topic altogether. We're talking about 50-100 watts per channel or more. So the amp needs to produce 40 volts peak for a 100 watt amp, and on the order of 10 amps if we have a 4 ohm speaker. Now, a class A push-pull amp has a maximum theoretical efficiency of 50% (single ended class A has a max efficiency of 25%), so that means it runs at 100-200 watts/channel all the time. This means lots of tubes (old days) or lots of heatsink for solid state. All this means lots of heat and lots of money (and that doesn't even include the cost of running the damn things). So basically, until Mark Levinson and Krell came along, pretty much all amplifiers were class AB because nobody thought that there was a big enough market to support a class A commercial amp. Class AB was considered the best compromise between cost, efficiency and sound quality. It was never considered the best, just that Class A was considered too expensive.
The first Mark Levinson Class A amp was rated at 25 watts (but was reportedly class A down to 2 ohms, at which point it was a 100 watt amp), was mono, and was big, heavy and cost a few thousand dollars - I think you could have bought over 100 NAD3020 integrated amps for the same money, and the 3020 was stereo, had roughly the same power rating, and included a phono section. But, it turned out that there was a market for it, and Krell came out with stereo class A amps that had more power and generated more heat. And it's gone on from there. The limiting factors for speaker amps is basically how much power do you need and how much money (and size, and heat) do you want to spend getting it.
For non-electrostatic headphone amps it is significantly different. Here, you only need a few watts at most, into higher impedances than speakers have. So low to moderate voltages (say 15 volts at most)and relatively low currents (an amp or so at most). Based on specs, the least efficient non-electrostatic headphone is the HFM Susvara, and 5 watts into that should produce 120 dB, which is enough to give you hearing damage in minutes if sustained. So in theory, a class A headphone rated at 5 watts into the Susvara's impedance should do the trick. Which means that an amp that draws 25-30 watts from the wall should be able get you close to the threshold of pain In that case, there isn't a lot to be gained by dropping back to class AB. That said, some cheap headphone amps ARE run in class AB because the manufacturer doesn't want to lay out money for heatsinks. Every penny counts.
For electrostatic headphones, it is different again, stat headphones have very high impedances, so you need very high voltages but relatively minuscule currents - a few mA. because you need to have high voltage - the Stax amps generally have +/-350V supplies, the KG amps generally run +/-350V to +/-450 volts, the Stax and DIY T2 run +/-500V supplies. Now the Stax amps run around 5-7 mA/output device, the KG amps up to 20 mA/output device, and there are 2 output devices per channel. So given that, the Stax amps draw about 40-55 watts from the wall, the KG amps run about 140 watts to more than 200 watts (the latter from a DIY T2, which also has 8 tube heaters using energy).