Just thinking aloud here.
In theory...a turbocharger not only helps pump air into the engine; it also recovers waste heat from the engine exhaust, thus increasing net efficiency. This is why an ideal supercharged engine will never match the efficiency of an ideal turbocharged engine. I might even argue the same could be said for peak power (the ultimate in artificial aspiration being the turbo-compound engine, which increases energy recovery even more).
This would mean that under all conditions, if you are burning any fuel at all, the turbo is *trying* to spool. Even at idle. Exhaust gas volume has to be larger than intake gas volume, otherwise the engine cannot perform work. However, this can get skewed when you factor in the efficiency map of the turbo. Rather, the fact that it's nowhere near efficient at those speeds. Like all mechanical devices, a turbo must consume some energy before it can transmit energy on to the next component in the process. At high engine loading, there is enough exhaust gas flow to render the turbocharger's friction and parasitic losses more or less insignificant.
Whether or not the turbo can actually result in a net loss in efficiency at extremely light engine loading (such as idling) is difficult to say without a controlled experiment. But I guess it might be possible.
Once cruising at 55mph, my gut tells me the turbo is efficient enough to at the very least not be a net negative. This is only my conjecture though.
EDIT: Sorry, Hydro. Looks like I went and made it more complicated