At least, according to ICv2's latest white paper-- in 2008 manga sales in the US and Canada dropped 17%, and in 2009 they dropped another 20%, meaning that they've diminished by over a third in just two years.
ICv2 notes that the sales drop was heavier in bookstores than it was in the direct market, and cites the loss of big-name anime on TV as a factor in the drop. This tells me that the audience that's being lost is NOT so much the hardcore fan-- though surely the regular consumption of scanlations isn't helping --but rather the casual fans, the people who wouldn't be on this site in the first place, most likely.
The rest of the factors listed are ones we're pretty used to: manga fans aren't "collectors" like comics fans, series are long and hard to collect in their entirety, fewer titles are being released, etc.
Also of note is that manga has seen a similar pattern in European markets-- a large boom followed by a drop before stabilizing --so in all likelihood we can hope for the same here. But that means you guys'll have to keep on buying your favorite series! What's on your shopping list these days?