U.S. Census Bureau data out Wednesday underscore just how lousy the recovery has been if you aren't rich.

Looking at eight groups of household income selected by Census, only those whose incomes are already high to begin with have seen improvement since 2006, the last full year of expansion before the recession. Households at the 95th and 90th percentiles had larger earnings through 2014, the latest year for which data are available.

Income for all others was below 2006 levels, indicating they're still clawing their way out of the hole caused by the deepest recession in the post-World War II era.

"Each decade, it's taken longer for the poor to recover from recession, for the poverty rate to start turning around after the official end of the recession," said Arloc Sherman, a senior fellow at the Center on Budget and Policy Priorities in Washington. "There's quite a bit of work left to do."

Median household income is 6.5 percent lower than in 2007, the year the recession started.

Overall, median income was $53,657 in 2014, not a statistically significant difference on an inflation-adjusted basis from 2013's median of $54,462. It's the third straight year that there's been no significant change, after two consecutive years of annual declines.

That's happened even though the labor market has posted steady progress. The number of men working full time and year-round in 2014 increased by 1.2 million from a year earlier while the number of women gained 1.6 million. The changes in their real median earnings, however, weren't statistically significant.

Meanwhile, the official poverty rate was 14.8 percent, with some 46.7 million people in poverty—both little changed from 2013. The rate is 2.3 percentage points higher than it was in 2007.