Annual earnings at The Fresh Market fell despite higher sales thanks to higher SG&A costs and impairment charges.