More than a third of food businesses operating in the UK have said they would become "unviable" without access to workers from the European Union.