A member of an investment consortium that struck a deal for South Africa's largest dairy company Clover Industries in February is pulling out.