International Trade Secretary issued warning ahead of the Conservative Party conference this weekend.