A power saving for intersective polynomial differences with an exponent depending only on the degree

An integer polynomial is intersective if it has a root modulo every positive integer. For each degree $k\ge2$, we prove that there is an exponent $c_k\gt 0$ such that every set $A\subseteq\{1,\ldots,N\}$ whose differences avoid all nonzero values $h(1),h(2),\ldots$, where $h$ is an intersective polynomial of degree $k$ with positive leading coefficient, satisfies $|A|=O_h(N^{1-c_k})$. The implied constant may depend on $h$, but the power-saving exponent depends only on its degree.