A Kent council cabinet member said he has "nightmares about potholes" as it aims to invest millions into fixing the problem.