Celebrity websites TMZ.com and Us Weekly quoted unnamed sources as saying Cole, 27, was being replaced in the new Fox television venture created by Simon Cowell.
Fox said it had no information and representatives of Cole, Cowell and the show's producers did not return calls for comment.
The much-hyped X Factor, created by former American Idol judge Cowell, is the highlight of Fox's fall TV schedule. It reunites Cowell with his old Idol sparring partner Paula Abdul and offers an unprecedented $5 million (£3m) prize to the winner.
Great website to make big money online.