It comes a day after the president's controversial announcement that all US troops would be withdrawn from Syria.
Mr Trump did not name a successor, but said one would be appointed shortly.
In his resignation letter, Gen Mattis described his views on "treating allies with respect" and using "all the tools of American power to provide for the common defence".
"Because you have the right to have a secretary of defence whose views are better aligned with yours on these and other subjects, I believe it is right for me to step down," Gen Mattis wrote.
Responding to the news, Republican Senator Lindsey Graham said Gen Mattis "should be proud of the service he has rendered to President [Trump] and our nation".
"He has been in the fight against radical Islam for decades and provided sound and ethical military advice to President Trump," he tweeted.
Democratic Senator Mark Warner said the development was "scary", describing Gen Mattis as "an island of stability amidst the chaos of the Trump administration".