The National Farmers Union, which opposes the changes, had urged Labour MPs to abstain to put pressure on ministers to water down the plans.
The government argues the current relief is unfair and the £1m threshold, in combination with the standard reliefs of up to £500,000 per person, means that "small family farms" will still be shielded from inheritance tax.
In a concession at last week's Budget, ministers announced that unused parts of the £1m allowance will be transferable to spouses and civil partners, a move expected to reduce the eventual tax take by £70m a year.
Jenny Riddell-Carpenter, the Labour MP for Suffolk Coastal, said she supported the concession but urged ministers to "go further" to mitigate the effect of the changes.
Samantha Niblett, Labour MP for South Derbyshire, said her party, which did not include the plans in its manifesto ahead of last year's general election, had "lost the trust" of farmers.
"For all the good that this Budget does - and it does a lot - many farmers in my constituency simply cannot see past the broader inheritance tax changes," she told MPs.
"Most farmers are not wealthy land barons; they live hand to mouth on tiny and sometimes non-existent profit margins."
Explaining his decision to rebel, Campbell-Savours said he had promised local voters before the election that Labour would not touch the relief, adding it would leave many farmers "devastated at the impact on their family farms".
Treasury minister James Murray told MPs the government's amended plans represented a "fair way forward" on the issue.
"They represent generous relief for people, while raising money for the public finances," he added.
The Conservatives and Lib Dems both oppose the government's proposals, with Tory shadow environment secretary Victoria Atkins saying she commended the Labour MPs who did not vote for the measure in the Commons.
