Health Secretary Wes Streeting said it was "not right" that manufacturers had been able to package their products in ways that took advantage of new parents who are concerned about what is best for their baby.
"These new measures mean parents will have confidence in the formula they are buying, no matter the price, and can now make the most of supermarket loyalty schemes too," he said.
Shereen Fisher, director of the Baby Friendly Initiative at UNICEF, welcomed the move saying infant formula was "a basic necessity".
"For too long, families have faced inflated prices for this essential product. The CMA has shown that many formulas are vastly overpriced, with many families struggling as a result," she said.
"Today's announcement signals the first step to tackle these issues, improve affordability and strengthen infant feeding support."
The announcement was also welcomed by infant feeding charity Feed.
Its chief executive, Clare Murphy, said: "We are delighted to see clarification that there is no law preventing the use of vouchers and loyalty points in the purchase of infant formula, a ludicrous restriction that simply stigmatises and punishes families' feeding decisions."
Andrea Martinez-Inchausti, assistant director of food at the British Retail Consortium (BRC), said the government's proposed next steps were sensible.
"We look forward to working through the detail with them to implement the necessary changes."
Other recommendations from the CMA, which the government has agreed to adopt in principle, include ensuring all infant formula is displayed together, separate from other formula milks, and to clarify what counts as advertising.
The government said further action was needed on other recommendations, including the prohibition of non-verifiable messages on infant and follow-on formula labels, and extending the restriction on advertising for follow-on formula.
A CMA spokesperson said the watchdog was ready to support governments and agencies across the UK in either implementing its recommendations or advising on measures that remained under consideration.
The authorities in all four devolved nations have agreed to the government's response.