The government has pledged to reduce the administrative cost of regulation by a quarter by the end of this Parliament.
Kyle said the previous government "did not do enough on deregulation" despite pledging to do so, particularly after Brexit.
"If you look at some of the reporting that needs to be done by directors, for example, directors' reports to Companies House, I'm eliminating a great deal of that today because some of it is just so unnecessary," he said.
But pushed on whether the government's changes to employment rights would add costs to businesses, Kyle insisted that the changes would be fair for both employers and employees.
"We are making sure that the rights and responsibilities that people have in the workplace as employers and as employees [are] right for the age we're living in."
Jane Gratton, the deputy director of public policy at the British Chambers of Commerce, said the plans would be welcomed by businesses.
"The burden of unnecessary red tape and bureaucracy ramps up their costs and damages competitiveness," she said.
But Tina McKenzie, policy chair at the Federation of Small Businesses, said Tuesday's announcement would "ring hollow" if the chancellor raised taxes for employers in next month's budget.
"The true test of whether Rachel Reeves will deliver for business will be at the Budget – small firms and entrepreneurs have heard these warm words on regulation before.
"The burden of compliance – in terms of money, time, and stress – weighs heavily on small firms, and cutting it needs to be a project undertaken by every part of the government."
Tom Ironside from the British Retail Consortium said that while retailers supported efforts to cut red tape, "there are several policies coming down the track that will add to, rather than cut back, business bureaucracy".
Liberal Democrats' Treasury spokeswoman Daisy Cooper said: "If the chancellor was serious about cutting red tape she would tackle the mind-blowing two billion extra pieces of business paperwork created by Brexit by pursuing an ambitious tailor-made UK-EU customs union."