Local government body Cosla said councils remained committed, within tight budgets, to ensuring that the best possible leisure and culture services were available.
In its report, the Accounts Commission said many local authorities were reviewing the services they offered while raising or introducing charges.
It warned that the most deprived communities would face "inequality and exclusion" if cuts to spending on culture and leisure services continued unchecked.
Prof Grant Jarvie, chairman of sport at the University of Edinburgh and chairman of the local authority's leisure, culture and sport trust, told the programme that sport, leisure and culture were disproportionately hit because they were not statutory such as schools, housing and transport.
He said: "The report suggests a 3% reduction in real terms but other reports talk about a 20% cut in local government investment in culture, sport and leisure services and as high as 33% in some local authorities."
He added that local authorities only needed to provide adequate provision and were "forced to make choices they shouldn't have to make".
He said sport brought about £2.5bn to the Scottish economy with even more coming from culture industries.
He also highlighted the community groups that benefit from sport and culture spending, including Street Soccer, the Sistema Scotland music education programme and the Homeless World Cup.
"Local authorities have to make choices, but governments of all colours also make choices and in this case it's the wrong ones," he said.
Overall, £675m was spent on culture and leisure in Scotland in 2023-24 , with £118m raised by charges - a 27% increase since 2018/19.
At the same time, local leaders have approved £9.9m worth of savings in these areas for the current year.
The Accounts Commission said gaps in how data was collected by councils meant "we don't have a full understanding of the impact of these decisions on health, wellbeing and prevention".
A total of 44 million visits were made to the nation's libraries in 2023/24, the same as 2018/19.
Scotland's public sports facilities saw numbers decline by 21% in that time to 43 million last year.
Visits to museums across the country have increased by 25% since 2018/19 to 15 million.
Satisfaction and attendance rates for some services remain below pre-pandemic levels.
Jo Armstrong, chairwoman of the Accounts Commission, said culture and leisure services were "vital" to health and wellbeing and called for gaps in national data to be addressed.
She added: "Communities must be fully consulted on decisions to close, centralise facilities or changes to charging. Failing to do this risks deepening inequalities and legal action by communities."
A spokesperson for Cosla said that vital services would "continue to be at risk" without sustainable long-term funding.
They said Cosla was already working with councils and partners to see how it could increase national and local investment in facilities and initiatives to support the health, wellbeing and education of communities.
"We will consider the Accounts Commission findings and recommendations as part of that ongoing work," they added.