Mr Evans added secondary headteachers told inspectors they would "get one applicant if they're lucky" when they advertise for maths teachers "and sometimes that person isn't employable".
Claire Morgan, strategic director of Estyn, said they were "really concerned" about numeracy, adding there should be a "determination to improve standards in mathematics".
"If we're to get numeracy right we have to get the quality of maths teaching right first, because standards in mathematics are too low," she told members of the committee.
On pupils' behaviour, Mr Evans said Estyn would be publishing a report on the topic soon but they had found it was less of a problem "where they have quite strong policies that have been developed with pupils and parents".
He said "very clear expectations" and "consistency of application so that everyone know where the boundaries are" was also important.
It came as the Welsh government said it would be providing an extra £20m for schools to support education standards in 2024-25.
Education Secretary Lynne Neagle said: "Through continued dialogue with education partners, I understand the scale of the difficulties being faced every day and I am thankful for the hard work of the education workforce across Wales.
"In recognition of the pressures our schools are facing, I want to provide as much funding as possible to raise school standards and support our learners to thrive."