In 2016, Walker came sixth in the London mayoral elections winning 2% of the vote.
However, the party failed to build on its vote share and in the general election earlier this year, it stood four candidates and won just 1,275 votes.
It had also received some criticism over its position on transgender rights.
Journalist Suzanne Moore was involved in setting up the party in 2015, but in a recent article on Substack, said the party leaders had ignored the views of their own membership "70% of whom did not support Self ID".
A self-ID system would allow someone to change their legal sex without a medical diagnosis.
Currently, UK adults can get a gender recognition certificate if they meet certain criteria including a diagnosis of gender dysphoria.
Defending their approach, Mayer said the party had been "very clear" and "supported the right, from the beginning, for people to determine their own gender".
"We also tried very, very hard to make space for all areas of opinion on this."
Toksvig said: "I couldn't be more trans-inclusive but I didn't want to shout at people.
"I don't want a polarised debate... what I wanted was a space where we could carry on having the discussion."
Asked if the party's position on transgender rights had led to a drop in membership numbers, Mayer said they lost some members but also gained members and that overall it was "net neutral if not slightly positive".
Toksvig said the cost of living crisis following the pandemic had led to an overall drop in funding.