Sheeran is married to Cherry Seaborn and the couple have two children. He also spoke to Theroux about the difficulties of fame and having to have security for his family.
He said that for him personally, "it's kind of the part of the normal of being in the public eye" but that is was different when it came to his wife and children.
"I'm very, very private about my children's image... I can't take them to zoos or parks or anything really without someone trying to film them.
"There are times where the normal parts of life I kind of mourn for and wish I could push my kind on a swing in a public park and it not be weird, you know?"
He said: "People go, well that's the trade-off. But my kids have not signed up to that."
He added it was sometimes tricky when he was flying as he usually travels on commercial flights due to environmental concerns.
"We're flying back on Ryanair from a gig in Italy last year but all of my fans who'd been at the gig were flying back!
"I like the environment, I like trees. I'd find it hard to justify. No one's perfect. When we're doing intense promo trips, there will be the odd time [taking a private jet]", he told Theroux.