Since the Netherlands recognised same-sex unions in 2001, a number of other EU countries have followed suit, but others, such as Poland, Hungary and Slovakia, have not.
In September, Slovakia's populist-nationalist government changed its constitution to allow only heterosexual couples to adopt children.
In its ruling on Tuesday, the ECJ said that while marriage rules fell within each member state's competence, "countries were required to comply with EU law in exercising that competence".
As EU citizens, the Polish couple who married in Germany had the right to freedom of movement and the right to lead a family life, the court explained.
"In particular by virtue of marriage, they must have the certainty to be able to pursue that family life upon returning to their member state of origin," it added, so forcing them to live as an unmarried couple in Poland was contrary to EU law and infringed their rights.
Poland's pro-European coalition government has tried to advance the rights of same-sex couples, but its move could be vetoed by conservative President Karol Nawrocki.
The ECJ's ruling was welcomed by Katarzyna Kotula, a secretary of state and coalition ally of Polish Prime Minister Donald Tusk, who said it was an important step on the path to marriage equality.
Meanwhile, centre-left member of the European Parliament Krzysztof Smiszek wrote on social media that "history is happening".
