https://academia.stackexchange.com/questions/102443/referee-recommends-paper-rejection-with-no-further-comment-how-to-react
In this case de-escalation is a good advice. That means, to accept that the journal is maybe right. So, you must ask yourself, if your article was really good and that the referee had done the correct decision. He gets lots of papers every day and must be fair to anybody, so it is normal that many papers are not good enough for the journal in which he works for.