James MacDougall, 37, went ahead with private sperm donations to gay women, despite knowing he suffered from Fragile X syndrome.
The genetic condition, which is inheritable, leads to low IQ, developmental delay, and cannot be cured.
MacDougall's identity was revealed after he requested the courts to spend some time with four of the tots, aged up to three.
Mrs Justice Lieven, sitting at Derby, denied him parental responsibility and said he should be named to stop other women from using him as a sperm donor.
But the 37-year-old, who advertised the private donations on social media, insists he has done nothing wrong and was actually helping these women.
He told Mail Online: "I haven't done anything wrong.
"I did a good thing by helping these women, I gave them children but people are saying that I was not honest.
"The full truth will come out. I'm very angry and upset."
His mum, Mrs MacDougall, 73, told the outlet how her son "just wanted to be a part" of the children's lives and was now "struggling with it all" after his request was denied.
She said: "He wanted to be part of their lives, he wanted to see his children.
"He is kind hearted and would do anything for anybody but he is gullible.
"He just wanted to help those people, help those women in a gay relationship fulfil their dreams and become parents.
"He didn't seek any money, as far as we are aware, for providing that service, he just did it."
While MacDougall initially signed an agreement saying he did not want contact with some of the children, he later asked to spend some time with them.
Three mums opposed Mr MacDougall's response.
The judge adjourned the case until she received a report from social services.
She found MacDougall was a complex person, with learning difficulties and on the autistic spectrum, with fixed views, concrete thinking and a profound lack of insight.