Extremists will also now be banned from taking a "leading role" in religious services under the new measures.
It follows a 2022 report which raised concerns about radicalisation and Islamist gangs in prisons.
New Justice Secretary Alex Chalk said the changes would stop terrorists advancing "their own sinister agenda".
Those in custody convicted of terrorism - about 200 people - will be limited to two medium-sized boxes of books that must weigh no more than 15kg.
Prisoners can obtain books from prison libraries, order from approved retailers or be sent them by friends and family.
But there are concerns around extremist materials being hidden inside approved books, or covers being swapped as a disguise.
The government said the move would make it easier for prison staff to search for prohibited material, citing a case in which one convicted terrorist had 200 books in his cell.
A previous blanket ban on prisoners being sent books from people outside prison was ruled to be unlawful by the High Court in 2014, less than a year after it was imposed.
The same ruling said there was "no good reason" to restrict the amount of books prisoners can have by volume, but the Ministry of Justice said it was not expecting a legal challenge as the change is limited to terror offenders.
The changes announced on Sunday do not require Commons approval and come into force immediately, the Ministry of Justice said.
Restrictions will also be strengthened so convicted terrorists can not have any formal role in religious services, such as delivering a reading.
Currently only the most dangerous prisoners are banned from leading Friday prayers. Now the ban will cover prisoners of any faith, and not just those in high-security prisons.
The changes follow recommendations made by Jonathan Hall, the Independent Reviewer of Terrorism Legislation, which drew particular attention to the influence of Islamists in prison.
He said the prison service had "lost its role in the national endeavour to reduce the risk of terrorism" and called for tighter restrictions on terror offenders while in custody.
Mr Chalk, who replaced Dominic Raab as justice secretary after his resignation earlier this month, said he recognised the role faith can play in a prisoners' rehabilitation but said some may abuse the rules.
He added: "These changes, alongside tougher sentences for terrorists who commit crimes behind bars and our work to separate more of the most radical terrorists, will better protect our hardworking staff, other prisoners and the public."
Labour responded by criticising the government's record on counter-terrorism policy, referencing concerns about the potential effect on terror trials raised by the security services in 2021 amid attempts to reform the Human Rights Act.
Shadow justice secretary Steve Reed said: "If the Conservatives really cared about stopping terrorists they would not be proposing changes in the law that will slow down prosecuting them.
"Britain's security services slapped down the government's disastrous proposals to slow down trials for foreign terrorists and risk cases collapsing so instead of being jailed or deported they remain loose on Britain's streets.
"Only Labour can be trusted with keeping the public safe. And that starts by working with our intelligence services to do their job, not working against them."
Liberal Democrat cabinet affairs spokesperson Christine Jardine accused the Conservatives of breaching pre-election rules, which restrict ministers from making party political announcements with the help of government resources in the run-up to a vote.
In a letter to the Cabinet Secretary Simon Case, she called for the timing of the announcement - which came via the Ministry of Justice press office days before the local elections - to be investigated.