It will be “as close to an autobiography” as Paul McCartney “may ever come”: the former Beatle is set to publish The Lyrics, a deep dive into his life, based on conversations he had with the prize-winning poet Paul Muldoon.

The Lyrics, a two-volume, 900-plus page “self-portrait in 154 songs”, will be released on 2 November. It will be “a kaleidoscopic rather than chronological account” of McCartney’s life, said publisher Allen Lane, and will cover the musician’s earliest boyhood compositions – he wrote his first song at 14 – through the Beatles catalogue to Wings, solo albums and his present life. The book will cover “the circumstances in which they were written, the people and places that inspired them, and what [McCartney] thinks of them now”.

McCartney said: “More often than I can count I’ve been asked if I would write an autobiography, but the time has never been right. The one thing I’ve always managed to do, whether at home or on the road, is to write new songs.

“Some people, when they get to a certain age, like to go to a diary to recall day-to-day events from the past, but I have no such notebooks. What I do have are my songs, hundreds of them, which I’ve learned serve much the same purpose. And these songs span my entire life.”

Muldoon, the Pulitzer prize-winning Irish poet, edited the book, which is based on conversations he has had with McCartney. “Sir Paul and I met regularly over a period of five years for two or three hour sessions in which we talked in a very intensive way about the background to a half dozen songs,” said Muldoon. “In a strange way, our process mimicked the afternoon sessions he had with John Lennon when they wrote for the Beatles. We were determined never to leave the room without something interesting.”

Muldoon said that he was “struck again and again by what you might term Paul McCartney’s ‘scholarly’ disposition” during the process.

“He’s one of the most buoyant, upbeat people I know, but his general demeanor shouldn’t suggest that he’s anything but a deep thinker. He looks long and hard into every aspect of life and I believe readers, old and new, will be struck by a book that will show that side of him. He’s going to come out of this book as a major literary figure,” said Muldoon.

“His insights into his artistic process confirm a notion at which we had but guessed: that Paul McCartney is a major literary figure who draws upon, and extends, the long tradition of poetry in English.”

The volumes will also include material from McCartney’s personal archive, including never publicly seen drafts, letters and photographs. “We learn intimately about the man, the creative process, the working out of melodies, the moments of inspiration,” said the publisher.

McCartney said he hoped what he had written would “show people something about my songs and my life which they haven’t seen before”. “I’ve tried to say something about how the music happens and what it means to me and I hope what it may mean to others too,” he said.

McCartney’s previous books include Blackbird Singing, a collection of lyrics and poems spanning 1965-99; High in the Clouds, a children’s book written with Philip Ardagh; and the picture book Hey Grandude!, based on his relationship with his eight grandchildren and illustrated by Kathryn Durst.