Sometimes when I ask myself why I've made a certain choice, I realise I don't actually know. To what extent we are ruled by things we aren't conscious of? – Paul, 43, London

Why did you buy your car? Why did you fall in love with your partner? When we start to examine the basis of our life choices, whether they are important or fairly simple ones, we might come to the realisation that we don't have much of a clue. We might even wonder whether we really know our own mind, and what goes on in it outside of our conscious awareness.

Luckily, psychological science gives us important and perhaps surprising insights. One of the most important findings comes from psychologist Benjamin Libet in the 1980s. He devised an experiment which was deceptively simple, but has created an enormous amount of debate ever since.

Participants were asked to sit in a relaxed manner in front of an adapted clock. On the clock face was a small light revolving around it. All those taking part had to do was to flex their finger whenever they felt the urge, and remember the position of the light on the clock face when they experienced the initial urge to move their finger. At the same time as that was all happening, the participants had their brain activity recorded via an electroencephalogram (EEG), which detects levels of electrical activity in the brain.

What Libet was able to show was that timings really matter, and they provide an important clue as to whether or not the unconscious plays a significant role in what we do. He showed that that the electrical activity in the brain built up well before people consciously intended to flex their finger, and then went on to do it.

In other words, unconscious mechanisms, through the preparation of neural activity, set us up for any action we decide to take. But this all happens before we consciously experience intending to do something. Our unconscious appears to rule all actions we ever take.

Some decisions we give far more thought to than others

But, as science progresses, we are able to revise and improve on what we know. We now know that there are several fundamental problems with the experimental set-up that suggest the claims that our unconscious fundamentally rules our behaviour are significantly exaggerated. For example, when correcting for biases in subjective estimates of conscious intention, the gap between conscious intentions and brain activity reduces. However, the original findings are still compelling even if they can't be used to claim our unconscious completely rules our behaviour.

Another way of approaching the idea of whether we are ultimately ruled by our unconscious is to look at instances where we might expect unconscious manipulation to occur. In fact, in my research I asked people what those were.

The most common example was marketing and advertising. This may not be a surprise given that we often come across terms such as "subliminal advertising", which implies that we are guided towards making consumer choices in ways that we don't have any control over consciously.

James Vicary, who was a marketer and psychologist in the 1950s, brought the concept to fame. He convinced a cinema owner to use his device to flash messages during a film screening. Messages such as "Drink Coca-Cola" flashed up for a 3,000th of a second. He claimed that sales of the drink shot up after the film ended. After the significant furore around the ethics of this finding, Vicary came clean and admitted the whole thing was a hoax – he had made up the data.

In fact, it is notoriously difficult to show in laboratory experiments that the flashing of words below the conscious threshold can prime us to even press buttons on a keyboard that are associated with those stimuli, let alone manipulate us into actually changing our choices in the real world.

Unconscious processes, such as intuition, function in ways that automatically and rapidly synthesise a range of complex information

The more interesting aspect around this controversy is that people still believe, as has been shown in recent studies, that methods such as subliminal advertising are in use, when in fact there is legislation protecting us from it.

But do we make decisions without consciously thinking? To find out, researchers have investigated three areas: the extent to which our choices are based on unconscious processes, whether those unconscious processes are fundamentally biased (for example, sexist or racist), and what, if anything, can be done to improve our biased, unconscious decision-making.

To the first point, a pivotal study examined whether the best choices made in consumer settings were based on active thinking or not. The startling findings were that people made better choices when not thinking at all, especially in complex consumer settings.

The researchers argued that this is because our unconscious processes are less constrained than conscious processes, which make huge demands on our cognitive system. Unconscious processes, such as intuition, function in ways that automatically and rapidly synthesise a range of complex information, and this gives an advantage over thinking deliberately.

As with the Libet study, this research motivated intense interest. Unfortunately, efforts to replicate such impressive findings were extremely difficult, not only in the original consumer contexts, but beyond into areas where unconscious processes are thought to be rife such as in unconscious lie detection, medical decision-making, and romantically motivated risky decision-making.

That said, there are of course things that can influence our decisions and steer our thinking that we don't always pay close attention to, such as emotions, moods, tiredness, hunger, stress and prior beliefs. But that doesn't mean we are ruled by our unconscious – it is possible to be conscious of these factors. We can sometimes even counteract them by putting the right systems in place, or accept that they contribute to our behaviour.

Changing road layouts and road markings are one way that nudge theorists have tried to manipulate people's behaviour

But what about bias in decision-making? A highly instructive study showed that, through the use of a now widely adopted technique called the implicit association test (IAT), people harbour unconscious, biased attitudes towards other people (such as racial or gender discrimination). It also suggested that these attitudes can actually motivate biased decisions in employment practices, and legal, medical and other important decisions that affect the lives of those on the receiving end.

However, the alarm can be muted when looking more closely at research on the topic, since it shows two critical problems with the IAT. First, if you look at an individual's test scores on the IAT at one time, and get them to do it again, the two don't match consistently – this is known as limited test-retest reliability. Also, it has been shown that IAT results are a poor predictor of actual decision-making behaviour, which means that the test has low validity.

There have also been efforts to try to improve the way we make decisions in our day-to-day lives (such as healthy eating or saving for retirement) where our unconscious biased processes might limit our ability to do so. Here the work by Nobel laureate Richard Thaler and Cass Sunstein has been revolutionary. The basic idea behind their work comes from cognitive scientist Daniel Kahneman, another Nobel prize winner, who argued that rash decisions which are primarily unconsciously motivated.

To help improve the way we make decisions, Thaler and Sunstein contend, we need to redirect unconsciously biased processes towards the better decision. The way to do this is through gently nudging people so that they can automatically detect which option is the better one to take. For example, you could make sweets less easily accessible in a supermarket than fruit. This research has been adopted globally by many public and private institutions.

We would rather claim conscious control and agency over our political voting than over what breakfast cereal we are purchasing

Recent research by my own team shows that nudge techniques often dramatically fail. They also backfire, leading to worse outcomes than if they weren't used at all. There are several reasons for this, such as applying the wrong nudge or misunderstanding the context. It seems that more is needed to change behaviour than nudging.

That said, nudgers lead us to believe that we are more easily influenced than we think, and than we are. A fundamental aspect of our psychological experiences is the belief that we are the agents of change, be it personal circumstances (such as having a family) or external ones (such as anthropogenic climate change).

On the whole, we would rather accept that we have free choice in all manner of contexts, even when we perceive it is under threat from mechanisms unconsciously manipulating us. However, we still strategically believe we have less agency, control and responsibility in certain areas, based on how consequential they are. For example, we would rather claim conscious control and agency over our political voting than over what breakfast cereal we are purchasing.

So, we may argue that our poor breakfast choice was down to subliminal advertising. However, we are less inclined to accept being duped into voting a certain way by big tech social media forces.

Subtle signals we don't always pay attention to, such as hunger or our emotions, can influence the decisions we make

Headline-grabbing scientific findings in psychology often don't help because they add to some of the extreme intuitions that we are fundamentally ruled by our unconscious. But the more robust scientific evidence indicates that we are more likely governed by conscious thinking than by unconscious thinking. We might get the sense that we aren't always fully aware of why we do what we do. This might be because we aren't always paying attention to our internal thoughts and motivations. But this isn't equivalent to our unconscious ruling our every decision.

While I don't think so, let's say that we are actually ruled by the unconscious. In this case, there is an advantage to entertaining the belief that we have more conscious control than not. In cases where things go wrong, believing that we can learn and change things for the better depends on us accepting a level of control and responsibility.

In cases where things go well, believing that we can repeat, or further improve on our successes, depends on accepting that we had a role to play in them. The alternative is to submit to the idea that either random, or unconscious forces dictate everything we do and in the long run that can be devastating mentally.

So why did you fall in love with your partner? Maybe they made you feel strong or secure, challenged you in some way, or smelt nice. Just like any other matter of importance, it is multifaceted, and there is no single answer. What I'd argue is that it's unlikely that your conscious self had nothing at all to do with it.