This state, known as a quantum spin liquid, was first predicted 40 years ago, researchers said.
The scientists, including those from the University of Cambridge in UK, measured the first signatures of these fractional particles, known as Majorana fermions, in a two-dimensional material with a structure similar to graphene.
Their experimental results matched with the Kitaev model, one of the main theoretical models for a quantum spin liquid.
The observation of one of their most intriguing properties - electron splitting, or fractionalisation - in real materials is a breakthrough.
Also Read
The resulting Majorana fermions may be used as building blocks of quantum computers, which would be far faster than conventional computers and would be able to perform calculations that could not be done otherwise.
"This is a new quantum state of matter, which has been predicted but hasn't been seen before," said Johannes Knolle from University of Cambridge in UK.
However, in a material containing a spin liquid state, even if that material is cooled to absolute zero, the bar magnets would not align but form an entangled soup caused by quantum fluctuations.
"Until recently, we didn't even know what the experimental fingerprints of a quantum spin liquid would look like," said Dmitry Kovrizhin, from University of Cambridge.
They tested the magnetic properties of a-RuCl3 powder by illuminating it with neutrons, and observing the pattern of ripples that the neutrons produced on a screen when they scattered from the sample.
A regular magnet would create distinct sharp lines, but it was a mystery what sort of pattern the Majorana fermions in a quantum spin liquid would make.
The theoretical prediction of distinct signatures by researchers in 2014 match well with the broad humps instead of sharp lines which experimentalists observed on the screen, providing for the first time direct evidence of a quantum spin liquid and the fractionalisation of electrons in a two dimensional material.