Is every pair of jeans like no other? According to the testimony of FBI forensic analysts, the patterns seen on denim are reliably unique and can be used to identify a suspect in surveillance footage.

The problem is, this technique has never been subjected to thorough scrutiny, and evidence acquired through it may not be as strong as it has been claimed to be. A paper published in PNAS this week puts denim-pattern analysis through its paces, finding that it isn’t particularly good at matching up identical pairs of jeans-and may create a number of “false alarm” errors to boot.

Shoddy evidence

For some time, there have been rumblings about the reliability and quality of commonly used forensic techniques. In 2009, the National Academy of Sciences published a weighty report observing that, apart from nuclear DNA analysis, “no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.”

The problems with forensic evidence-including fingerprint, bloodstain, and ballistics analysis-have terrible real-world consequences. According to the National Registry of Exonerations, nearly a quarter of wrongful convictions in the United States for the last 30 years can be attributed to flawed or misleading forensic evidence.

Computer scientists Sophie Nightingale and Hany Farid wanted to look at one technique in particular: photographic pattern analysis, which matches up the patterns of details on faces, hands, or clothing between suspects and crime-scene photographs. Jeans, for example, have a “barcode” pattern of dark and light splotches along their seams.

Denim barcodes

These patterns have been used as central evidence to convict people, but is this kind of analysis reliable? That hasn’t been established. To test it out, Nightingale and Farid went out to buy 100 pairs of jeans from second-hand stores. They laid the jeans out flat on a hard surface, photographed the seams along the legs, and digitally traced the pattern of light and dark points along the seams. To bump up their sample, they had Amazon Turk workers supply images from another 111 pairs, photographed using careful instructions.

Then, the researchers set about quantifying how different the patterns were across different pairs of jeans. Obviously, there’s a lot of randomness at play here-two pairs could be quite similar, just by chance, while another two pairs could be entirely different, also by chance. And most pairs would fall somewhere in the middle, with some degree of similarity. Based on these measurements, Nightingale and Farid worked out the range of similarity between the "barcode" patterns on different pairs of jeans.

The important question, of course, is whether these patterns can be used to determine whether two images show the same pair of jeans. So the researchers selected 10 pairs of jeans and took 10 photos of each using different cameras, in different lighting, and with different draping. What they found was that any given pair of photos could come back with a lot of similarities but could also come back with very different readings on the pattern. The range was broad-as Nightingale and Farid point out, soft fabric photographed in a bunch of different ways is going to have distortions that vary from one image to the next.

False alarms

So if one pair of jeans can look noticeably different in different photos, is denim-pattern analysis actually a useful forensic technique? The researchers used their measurements to estimate how often a true match would come up. How often their jeans would throw up a “false alarm“-a score that looked like a match even though the images actually came from two different pairs.

They found that the false alarm rate could be as high as one in a thousand. Given that the FBI has reported using photographic pattern analysis in hundreds of cases each year, that’s a meaningful possibility. The true match rate was also not great, at around 40 to 50 percent, depending on factors like the length of the seam being analyzed.

This means the technique of matching up jeans is likely to be pretty hit and miss-not catching actual similarities a lot of the time and possibly throwing up a high rate of false alarms. And that’s under controlled experimental conditions using high-quality images and jeans laid out nice and flat, not grainy security footage showing jeans being worn. On the other hand, different features like damage, branding, and size could corroborate an analysis to improve the evidence one way or another.

Costway US $314.00 Modern Loveseat Sofa Upholstered Dutch Velvet Sofa Couch-Green Costway US $484.00 4 Pieces Patio Acacia Wood Thick Cushion Loveseat Sofa Set-White Costway US $225.00 2-Person Double Rocking Loveseat with Mesh Fabric and Center Tempered Glass Table-Brown Costway US $361.00 4 Pieces Rattan Patio Rocking Furniture Set with Loveseat and Coffee Table Costway US $139.00 50 Inch Patio Garden Bench Loveseats for Outdoor Costway US $348.00 Modern Fabric Loveseat Sofa for with Metal Legs and Armrest Pillows-Army Green Costway US $225.00 2-Person Rocking Loveseat with Mesh Fabric and Tempered Glass Table Costway US $236.00 Outdoor Loveseat Chair Set with Coffee Table and Seat Back Cushions-Navy Costway US $212.00 Living Room Wooden Leg Armless Loveseat Sofa-Multicolor Costway US $311.00 Modern Loveseat Sofa Couch with Side Storage Pocket and Sponged Padded Seat Cushions-Blue