Data Generation Aids Material Characterisation from Images

Scientists are tackling the difficult task of characterising two-dimensional materials from optical microscopy images, a process hampered by subtle contrast, limited labelled data, and inconsistencies between laboratories. Xuan-Bac Nguyen and Hoang-Quan Nguyen, leading the research at CVIU Lab, University of Arkansas, USA, in collaboration with Tim Faltermeier and Nicholas Borys from the University of Utah, and working with colleagues at the Department of Physics, University of Arkansas, USA, present a novel physics-aware multimodal framework to overcome these challenges. Their work introduces Synthia, a physics-based synthetic data generator, and QMat-Instruct, a large-scale instruction dataset, to train Multimodal Large Language Models to accurately assess flake appearance and thickness. Significantly, the team proposes Physics-Aware Instruction Tuning (QuPAINT), a new architecture incorporating physics-informed attention, and establishes QF-Bench, a comprehensive benchmark for standardised evaluation, promising to accelerate discovery in quantum materials. For years, accurately identifying layered materials from microscope images has relied on painstaking manual analysis. Now, a new approach combines simulated data with artificial intelligence to automatically determine material thickness and properties, offering a faster. More reliable route to discovering and characterising advanced materials. Scientists are increasingly focused on two-dimensional (2D) quantum materials like graphene and molybdenum disulfide due to their potential in next-generation technologies. These materials, possessing unique electronic and optical properties stemming from their atomic-scale variations, are attracting attention for applications in electronics, photonics, and quantum computing. Precisely characterising these materials, particularly determining flake thickness and identifying defects, remains a significant challenge. Current methods rely heavily on manual inspection, creating inconsistencies and limiting the speed of discovery. This laborious process hinders efforts to fully exploit the quantum functionalities within these materials and impedes wider adoption. To discern subtle differences between single-, bi-, and few-layered flakes using conventional imaging techniques proves difficult. Optical contrast can be obscured by interference patterns and variations in laboratory conditions, making automated analysis unreliable. By existing computer vision models, designed for identifying objects with strong visual cues, struggle with these subtle variations. These models often overfit to specific datasets and fail to generalise across different imaging setups or materials. The scarcity of large, well-annotated datasets further compounds the problem, restricting the development of strong and transferable analytical tools.

Scientists have now developed a new physics-aware multimodal framework to address these limitations. This approach moves beyond standard vision models by incorporating physical principles into both data generation and model design. A key component is Synthia, a synthetic data generator that simulates realistic optical responses of quantum material flakes. Here, this module fuses visual information with optical principles, creating more dependable representations of the flakes. To ensure fair and reproducible evaluation, The team also established QF-Bench, a benchmark spanning diverse materials, substrates, and imaging conditions. By combining synthetic data, instruction-based learning, and physics-informed modelling, this effort promises to accelerate the discovery and characterisation of advanced quantum materials. Generating synthetic flake imagery and a physics-informed instruction dataset for material property correlation Synthia, a physics-based synthetic data generator, formed the basis of this effort’s methodological approach. In turn, this generator simulates the optical responses of flakes undergoing thin-film interference, creating diverse and high-quality samples to lessen reliance on manually annotated data. Through modelling the physics of light interacting with these materials, Synthia produces images that closely resemble those obtained through optical microscopy. Yet offers complete control over ground truth parameters like layer count and flake size. Such synthetic images were used to pre-train models and augment limited real-world datasets. Through comprising multimodal, physics-informed question-answer pairs, QMat-Instruct teaches MLLMs to correlate flake appearance with physical properties such as thickness. Each question within the dataset prompts the model to reason about the visual characteristics of a flake. Demanding an understanding of how light interacts with different layer configurations. For example, a question might ask “What is the approximate thickness of this flake, given its colour and interference pattern”. Rather than treating visual data in isolation, QuPAINT explicitly incorporates knowledge of thin-film interference, guiding the model’s attention towards features relevant to material thickness and layer count. At the core of this module lies a mechanism for weighting visual features based on their consistency with established optical principles. Meanwhile, to ensure fair and reproducible evaluation, The effort established QF-Bench, a benchmark spanning multiple materials, substrates, and imaging settings. Standardized protocols were implemented to allow for consistent assessment of performance across different models and laboratories. Unlike previous evaluations limited to specific experimental setups, QF-Bench incorporates variations in optical magnification, illumination spectra, and even environmental factors, mirroring the diversity encountered in real-world research environments. To illustrate, the benchmark includes images acquired using both brightfield and darkfield microscopy, challenging models to generalise beyond a single imaging modality. QuPAINT achieves state-of-the-art flake detection across multiple model scales Across all detectors, the method consistently improves general flake detection performance. Traditional baselines, such as MaskRCNN-R50 and ViTDet (base), achieve modest average precision (AP) values ranging from 17 to 20 percent. YOLOv11 models offer stronger results, raising AP to approximately 25-30 percent, yet still struggle with capturing fine-grained flake boundaries. Particularly at higher intersection over union (IoU) thresholds. Co, DETR and RT, DETRv2 perform around 22-28 percent AP, while Mask-Terial reaches approximately 24 percent. Another physics-based flake detection approach improves AP to around 30 percent. In contrast, the QuPAINT-1B model already achieves 36.9 percent AP, substantially exceeding the performance of YOLOv8-x. As the model scale increases from 1 billion to 8 billion parameters, performance steadily improves, culminating in 45.6 percent and 60.5 percent AP50. These the multimodal architecture effectively models complex visual cues and material textures compared to conventional two-dimensional detectors. The improvement at AP75 is particularly strong, indicating more accurate and tighter localization. This is critical for identifying thin flakes. Mono-layer flake detection, a more challenging setting due to extremely thin structures and low contrast, reveals a significant performance drop for all baseline detectors. For instance, YOLOv8-x declines from 28.3 percent AP to only 19.0 percent AP, confirming the difficulty of localizing single-layer flakes using standard detection methods. Meanwhile, the method maintains strong performance and exhibits a much smaller performance decrease. QuPAINT-1B achieves 28.0 percent AP, surpassing all baseline models. Scaling up to QuPAINT-8B further boosts accuracy to 37.3 percent AP and 52.8 percent AP50. This shows that the model captures subtle cues. The full model, integrating both PAD and PIA, achieves the best performance with 34.1 percent AP, 50.2 percent AP50, and 38.0 percent AP75, representing consistent improvements across all metrics. Physics-informed artificial intelligence unlocks automated two-dimensional material characterisation The long-standing challenge of automatically interpreting images from optical microscopes has edged closer to a solution, thanks to a new approach blending artificial intelligence with fundamental physics. For years, identifying and characterising two-dimensional materials in these images proved difficult because subtle visual cues define layer thickness, and each microscope setup introduces its own distortions. By existing computer vision systems, trained on limited datasets, struggled to generalise beyond the specific conditions they encountered.

Scientists have moved beyond simply feeding machines more images, instead embedding physical understanding directly into the analytical process. Initially, a synthetic data generator named Synthia creates realistic simulations of how light interacts with these materials. Effectively providing a limitless supply of training examples. Beyond data, the team constructed QMat-Instruct, a dataset of questions and answers designed to teach AI models about flake appearance and thickness, informed by physical principles. The real step forward lies in QuPAINT, a new architecture that combines visual information with these optical priors, creating a more reliable system for recognising different material thicknesses. Reliance on synthetic data always carries a risk of mismatch between simulation and reality. While the new QF-Bench benchmark offers a standardised way to test performance across different conditions. It remains to be seen how well these models will perform when faced with genuinely unexpected variations in sample preparation or imaging quality. Once these systems move beyond controlled laboratory environments, the need for continual adaptation will be apparent. This effort represents a shift towards more intelligent image analysis, where AI isn’t just pattern recognition but incorporates underlying scientific principles. Beyond materials science, this physics-aware approach could be applied to other imaging techniques, such as biological microscopy. Where understanding the physics of light is essential for accurate interpretation. Instead of chasing ever-larger datasets, the future may lie in building AI systems that reason more like scientists. 👉 More information 🗞 QuPAINT: Physics-Aware Instruction Tuning Approach to Quantum Material Discovery 🧠 ArXiv: https://arxiv.org/abs/2602.17478 Tags: