Mixed reality In-situ Welding

Recently, researchers Daragh Byrne, Tate J., Zhenfang Chen, Semina Yi, Andrew Knowles, Yumeng Zhuang, Ann Li, and Tiger Man To Tang from CMU have introduced Mixed Reality In-Situ Welding. Integrating XR (Extended Reality) and TinyML (Tiny Machine Learning) to train skilled workers in creative arts and manufacturing means leveraging new technologies to enhance the learning process and job readiness in these fields.

Here’s a breakdown:

1. Extended Reality (XR): XR is an umbrella term that includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). These technologies can create immersive learning experiences, enabling workers to practice skills in virtual environments that closely replicate real-world scenarios in arts and manufacturing. For instance, artists can learn 3D modeling in a virtual studio, or manufacturing workers can practice assembling machinery in a simulated environment.
2. TinyML: TinyML refers to machine learning models optimized to run on low-power, resource-constrained devices like microcontrollers and edge devices. In training, TinyML can be used to develop portable, intelligent tools that help in real-time performance feedback, monitoring of skill development, and even automation tasks. For example, in manufacturing, TinyML can be embedded into wearable devices that track and enhance workers' precision in tasks like welding or machining.

By combining XR's immersive training capabilities with TinyML's data-driven insights and feedback mechanisms, you can create a holistic and scalable training environment. This helps workers quickly adapt to new skills, future-proofing them for emerging jobs in both creative and industrial sectors.

The team partnered with the Industrial Arts Workshop (IAW), which specializes in teaching creative arts welding to teens in Hazelwood, a community often overlooked when it comes to resources. Together, they developed, tested, and deployed an in-situ training system. This initiative is a step towards making learning more accessible and transferable across various fields—whether it’s operating machines, woodworking, or physical computing. The goal is to ultimately enhance recruitment and retention in skilled trades by creating immersive learning environments.

They are currently running design workshops and focus groups with welding instructors and students to pinpoint and refine the critical elements, moments, and actions within the teaching experience. These sessions are also addressing challenges in welding training, recruitment, and retention. In addition, they’re exploring tasks and creative activities that align students’ real-world interests with meaningful skills development, ensuring that learning outcomes are both engaging and practical for the students involved.