Authors: Varvara Vasdeki and Lawrence Jongi
Abstract
This paper explores the development of digital systems to identify and reduce cognitive stresses in contemporary manufacturing environments with increasing numbers of robots and smart machines. To achieve this, the research attempts to answer the following research question: How can technology-driven advancements in industrial psychology be leveraged to foster more productive, ethical, and psychologically supportive collaboration between humans and robots in the context of modern manufacturing environments? The study employs a Literature Review to gain deeper insights into the subject and follows a methodology succeeded by the development of a prototype. By improving cognitive ergonomics through the detection and recognition of non-verbal cues, as well as reducing cognitive stress by providing real-time information on the positions of mobile robots, this study offers potential solutions to the social and psychological challenges of human-robot collaboration. The paper concludes with an assessment of the results, a discussion on sustainability implications, and recommendations for future research. Overall, this research aims to bridge the gap between human workers and technology in the manufacturing sector, facilitating a harmonious and productive collaboration that aligns with the goals of Industry 5.0.
Key Words: Industry 5.0, Industrial Psychology, Human-Robot Collaboration
1. Introduction
The next industrial revolution – Industry 5.0 (I5.0) – seems to have already begun and is expected to be human-centric at its core. Human workers are required to perform highvalue tasks in close collaboration with robots and “smart” machines. This raises a lot of concern on the incorporation of ethical principles into such collaborative systems [1] as well as the smooth transition to an interconnectional system between workers, organizations, and technology [2]. Psychological risks in the workplace are becoming more evident than physical ones, due to mental overload and work density induced by even more flexible and dynamic smart manufacturing activities [3]. Digitalization is key to understanding these concerns, as well as a powerful tool towards solutions.
In this paper, the use of digital tools as solutions for improving Human-Robot Collaboration (HRC) in a contemporary manufacturing environment is investigated. The specifics of the problem and the research question are presented in the subchapters below, while a review of related work was conducted for deeper insights. In Chapter 3 the research approach and the experimental design process are underlined. In the following chapters, the prototype development is described followed by its results. Finally, assessment of the results and relevant discussion with sustainability implications and future recommendations are presented.
1.1. Problem Definition
In contemporary manufacturing environments characterized by a growing presence of robots and smart machines, the human workforce faces escalating cognitive stresses. These challenges are exacerbated by the need for harmonious collaboration between humans and technology, a cornerstone of I5.0. The overarching problem centers on the development of digital solutions to enhance HRC within these evolving industrial landscapes. The aim of this paper is to develop such solutions with focus on addressing the cognitive ergonomics of human-robot interaction, including the recognition of nonverbal cues, while also mitigating cognitive stress through the provision of real-time information on mobile robot positions.
1.2. Research Question
The core challenge can be articulated as follows: How can technology-driven advancements in industrial psychology be leveraged to foster more productive, ethical, and psychologically supportive collaboration between humans and robots in the context of modern manufacturing environments? This problem calls for innovative approaches that blend industrial psychology, technology, and sustainability to facilitate a seamless transition into I5.0.
2. Related work
Technologies enabling human-machine interaction, such as computer vision, collaborative robotics, augmented and virtual reality, are part of the concept of Industry 4.0 (I4.0) and are currently widely used to support human employees and generate value. The relatively new human-centric concept of I5.0 is not based just on such technological advancements, but it is centered around core human values towards enabling a humancentric and ethical society. Although I5.0 proposes a major shift in the focus to humancentric approach, I4.0 technologies are still facing challenges for feasibility, which is one main thrust of this paper.
One major challenge that is often overlooked during implementation of digitalized solutions is workplace mental health, which should be considered on an equal footing as the physical health when designing new digitalized workplaces. The intensity of machinery in the workshops is expected to increase dramatically, hence replacing the human workers on the shop floor. Some workers will inevitably remain on the shop floor for more sensitive or cognitive tasks, which might lead to negative impacts for their mental health, such as increased feelings of loneliness and alienation. Workers may experience occupational stresses because of digitalization, mainly due to lack of control over their duties or lack of support [4]. In manufacturing environments, where robots and smart machines are rapidly integrated, the effects of these occupational stresses might be multiplied, ranging from physical symptoms like headaches and muscle tension to emotional and behavioral changes such as anxiety, irritability, and depression - affecting overall productivity [4].
According to Lu et.al. [4], one way to reduce mental stress in a collaborative setting is to enable a robot to proactively undertake actions responsive to the psychological states of workers. This reciprocal interaction dictates that not only do humans react to the actions of robots, but robots must also respond to human behavior to establish a robust communication channel [4]. If robots were able to infer employees’ internal mental stress and adjust their actions according to that state, it could potentially lead to a psychologically safer workplace as well as a more productive HRC. Nonetheless, it is worth noting that only a limited number of studies have explored the implementation of customized robot actions in response to employees’ psychological state within the context of collaborative tasks, as of the date of this research.
A closed-loop control system involving identification of the human’s psychological state by the collaborative robot as well as informed actions to the robot’s movement based on that state could improve various aspects of the workplace. One research shows that when working with the adaptive robot, participants completed various tasks much faster, with less human and robot idle time. In the same research, a questionnaire filled by the participants indicated that they felt safer and more comfortable when working with an adaptive robot and were more satisfied with it as a teammate [5].
The most common way in literature of identifying a human employee’s psychological state is through emotion recognition via computer vision. In [6] it is investigated how the signals expressed by the human operator can be collected and analyzed through deep learning algorithms. The authors compare the most used algorithms for Facial Emotion Classification and the results are briefly shown in Table 1 in descending order regarding their training accuracy scores.
Utilizing computer vision for emotion recognition might pose various challenges as there are a lot of unpredictable factors that might interfere with the accuracy of the result. Face angle, lighting, background environment, and wearables are some examples mentioned in literature [7]. Another issue that might occur while applying such algorithms in industry is the limited range of emotions typically displayed by workers around robots or machines. Workers often exhibit a neutral expression during task execution, reserving expressions of surprise or fear for unexpected events. Such emotions could signal either a production issue, necessitating a robot slowdown or stop, or an external event which does not pose a threat to the operator, in which case no action is needed from the robot. Addressing such challenges requires a more advanced context-aware intelligence [7].
Literature also explores open-loop systems alongside closed-loop control systems designed to identify and respond to workers' emotions. Some researchers highlight the effectiveness of visual reminders in alleviating occupational stress, employing techniques like Just-In-Time interventions [8].
Efforts to enhance HRC have examined techniques such as minimum-energy and minimum-jerk strategies to achieve smoother interactions. While these methods can improve workers' physical comfort to some extent, an exclusive focus on adapting and optimizing trajectories may inadvertently reduce predictability. Trajectories generated under these criteria are less predictable, potentially causing confusion and adding mental stress for workers [9]. In the context of HRC, it is essential for workers to be clearly informed of the intentions of robots. Robots must plan trajectories that are both psychologically acceptable and readable to workers.
A promising approach to reduce mental stress involves providing workers with notifications before executing high-risk activities [4]. Research discussed in [10] evaluates the impact of advanced notice of robot motions on human mental stress. Comparative experiments demonstrate that providing advance notice of the maximum speed of robot motion can effectively reduce workers' mental stress.
3. Research Methodology
This chapter describes the methods, tools, and processes that are acquired in this paper for research design, and data management. For the synthesis and analysis of the research, the Stage-Gate Process [11] was utilized to continuously reassess the objectives against available resources and practicality of implementation throughout various process stages.
3.1. Research Approach & Methods
The research of this paper utilizes an experimental approach; the authors follow an abductive approach to explain the findings of these experiments by providing reasonable explanations and proceeding in educated guesses to account for the collected evidence.
Two experiments were conducted during the pursuit of feasible solutions that could answer the research question. The first experiment employed a quantitative approach to data collection and analysis. The second experiment employed a mixed-methods research approach since both quantitative and qualitative data were collected.
3.2. Experimental Design
For the initiation of the design, the Stage-Gate model was utilized to have a standard procedure as guidance in the design and development phase. The first stage of the model is Discovery. The authors have confidence in the idea that the world manufacturing ecosystem is shifting towards I5.0 and global trends are pushing for more humanoriented conditions on the shop floor for maximized resource utilization and socially sustainable production. In search of an idea leaning towards these shifts, brief market research was conducted to identify the gaps in the field, through accelerator, seed, and challenge programs on the internet, with reference to Table 2.
There seems to be a gap in assessing and acting according to the psychological state of the employee and incorporating this information into the production flows. Based on the literature review and the market overview, the authors chose to develop digital solutions that take into consideration the stress and frustration levels of workstation employees, especially when they are working with autonomous machines and devices and provide an end-to-end service to reduce those negative emotions.
Several ideas emerged during the brainstorming phase. Upon reaching Gate 1 of the standardized process, the authors evaluated the equipment available in the lab, including microcontroller kits, sensors, and laboratorial facilities. It was concluded that the existing resources are sufficient for conducting feasible experiments, offering various implementation options for each idea and any scaling up of the prototype is possible with further availability of resources.
The experiments were meant to be set up in a laboratory to design technical solutions for improvement of industrial psychology in a production logistics environment where humans and robots collaborate. The focus was to design and develop the production system (human-robot psychological relation), not the movement of robots in the plant.
The first experiment was inspired by the approach described in [4]. The idea was to set up a network of proximity sensors at the laboratory which simulates a production logistics facility. An autonomous mobile robot (AMR) carrying materials from one station to another would perform pre-assigned missions around the provided space. A worker would be performing a task in the main workstation while a screen would provide real-time information on the estimated time of arrival (ETA) of the AMR at the workstation. The literature implies that providing workers with notifications before important events that will possibly affect them, such as an AMR arriving at the station to pick up something that is not yet ready to be picked up, reduces mental stress.
As for the second experiment, the authors attempted to achieve closed-loop cognitive communication between the AMR and the human worker by recognizing non-verbal cues, interpreting the emotions behind the cues, and acting accordingly. The experiment utilized computer vision to detect facial expressions of the employee. A machine learning (ML) algorithm was utilized to interpret facial expressions and trained with secondary data, open-source and available on the internet. The emotion label was fed back into the system to trigger the AMR to act accordingly, thus enhancing the work environment for the employee, reducing loneliness and cognitive stresses. The authors performed this experiment to design and model the behavior of the AMR to various human non-verbal cues, and did not evaluate stress levels of employees, which can be further researched to evaluate the psychological effects within a cyber-physical system (CPS).
Although it is not in the delimitations of this project, the authors aimed to make the system intelligent and connected by using output data from the first experiment as trigger information for the second experiment. More specifically, when the AMR is sufficiently close to the station, visual to the worker through the human-machine interface, a background connection triggers the webcam used for the computer vision to analyze emotional state of the worker, thus integrating the two experiments.
4. Prototype Development
To provide a robust digital solution, it is essential to develop a functional prototype of the end-to-end integrated service. A prototype serves to iteratively test the design and initial model of the product and service - allowing for the identification of flaws, limitations, and aspects that may not be apparent during the proof of concept. Moreover, a well-executed prototype serves as a tangible demonstration for stakeholders and potential investors, showcasing the value and potential of the service.
For this research, the experimental scenario took place in a laboratory at KTH Royal Institute of Technology in Södertälje, Sweden. The lab is simulating material handling facilities with several workstations. An AMR transfers materials and components between these workstations. The main assumption made for the first experiment is that the stations are far apart from each other, resulting in limited communication between the workers and sole collaboration with the AMR. The main assumption for the second experiment is that the ML algorithm is unbiased regarding race and gender. The authors assume that the sensors operate within the design and manufacturing parameters of the available ones in the laboratory, for example, a maximum range of 400cm. Both experiments are based on a prototype of one AMR, one main workstation and three passing stations.
4.1. Proximity Experiment
Among the available sensor options, ultrasonic sensors were chosen for this experiment. They were selected over laser sensors and other proximity sensors due to their wider angle of detection and suitability for the required application. It should be noted that the AMR already had position trackers installed, rendering other sensor technologies redundant. The experiment aimed to make a case for non-smart technologies by leveraging the existing capabilities of the AMR.
An Arduino circuit board was connected with the ultrasonic sensor and programmed to provide real-time distance of the moving AMR. The output was integrated into NodeRed, a visual programming tool, where a function was utilized to convert the real-time distance to ETA. Through several repetitions of a specific timeframe, primary quantitative data was collected and analyzed.
To present the collected data in a user-friendly manner, Node-Red's User Interface was utilized for visualization, as shown in Figure 1. This visual feedback system served as a warning mechanism to reduce work hazard risks for the station operator.
4.2. Emotion Detection Experiment
In the second experiment, the initial design was to install a camera on the AMR to capture non-verbal cues from human employees and utilize an algorithm to understand their emotional states. During a review through the Stage-Gate model, it was decided to utilize a laptop webcam installed in the workstation, for stable resolution. The webcam provided adequate resolution and compatibility with the ML model, ensuring reliable and accurate data capture for emotion recognition purposes. The author decided to employ a pretrained ML library, eliminating the need for training samples. The testing of the algorithm occurred through participatory observation. The algorithm was integrated into Node-Red and its output triggered the AMR to act accordingly. Primary data of facial expressions of varying physiological features, for example gender and race, were collected, preprocessed, and used to test the algorithm.
The emotional classes of the ML model were the following: happy, frustrated, and neutral faces. The extra features attributed through the model were gender and race; gender as in female and male categorization and race as in white, black, Asian.
To enable the AMR to respond appropriately to employee’s emotional states, appropriate response commands were written and aligned with the AMR's digital twin through NodeRed. These response commands facilitated the AMR's actions, ensuring its behavior was in line with the emotional cues detected from the employee. Node-Red enabled seamless integration between the ML model, AMR control, and data storage, serving as a centralized platform for managing workflow and ensuring smooth operation.
As for the correspondence of the AMR to the employee’s emotional state, three responses were deployed. In the first case when a happy face was detected, the AMR was approaching the station and, with the use of natural language, it responded “Oh! You look happy today! What’s on the menu?”, creating a discourse. In the case of a neutral face, the AMR just completed its mission and moved on. In the final scenario where the employee was frustrated, angry, or sad, the AMR kept a safe distance and responded in natural language syntax like “I hate to see you like this. What’s wrong? Do you want me to give you some space?”.
5. Results
The prototype results are described in detail in this video link, which the authors created and uploaded with the results from the experiments conducted in the lab.
Experiment 1 produced an output dataset in excel format, which had the structure obvious in Table 3.
Experiment 2 produced another output dataset in excel format as well with the structure of Table 4.
These outputs can be used in further research into Data Analytics and the development of an intelligent system.
6. Discussion
Through the proposed integrated solution, comprehensive tracking and tracing of shopfloor key performance indicators (KPIs) becomes feasible. These KPIs might include productivity, resource efficiency, capacity utilization, ergonomic deviations, automation level, workload variation, and employee work-related stresses. Some of these KPIs can be used to determine the environmental impact of the solution. For instance, while the energy consumption of the entire system falls outside the scope of this paper, monitoring and controlling the energy usage of the system are important for evaluating its environmental impact, particularly in terms of SCOPE 2 and 3 emissions. Further research is needed to address this aspect and fill the existing knowledge gap.
During digital transformation, careful consideration should be given to both greenfield and brownfield opportunities and their implications for system integration and operability. It is crucial to ensure that the installation of infrastructure aligns with existing platforms and corporate production systems, avoiding the creation of multiple platforms within a single system, which would lead to inefficiencies and waste. While digitalization may seem appealing, it should be driven by value creation, aligning with digitalization objectives, organizational competencies, employee competencies, and long-term goals. It is essential to recognize that the successful launch of I5.0 relies heavily on employee engagement and interaction. Without active involvement and participation from employees, the full potential of I5.0 cannot be realized.
Organizational culture plays a pivotal role in the successful implementation of digital transformation, as it requires a collaborative mindset. If this mindset is lacking within the organization, it becomes necessary to initiate a fundamental change in mindset to create an environment conducive to development. This process can begin by appointing a transformation champion who will be responsible for driving the change and fostering accountability. The champion can be supported by a team, either internal or external consultants specializing in change management, to facilitate the organization's exploration of new possibilities. The designated team plays a crucial role in mobilizing and allocating resources for the transformation process, while also providing regular updates and reports to senior management and shareholders. Employee engagement and interaction are of utmost importance as the production system is inherently a sociotechnical system. It is necessary to communicate with and gain the consent of employees regarding the increased visibility that accompanies digital transformation, while also respecting their privacy concerns.
Before embarking on digitalization, it is crucial to eliminate waste through lean techniques. This helps prevent the compounding of corporate waste through uncontrolled digitalization efforts. Coordinated initiatives are necessary to avoid duplicated efforts and excessive investments, aligning with the principles of smart factories. Integrating technology-driven smart factory initiatives with existing production systems is essential. Integration of the proposed solution with the AMR interface is particularly important in this case. Additionally, integrating the system with Manufacturing Execution System, Enterprise Resource Planning, Production Planning System, and Inventory Management System is invaluable in creating a unified platform and system, reducing task execution time and minimizing waste.
To address the challenges posed by remote factories and the volatility of Wi-Fi bandwidth, it is crucial to incorporate offline programming, visualization, simulation, and control features into the project. This feature allows for uninterrupted operations and mitigates the reliance on real-time data flow, which can be affected by limited connectivity. To ensure the robustness of the production system and handle large volumes of data flow with high velocity and storage requirements, the implementation of industrial CPS is necessary. These systems enable efficient handling of data and ensure the smooth operation of the overall system.
Data privacy encompasses aspects such as autonomy, the desire for privacy, and data ownership, which vary according to individual customer preferences and business considerations. Privacy measures should encompass data collection, processing, storage, usage, and disposal, ensuring that customer rights are respected throughout the data lifecycle. Both employees and the organizations they belong to should have rights that include transparency, access, objection, restriction of processing, the right to be forgotten, and the right to be informed. The presence of multiple parties involved in data governance introduces further complexity to the equation. It is imperative to acknowledge that, another option is just to restrict access of personal data to management and strictly giving access to the employee, for their edification and correctly assessing, monitoring and controlling their own wellbeing.
Although the results are promising, it is important to acknowledge that the human-robot interaction will result in less productivity as the AMR is programmed to understand nonverbal cues and initiate a smooth discourse and sometimes wait for appropriate human response to move forward. In the long run, when employees understand the costs of their cognitive stress to productivity might fake their non-verbal cues or actually try to be happy always, to ascertain the difference between the two might be a challenge.
7. Conclusion
In conclusion, this paper presents an end-to-end digital service with two integrated solutions for monitoring and improving the mental condition of employees on the shopfloor, enabling the evaluation of productivity in relation to ergonomic deviations and work-related cognitive stress. It is possible to digitalize the psychological state of the worker and optimize production operations, although it presents its own challenges, such as high capital expenditure, latency for scalability, technical competencies gap, limited computational power and an increase in power consumption.
To ensure robustness and reliability of the production system, the paper suggests the implementation of industrial CPS capable of handling high volumes of real-time data, high velocity, and storage. Additionally, the inclusion of features for off-line programming, visualization, simulation, and control is proposed to address issues related to remote factories and volatile Wi-Fi bandwidth.
The authors emphasize that employee engagement and interaction are vital for the successful launch of I5.0. Organizational culture plays a significant role in digital transformation, necessitating a collaborative mindset and fundamental mindset change within the organization. A transformation champion and a dedicated team are recommended to drive the exploration and resource allocation process, reporting to senior management and shareholders.
Possible areas of research include the integration of the AMR with the psychological natural language processing artificial intelligence to maintain a near human natural discourse between the shop-floor worker and the AMR, this will greatly reduce stress levels. Another possible area of research is network agnostics of different domains, dimensions, and functional areas of the organization in developing an intelligent smart production system which is connected, adaptive and prognostic, and can pass recommender systems on all domains with a systematic approach.
Acknowledgements
The authors would like to seize this opportunity to thank their professors for taking their time and effort to guide, mentor, and teach with regards to digitalization. It is important to acknowledge that without the availability of the KTH Logistics Laboratory and CPS equipment the success of this project would never have been realized.
References
[1] S. Nahavandi, “Industry 5.0—A Human-Centric Solution,” Sustainability, vol. 11, p. 4371, Aug. 2019, doi: 10.3390/su11164371.
[2] C. Ghislieri, M. Molino, and C. G. Cortese, “Work and Organizational Psychology Looks at the Fourth Industrial Revolution: How to Support Workers and Organizations?,” Front. Psychol., vol. 9, p. 2365, Nov. 2018, doi: 10.3389/fpsyg.2018.02365.
[3] V. Leso, L. Fontana, and I. Iavicoli, “The occupational health and safety dimension of Industry 4.0: Industry 4.0 and occupational health,” Med. Lav. Work Environ. Health, vol. 109, no. 5, pp. 327–338, Oct. 2018, doi: 10.23749/mdl.v110i5.7282.
[4] L. Lu, Z. Xie, H. Wang, L. Li, and X. Xu, “Mental stress and safety awareness during human-robot collaboration - Review,” Appl. Ergon., vol. 105, 2022, doi: 10.1016/j.apergo.2022.103832.
[5] P. A. Lasota and J. A. Shah, “Analyzing the Effects of Human-Aware Motion Planning on CloseProximity Human–Robot Collaboration,” Hum. Factors, vol. 57, no. 1, pp. 21–33, 2015, doi: 10.1177/0018720814565188.
[6] A. Baratta, F. Longo, G. Mirabelli, A. Padovano, and V. Solina, “A Digital Twin-Based Approach for Emotion Recognition in Human-Robot Collaboration,” in Towards a Smart, Resilient and Sustainable Industry, Y. Borgianni, D. T. Matt, M. Molinaro, and G. Orzes, Eds., Cham: Springer Nature Switzerland, 2023, pp. 155–166.
[7] A. Chiurco et al., “Real-time Detection of Worker’s Emotions for Advanced Human-Robot Interaction during Collaborative Tasks in Smart Factories,” Procedia Comput. Sci., vol. 200, pp. 1875–1884, 2022, doi: https://doi.org/10.1016/j.procs.2022.01.388.
[8] Smyth and Heron, “Is providing mobile interventions ‘just-in-time’ helpful? an experimental proof of concept study of just-in-time intervention for stress management,” in 2016 IEEE Wireless Health (WH), 2016, pp. 1–7. doi: 10.1109/WH.2016.7764561.
[9] F. Dehais, E. A. Sisbot, R. Alami, and M. Causse, “Physiological and subjective evaluation of a human– robot object hand-over task,” Appl. Ergon., vol. 42, no. 6, pp. 785–791, 2011, doi: https://doi.org/10.1016/j.apergo.2010.12.005.
[10] T. Arai, R. Kato, and M. Fujita, “Assessment of operator stress induced by robot collaboration in assembly,” CIRP Ann., vol. 59, no. 1, pp. 5–8, 2010, doi: https://doi.org/10.1016/j.cirp.2010.03.043.
[11] R. G. Cooper, “Stage-gate systems: A new tool for managing new products,” Bus. Horiz., vol. 33, no. 3, pp. 44–54, 1990, doi: 10.1016/0007-6813(90)90040-I