Towards a Cognitive Approach to Feature Model Comprehension
Purchased from Gettyimages. Copyright.
Software Product Line (SPL) is a new discipline in software engineering designed to answer the needs of fast-growing markets by developing software systems from reusable software assets instead of developing them from scratch. Product Line is a well-established method among large companies like General Motors, Toshiba, BMW, and Nokia. These companies enhance their services by applying the SPL approach. Indeed, SPL is a family of related software systems in which each system is differentiated by specific components. Designers in the early stage of SPL development use feature models (See Figure 1) with tree-shape structures to manage differences and similarities in various software systems. Feature models play a critical role in obtaining valid software products in the SPL cycle. We studied feature model comprehensibility with the help of eye-tracking technology on the eyes of developers.
Figure 1: Example of Feature Model for laptop product line
Evaluating the Cognitive Effort of Developers
In this exploratory study, our goal was to observe the challenges software developers are faced with when attempting to understand feature model tasks. Feature models used in our study were different in terms of size and number of cross-tree constraints (Numbers 4, 3 in Figure 2). According to our hypotheses, developers needed to understand the feature model, cross-tree constraints formula, and questions (numbers 4, 3 and 6 in Figure 2) in order to complete the comprehension tasks. While they were performing these tasks, we tracked and collected the following data: gaze points (using eye-tracker software), spent time and accuracy of answers.
As a result of this study, we expect to achieve three distinct mental models associated with these areas, which will also confirm our hypotheses. By increasing the size of the feature model and cross-tree constraints, we assume that participants will exert more cognitive effort and have a difficult time completing tasks. We applied a series of statistical analyses on collected data to support the study’s hypotheses and observed how participants build mental models. To collect gaze data, we first defined areas in tasks that we are most interested in studying. We used Heat-Map as one of the visualization techniques provided by eye-tracker software to accomplish this (see Figure 3).
Researchers have extensively studied program comprehension (source code) to understand how software developers read and debug code. They applied different technologies to study human mind such as Eye-tracking, EEG, and FMRI. Using these technologies helped them to visualize the pattern software developers follow to understand codes and use them to improve programming tactics and tools . The study of the human mind is also important to other disciplines such as Psychology and Human-Computer Interaction which strive to create solutions to difficult problems based on the study of the mind. Using mental models, we can better understand how humans’ minds work when faced with new challenges. Mental models include the way you perceive something or the way it is supposed to work. Applying the same approach, this study also sought to understand how mental models for feature model comprehensibility develop in participants’ minds when completing designed tasks. We show that researching feature model comprehension can inform language design and tool development and will provide more suitable language structures, user interfaces and support for these kinds of models.
Figure 2: 1. Button, 2. Answer, 3. Cross-tree constraints, 4. Feature model, 5. Legend, 6. Question, 7. Window- Taken from Elmira et al.(2022)
Eye-Tracking Technology
In this study we used eye-tracking technology for the first time as a method to study human behaviour and investigate feature model comprehension. An eye tracker is a sensor device that measures eye movement, eye position, and pupil dilation to record data obtained from participants looking at an object of study, which can be physical (smartphones) or digital (webpages). The participants in this study were required to answer 24 tasks on the comprehensibility of feature models. Identifying an answer began with understanding the question requirements, followed by understanding the feature model.
Figure 3: Participant gaze pattern using visualization technique (Heat-Map)
Factors Increasing the Cognitive Effort
To explain the cognitive load, process, and effort associated with SPL tasks that use feature models, we obtained mental models formed in participants’ minds by analyzing their gaze data and the data collected based on spent time and accuracy of answers for each task. From a cognitive perspective, our designed tasks involved building, manipulating, and comparing mental models of different task components that intervene in their resolution. Results of this study show, first, tasks in which the size of feature models and cross-tree constraints increase require more cognitive operations to answer correctly as response time increases. Second, by spending more time and attention on a feature model area, the likelihood of providing an incorrect answer increase. Finally, results show participants spent most of their time and attention on understanding the feature model, cross-tree constraints formula, and question areas. These findings support and confirm our hypotheses.
Contribution
For the first time, we conducted an empirical study of feature model comprehension for a simple comprehension task using eye-tracking technology. We proposed the foundations of a cognitive model for feature model comprehension containing three distinct mental models. Their relevance was attested by the collected gaze data obtained on the corresponding areas most visited by participants (feature model, cross-tree constraints, and question).
Additional Information
For more information on this research, please read the following conference paper: Rezaei Sepasi, E; Nezami Balouchi, K.; Mercier, J.; Lopez-Herrejon. R. E. 2022. Towards a Cognitive Model of Feature Model Comprehension: An Exploratory Study using Eye-Tracking, SPLC ’22: Proceedings of the 26th ACM International Systems and Software Product Line Conference – Volume A September 2022. Pages 21–31. https://doi.org/10.1145/3546932.3546995
