Overview

• Introduction
  • Experiments and Observation Studies
  • Surveys
• Results
  • Experiments
  • Observation Studies
  • Surveys

We reviewed documentation from various automation-related experiments, surveys (including our own), observation studies, andother studies that represented a combination of approaches. In these documents we identified and recorded supportive and contradictory evidence related to the flight deck automation issues in 24 experiments, 29 surveys, and 15 observational studies. Strength ratings depended on the type of study reviewed, the methodology and type of subjects used in the study, and the type of evidence yielded by the study. Details are described below.

Some of the studies we reviewed were experiments conducted in simulators or in laboratories. Others were observation studies in which an observer observed pilots in simulators or in flight operations. In such studies, subjects could perform in a manner consistent with or contradictory to the problem suggested by an issue statement. We reviewed these studies for such supportive and contradictory evidence. Where the results of such studies were reported as percentages of subjects performing in a manner consistent with or in a manner contradictory to an issue, we assigned strengths according to the following table.

 For example, given an experiment in which 25% of the line pilot subjects did not know how to perform an important flight management system operation we would have recorded two instances of evidence related to issue105 (understanding of automation inadequate), one supportive and one contradictory. The first would have been recorded with strength +2 for those pilots who lacked understanding, the second with strength -4 for the remaining pilots.

In such a study in which we could not determine exact percentages, we used the following table to assign strengths.

For example, if 'most' of the data from such a study supported (or contradicted) the problem suggested by the issue statement of issue105, we would have recorded evidence for it with strength +3 (or -3). If 'some' of the data supported (or contradicted) issue105, we would have recorded evidence for it with strength +1 (or -1).

Some experimental studies we reviewed tested hypotheses. For each of these we based evidence strength on the type of subjects used in the experiment, the type of tasks the subjects performed, and the results. We used the following table to assign strengths.

Subjects in such a study could be line pilots actually trained on the equipment used in the experiment, other line pilots (not trained on the equipment used in the experiment), general aviation (GA) pilots, or student pilots. The tasks the subjects performed in the experiments could conceivably be tasks performed in actual line operations or they could be tasks performed in simulated line operations, simulated flight tasks in part-task simulators, or generic automation tasks performed in laboratories. The results could either be supportive of or contradictory to an issue.

For example, consider an experiment conducted to test the hypothesis that flightcrews respond more quickly to air traffic control (ATC) clearances when flying the airplane manually than when using the flight management system (FMS). The experiment involves line pilots using a part-task simulator modeling equipment they have been trained on. Each flies several scenarios, flying half of them manually and half with the FMS, and responds to ATC clearances. The results show that mean time to begin complying with an ATC clearance takes, on the average, 4.5 seconds manually and 8.1 seconds with the FMS, and that the difference is statistically significant at the p = 0.0963 level (good statistical significance for this type of experiment). This would be supportive evidence for issue161, When using automation, pilot response to unanticipated events and clearances may be slower than it would be under manual control, possibly increasing the likelihood of unsafe conditions. We would rate the strength of this evidence as +3 (line pilots trained on equipment, other simulated flight tasks, supportive of issue).

In some of the surveys we reviewed, respondents were asked to rate their level of agreement with assertions equivalent to our issue statements. We assigned strengths based on the percentage of respondents agreeing with or disagreeing with these assertions, according to the following table.

For example, suppose that, in such a survey, 63% of the subjects agreed or strongly agreed with the assertion "Overall, the flight management system reduces workload." We would have recorded evidence related to issue079 (automation may adversely affect pilot workload) with strength -3, for these results contradict the problem suggested by issue079's issue statement. In this case we would not have recorded supportive evidence, because the fact that some of the respondents (maybe as many as 37%) disagreed that the FMS reduces workload does not mean that they think that it actually increases workload.

In some of the surveys we reviewed, subjects were asked to respond to assertions equivalent to our issue statements by giving their level of agreement as Likert scores (e.g., 1 means strongly disagree, 5 means strongly agree). When the results were given as a mean score as a percentage of the maximum possible score (a percentage of 5, in the example), we used the following table to assign strengths to evidence.

The table is based on the assumptions that scores could range from 0% of maximum score (for strongly disagree) to 100% of maximum score (for strongly agree), that the survey question was worded consistently with the asserted issue, and that 50% of maximum score was neutral (did not agree or disagree).

If the assertion to which the subjects responded was worded to be opposite to that of the issue statement, we reversed the strength signs. Unless response distribution information was given, we did not count evidence both for and against the issue unless minimum and maximum responses were given. For example, consider a survey in which pilot respondents were asked to give their level of agreement with the assertion "Pilots fully understand the flight management system." If the mean response was 1.9 on a scale of 1 (strongly disagree) to 5 (strongly agree), we would have recorded evidence with strength +2 (1.9 is 38% of 5) since the subjects as a group tended to disagree with the survey statement, thereby agreeing with the issue statement of issue105. If it was also reported that the maximum response was 5, we would also have recorded evidence with strength -1, since at least one subject strongly disagreed with the issue105.

In surveys in which we could not determine exact percentages, we used the following table to assign strengths.

For example, if 'most' of the respondents from such a study agreed with an assertion consistent with the issue statement of issue079, we would have recorded evidence for it with strength +3. If 'some' of the respondents agreed with the assertion, we would have recorded evidence for it with strength +1.

We found evidence for flight deck automation issues in 24 of the experiments we reviewed. The experiments are listed alphabetically by author and include links to the bibliographic information and evidence found in the report.

We found evidence for the flight deck automation issues in 15 of the observation studies we reviewed. The observation studies are listed alphabetically by author and include links to the bibliographic information and evidence found in the report.

We found evidence for the flight deck automation issues in 29 of the surveys we reviewed. The surveys are listed alphabetically by author and include links to the bibliographic information and evidence found in the report.