Overview

• Introduction
• Types of Studies
• Document Review and Evidence Approval
• Evidence Strengths

In Phase 2 of the study, we found and recorded evidence related to each issue identified as a problem or concern in Phase 1 (see Phase 1 Report for more information). Unlike in Phase 1, where we considered any assertion that a problem or concern exists as satisfactory grounds for recording a citation, in Phase 2 we looked for clear, documented evidence supporting or contradicting the issue statement.

• data from an experiment showing that a significant number of line pilot subjects demonstrated inadequate knowledge to successfully perform tasks in a simulator scenario (supportive evidence), or
• data from an experiment showing that a significant number of line pilot subjects demonstrated adequate knowledge to successfully perform tasks in a simulator scenario (contradictory evidence).
• results of a survey in which a significant number of line pilots claimed adequate knowledge of automation (contradictory evidence), or
• the conclusions of an accident investigation board that the flightcrew of the accident aircraft did not understand the automation and that lack of understanding was contributory to the accident (supportive evidence).

Our analysis consisted of reviewing documented studies potentially yielding evidence related to one or more issues:

• Accident Analysis
• ASRS Incident Report Analysis
• Analysis of Incident Studies
• Analysis of Observational Studies, Experiments, and Surveys

We identified these studies in several ways. First, our Phase 1 review yielded a large bibliography of documents related to flight deck automation. Since we were familiar with these documents, having reviewed them for citations of problems and concerns, we knew which ones described studies containing evidence. These documents in turn led us to other documents describing other studies. Our own survey of experts yielded evidence, plus the experts we surveyed in many cases identified documented studies on which they based their responses. Our Phase 1 incident analysis yielded many Aviation Safety Reporting System incident reports potentially containing evidence for issues. Reviews of news reports and accident report abstracts helped us identify accident reports that potentially contained evidence. Finally, by monitoring the current literature, attending professional conferences, and consulting our colleagues in the field, we supplemented our list of documented studies.

We carefully read the documentation from each study, looking for information meeting the criterion for evidence described above. When we found it, we recorded the following in our database:

• the issue supported or contradicted by the evidence
• the study which yielded the evidence
• the excerpt from the document which described the evidence
• the aircraft to which the evidence applied, if specified, such as
  • Airbus A320
  • Boeing B737-300
  • McDonnell Douglas MD11
• the equipment to which the evidence applied, if specified, including
  • autoflight:
    • autothrust
    • autopilot
  • EICAS (Engine Indication and Crew Alerting System - Boeing)
  • ECAM (Electronic Centralized Aircraft Monitor - Airbus)
  • EFIS (Electronic Flight Instrument System)
  • FMS (Flight Management System)
  • flight control (e.g. fly-by-wire)
  • automation (general or not specified)
• a numeric value indicating the strength of the evidence supporting or contradicting the asserted issue (see Evidence Strengths section)
• the name of the analyst

Each evidence record in the database also contained fields for analyst's notes and two approval fields. Both principal investigators (Lyall and Funk) reviewed each evidence record, checking the excerpt against the criterion for evidence, checking the excerpt and other data for completeness and correctness, correcting the record where necessary, and marking the approval fields accordingly. Consequently, each evidence record was reviewed at least three times: once by the original analyst and once each by both principal investigators. Through this process we documented what we consider to be a sound body of evidence related to the issues.

We found both supportive and contradictory evidence related to the flight deck automation issues, and some evidence was stronger than others. Although comparing the strength of two instances of evidence related to an issue from the same pilot survey, for example, was relatively straightforward, the same cannot be said for comparing the strength of evidence from, say, a pilot survey and an accident report. To try to make the assignment of strengths as objective and consistent as possible, we developed a set of strength assignment rules for each type of study. Evidence strength can range from -5, for strongly contradictory evidence, to +5, for strongly supportive evidence. Although a strength of 0 could be used for evidence that both supported and contradicted an issue, no such evidence was recorded. Rather, for studies with results supporting both sides of an issue, we recorded two instances of evidence, one supportive and one contradictory. The following table summarizes our strength assignment rules. Details are provided on the pages describing our studies.

Evidence Strengths	Experiments, Observation Studies, Surveys, and Incident Studies	Surveys	Experiments to Test Hypotheses			Accident Reports
	subjects' performance, respondent responses, or report content	mean response as a % of max possible (i.e., full agreement with issue statement) rating	subjects	tasks	results	statement of investigating board's findings, as compared to problem suggested by issue statement	contribution of problem to accident
+5	90-100% supportive	91-100% (strongly agree with issue)	line pilots trained on equipment	line operations	support issue	equivalent	probable
+4	75-89% supportive	81-90%	line pilots trained on equipment	simulated line operations	support issue	equivalent	possible
+3	50-74% (or 'most') supportive	71-80% (agree with issue)	other line pilots	simulated line operations	support issue	similar	probable
			line pilots trained on equipment	other simulated flight tasks
+2	25-49% supportive	61-70%	other line pilots	other simulated flight tasks	support issue	similar	possible
			GA/student pilots	simulated line operations
+1	1-24% (or 'some') supportive	50-60% (slightly agree with issue)	line pilots	generic automation tasks	support issue	analyst unsure	probable or possible
			GA/student pilots	other flight automation tasks		any of above	analyst unsure
-1	1-24% (or 'some') contradictory	40-50% (slightly disagree with issue)	line pilots	generic automation tasks	contradict issue
			GA/student pilots	other flight automation task
-2	25-49% contradictory	30-39%	other line pilots	other simulated flight tasks	contradict issue
			GA/student pilots	simulated line operations
-3	50-74% (or 'most') contradictory	20-29% (disagree with issue)	other line pilots	simulated line operations	contradict issue
			line pilots trained on equipment	other simulated flight tasks
-4	75-89% contradictory	10-19%	line pilots trained on equipment	simulated line operations	contradict issue
-5	90-100% contradictory	0-9% (strongly disagree with issue)	line pilots trained on equipment	line operations	contradict issue

 The reader should exercise some discretion in interpreting strength values. These values reflect our assessments of the extent to which an instance of evidence supports one side or the other of an issue and, as such, are relative assessments. While we found this process extremely useful in analyzing, comparing, and organizing evidence related to the issues, we make no claim that these strengths have universal validity. We urge the reader to make his/her own assessment of how much significance to attach to a particular instance of evidence and to use our numbers merely as a relative guide.