National Aeronautics and Space Administration

International Space Station (ISS) EVA Suit

Water Intrusion
.
High Visibility Close Call
IRIS Case Number: S201319900005

Date of Mishap: July 16, 2013

Date of Report: December 20th, 2013
This page intentionally left blank.
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Table of Contents

ACKNOWLEDGMENTS ............................................................................................................... 1

1.0 CHARTER AND RESPONSE.......................................................................................... 3

1.1 Signature Pages .................................................................................................................................. 3

1.1.1 Investigating Authority Signatures .............................................................................................. 3
1.1.2 Ex Officio Signature ..................................................................................................................... 4
1.1.3 Advisor Signatures ...................................................................................................................... 5

1.2 Transmittal Letter ............................................................................................................................... 7

1.3 Appointment Letter ............................................................................................................................ 8

1.4 Executive Summary .......................................................................................................................... 12

2.0 NARRATIVE DESCRIPTION AND FACTS .............................................................. 23

2.1 Overview .......................................................................................................................................... 23

2.2 Background ....................................................................................................................................... 23

2.2.1 Mission Control Center Team .................................................................................................... 24

2.3 Events Immediately Leading Up to Mishap ...................................................................................... 26

2.4 Events at Time of Mishap ................................................................................................................. 27

2.5 Events Following the Mishap including Secondary Undesired Outcome...................................... 28

3.0 DATA AND ANALYSIS ................................................................................................ 30

3.1 Approach .......................................................................................................................................... 30

3.2 Type of Data Gathered: List of Documents ....................................................................................... 30

3.3 Timeline ............................................................................................................................................ 31

3.4 Extravehicular Mobility Unit (EMU) Operational Description........................................................... 32

3.4.1 System Level EMU Description .................................................................................................. 32
3.4.2 Ventilation Circuit ..................................................................................................................... 37
3.4.3 Cooling Loop ............................................................................................................................. 40
3.4.4 Water Separator Loop ............................................................................................................... 42
3.4.5 ISS EMU Coolant Loop ............................................................................................................... 44
3.4.6 Airlock Cooling Loop Recovery (ALCLR) Kit ................................................................................ 45
3.4.7 Maximized EMU Ground Activity (MEGA) Certification Process ................................................ 46

3.5 Concurrent Investigations................................................................................................................. 47

iii
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

3.5.1 Initial Activities ......................................................................................................................... 47

3.5.2 Initial Investigative Steps .......................................................................................................... 51
3.5.3 Condition of the Other EMUs On Orbit..................................................................................... 54
3.5.4 Specific Testing of the EMU 3011 Water Separator Circuit ........................................................ 55
3.5.5 Removal and Replacement of the Item 134 Valve and Filter, and Subsequent Retest ............... 57
3.5.6 Removal and Replacement of the Item 141 Gas Trap Cartridge, and Subsequent Retest........... 58
3.5.7 Ground-based Analysis of the Item 134 Filter and Valve, and the Item 141 Gas Trap ................ 60
3.5.8 Removal and Replacement of the Item 123 Fan-Pump-Separator ............................................. 61
3.5.9 Post R&R Verification and SEMU 3011 Screening ...................................................................... 66
3.5.10 Post-flight Evaluation Planning.................................................................................................. 68
3.5.11 Post-flight Evaluation of the FPS and ALCLR Filters .................................................................... 68
3.5.12 Forward Planning ...................................................................................................................... 76

3.6 Root Cause Analysis .......................................................................................................................... 78

3.6.1 Proximate Causes...................................................................................................................... 80
3.6.2 Intermediate Causes, Root Causes and Contributing Factors..................................................... 82

3.7 Observations................................................................................................................................... 116

4.0 FINDINGS .................................................................................................................... 140

4.1 Proximate Causes ........................................................................................................................... 140

4.2 Intermediate Causes ....................................................................................................................... 140

4.3 Contributing Factors ....................................................................................................................... 141

4.4 Root Causes .................................................................................................................................... 142

4.5 Observations................................................................................................................................... 142

5.0 RECOMMENDATIONS .............................................................................................. 145

APPENDIX A: HUMAN FACTORS ANALYSIS ................................................................ A-1

APPENDIX B: LIST OF DOCUMENTS .............................................................................. B-1

APPENDIX C: DEFINITIONS .............................................................................................. C-1

APPENDIX D: ACRONYMS ................................................................................................ D-1

APPENDIX E: DETAILED TIMELINE ............................................................................... E-1

APPENDIX F: EMU SCHEMATICS ..................................................................................... F-1

APPENDIX G: ISS INVESTIGATION TEAM FAULT TREE .......................................... G-1

iv
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

APPENDIX H: EMU EVOLUTION HISTORY (H2O ORIENTED) ............................... H-1

APPENDIX I: EVENT AND CAUSAL FACTOR TREE ...................................................... I-1

List of Figures
Figure 1-1. Extravehicular Mobility Unit (EMU) With Water in Helmet During Post-EVA 23 Screening
Test ................................................................................................................................................... 12
Figure 2-1. EVA Communication Paths.................................................................................................... 25
Figure 3-1. NASA EMU High-Level Description Illustrating Components of EMU Pressure Garment and
Life Support Assemblies ................................................................................................................... 33
Figure 3-2. Extravehicular Mobility Unit Components ............................................................................. 34
Figure 3-3. Annotated EMU Schematic .................................................................................................... 36
Figure 3-4. Item 140, Sublimator Vent Flow Path .................................................................................... 38
Figure 3-5. Ventilation pad.................................................................................................................... 39
Figure 3-6. HUT Water and Ventilation Lines .......................................................................................... 39
Figure 3-7. Liquid Cooling Ventilation Garment (LCVG) Ducts ............................................................. 40
Figure 3-8. Sublimator Cooling Loop Cross Section ................................................................................ 41
Figure 3-9. Sublimator Slurper ................................................................................................................. 41
Figure 3-10. Fan Pump Separator (I-123) Cross Section........................................................................... 42
Figure 3-11. Water Separator Operation ................................................................................................... 43
Figure 3-12. Gas Trap Assembly (I-141) .................................................................................................. 44
Figure 3-13 Simplified Schematic of ISS Cooling Loop and EMU .......................................................... 45
Figure 3-14 ALCLR Kit............................................................................................................................ 46
Figure 3-15 Iodination Kit ........................................................................................................................ 46
Figure 3-16. Disposable In-suit Drink Bag (DIDB) .................................................................................. 48
Figure 3-17. LCVG Tubing and Connector .............................................................................................. 49
Figure 3-18. Water Separator Spinning Drum and Pitot Components....................................................... 50
Figure 3-19. Check Valve and Filter I-134 ............................................................................................... 50
Figure 3-20. Pump Priming Valve I-125 ................................................................................................... 51
Figure 3-21. EMU SOP Check Out Fixture (SCOF) ................................................................................. 52
Figure 3-22. SOP Regulator Shown in OFF Position ............................................................................. 53
Figure 3-23. EMU 3011 Screening Test Data ........................................................................................... 56
Figure 3-24. Item 134 Valve and Filter from EMU 3011 Post-Anomaly .................................................. 57
Figure 3-25. EMU 3011 Screening Test Data Post I-134 R&R ................................................................ 58

v
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Figure 3-26. Item 141 Gas Trap from EMU 3011 Post-Anomaly ............................................................. 59
Figure 3-27. Item 141 Gas Trap from EMU 3011 Post-Anomaly Inspection ............................................ 59
Figure 3-28. EMU 3011 Screening Test Data Post I-141 R&R ................................................................ 60
Figure 3-29. Wire Insertion from TP B ..................................................................................................... 62
Figure 3-30. Absorbent Plug in Bore ........................................................................................................ 63
Figure 3-31. Areas of Concern for Fan Strikes ......................................................................................... 63
Figure 3-32. Fan Outlet to Sublimator Inlet Adaptor ................................................................................ 65
Figure 3-33. Face Seal O-ring ................................................................................................................... 66
Figure 3-34. Comparison of Fan Motor Speed and UIA Current Before and After I-134, I-141, and I-123
Replacement ..................................................................................................................................... 67
Figure 3-35. Pitot Detail ........................................................................................................................... 70
Figure 3-36. Contamination Site at Drum ................................................................................................. 70
Figure 3-37. Contamination in Drum ........................................................................................................ 71
Figure 3-38. Typical of 8 Blocked Holes .................................................................................................. 72
Figure 3-39 Residue on Pump Housing and Corroded Braze Joint ........................................................... 72
Figure 3-40. Contamination At and In Drum Hole ................................................................................... 73
Figure 3-41. N-Ray of Pitot ...................................................................................................................... 74
Figure 3-42. FPS Performance Test Schematic ......................................................................................... 75
Figure 3-43. ECFT Proximate Causes....................................................................................................... 80
Figure 3-44 Intermediate and Root Causes under ECFT-1 ....................................................................... 82
Figure 3-45 Intermediate Cause Directly Under ECFT-2 ......................................................................... 86
Figure 3-46 Intermediate Causes Directly Under ECFT-3 ........................................................................ 87
Figure 3-47 Direct Intermediate Causes and Contributing Factor Under ECFT-3.1 ................................. 88
Figure 3-48 Causes Under ECFT 3.1.1 ..................................................................................................... 89
Figure 3-49 Intermediate Causes Under ECFT-3.1.2 ................................................................................ 91
Figure 3-50 Causes and Contributing Factors under ECFT 3.1.2.1.1 ........................................................ 93
Figure 3-51. Causes and Contributing Factors under ECFT 3.1.2.1 for reference..................................... 96
Figure 3-52 Cause under ECFT 3.1.2.1.3.................................................................................................. 97
Figure 3-53 Root Cause and Contributing Factors under ECFT 3.1.2.1.4................................................. 98
Figure 3-54 Causes under ECFT 3.1.3 ...................................................................................................... 99
Figure 3-55 Causes under ECFT 3.1.4 .................................................................................................... 102
Figure 3-56 Causes and Contributing Factors under ECFT 3.1 for reference ......................................... 103
Figure 3-57 Causes and Contributing Factors under ECFT 3.2............................................................... 104
Figure 3-58 Causes under ECFT 3.2.1 .................................................................................................... 104
Figure 3-59 Causes and Contributing Factors under ECFT 3.2.1.1 ......................................................... 105

vi
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Figure 3-60 Causes and Contributing Factors under ECFT 3.2.1.2 ......................................................... 106
Figure 3-61 Causes and Contributing Factors under ECFT 3.2.1.3 ......................................................... 106
Figure 3-62 Causes and Contributing Factors under ECFT 3.2.for reference ......................................... 107
Figure 3-63. Contributing Factors under ECFT 3.2.2 ............................................................................. 108
Figure 3-64. Contributing Factors under ECFT 3.2.2.1 .......................................................................... 108
Figure 3-65 Contributing Factors under ECFT 3.2.2 .............................................................................. 111
Figure 3-66 Contributing Factors under ECFT 3.2.2.3 ........................................................................... 112
Figure 3-67. Contributing Factors under ECFT 3.2.2 for reference ........................................................ 113
Figure 3-68. Contributing Factors under ECFT 4 for reference .............................................................. 114
Figure 3-69 Projected Future Maintenance EVAs .................................................................................. 135

List of Tables
Table 2-1. Mission Control Center Teams/Positions ................................................................................. 25
Table 3-1. Short Timeline of EVA 23 Real-time Events........................................................................... 31
Table 3-2 Status of EMUs currently on ISS ............................................................................................ 118
Table 3-3 EMU failures on ISS............................................................................................................... 121

vii
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Acknowledgments
The Mishap Investigation Board acknowledges the professionalism, courtesy, openness, and honesty of
all parties who contributed to this effort. The Board received much valuable input from the Mission
Operations Directorate Personnel, which was very informative and necessary for it to complete the
investigation. The Board also immensely benefited from Ms. Dana Weigels leadership while managing
the initial Contingency EVA Capability Team and the EVA Recovery Team. Her teams work was very
timely and impactful, given that the failed equipment was not available for the Board during
investigation.

The Board expresses its sincere appreciation to the team of experts at the United Technologies Aerospace
Systems (at Windsor Locks, CT) for allowing the Board to tour, meet, and gather data and materials
relevant to the maintenance and refurbishing of the suits Portable Life Support System (PLSS)
components.

The Board appreciates the assistance provided by JSC Aircraft Operations Division personnel for hand-
carrying faulty EMU parts from the Soyuz landing site in Kazhakstan to Bangor, ME which expedited the
hardware investigation process. The Board also thanks Ms. Robin Hetherington, who patiently and
graciously answered many EMU questions from the MIB members throughout the investigation.
Additionally, Ms. Allison Bollinger and Mr. Zeb Scoville contributed to the Board by providing
invaluable hardware information. The Board thanks them for their professional courtesy and commitment
to the MIB. The Board also benefitted from Mr. Mike Mullane, former astronaut, for volunteering his
time to meet and talk with the Board about Normalization of Deviance and other pertinent topics, which
were helpful in our deliberations. The Board also would like to recognize LtCol Mark Glissman, from the
United States Air Force Safety Center, who spent a week with the Board providing valuable insights,
observations, and recommendations.

The Board appreciates the NASA Safety Center for providing excellent tactical support throughout the
investigation and for providing editorial support in preparation of this report. The Board also thanks the
Johnson Space Center for providing a comfortable and secure facility, with excellent Information
Technology and Logistics Support.

The Board would not have accomplished its task without the following personnel, who went above and
beyond what they were expected to provide.

Mr. Charles Armstrong, JSC Safety and EVA Ops Advisor, made significant contributions to the team by
being able to recall and direct the Board to early processes and knowledge regarding the EMUs Life
Support System. Mr. Armstrong also contributed significant amounts of time and energy to the
production of the final report.

Mr. Joe McMann, while serving as an EVA consultant, provided considerable detail regarding the early
design of the EMU Life Support System and the subsequent procurement process. Mr. McMann also
provided significant technical support to the ISS Investigation Team and was an invaluable resource to
them as well as the MIB.

1
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Dr. Lars Ulissey, while serving as Medical Advisor, not only kept the Board apprised of medical issues
related to the mishap, but also brought to bear his many years of professional experience in conducting
human factors mishap investigations for the United States Air Force. Because of his valuable involvement
with the Board, this report is infused with DOD Human Factor Codes, in conjunction with NASAs Root
Cause Analysis.

The Board also would like to acknowledge Ms. Susan Schuh, Ms. Katie Vasser, and Ms. Marla Gonzalez
for the most arduous task of transcribing the communications loop traffic and all of our interviews in
support of this investigation. In addition to expeditiously transcribing all these audio logs and interviews,
these Human Factors Specialists created an extensive database with filters and categories for the Board to
assimilate information quickly and efficiently. The Board appreciates their professionalism and
willingness to work with the Board in creating and establishing human factors issues with relevant
evidence, findings, and recommendations.

Mr. Ryan Schulte did an exemplary job as the Boards Executive Secretary and the in-house computer IT
expert and also made significant technical contributions to the MIB investigation and root cause analysis.

2
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

1.0 Charter and Response

1.1 Signature Pages

1.1.1 Investigating Authority Signatures

To the best of our knowledge the report contents are accurate and complete, and we concur with the
documented findings and recommendations.

3
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

1.1.2 Ex Officio Signature

I assure the following:

The investigation was conducted in conformance with NASA policy and NASA Procedural Requirements
8621.1B.

a. The investigation process is fair, independent, and non-punitive.

b. The mishap report contains all the required elements.
c. Adequate facts have been gathered and analyzed to substantiate the findings.
d. The mishap report accurately identifies the proximate cause(s), root cause(s), and contributing
factor(s).
e. The recommendations reasonably address the causes and findings.
f. Each recommendation can be tied to a finding.
I also concur with this report.

4
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

1.1.3 Advisor Signatures

I sign this report indicating that any privileged or proprietary, ITAR, or EAR information, or material
subject to the Privacy Act has been identified and marked as non-releasable to the public (e.g., NASA
Sensitive But Unclassified). In addition, this report is consistent with the policies and procedures in my
functional area.

I sign this report indicating that the report is technically correct in my functional area.

I sign this report indicating that the report is technically correct in my functional area.

5
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

I sign this report indicating that any privileged or proprietary, ITAR, or EAR information, or material
subject to the Privacy Act has been identified and marked as non-releasable to the public (e.g., NASA
Sensitive But Unclassified); and that volumes/appendices that are releasable to the public are marked
releasable. In addition, this report is consistent with the policies and procedures in my functional area.

6
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

1.2 Transmittal Letter

National Aeronautics and

Space Administration
Lyndon B. Johnson Space Center
2101 NASA Parkway
Houston, Texas 77058-3696

December 20, 2013

TO: Associate Administrator for Human Exploration and Operations

FROM: Chair, Mishap Investigation Board for International Space Station (ISS) EVA Suit
Water Intrusion, High Visibility Close Call

SUBJECT: Final Report for International Space Station EVA Suit Water Intrusion, High Visibility
Close Call.

Reference your letter dated July 22, 2013, which established the Mishap Investigation Board for
International Space Station EVA Suite Water Intrusion High Visibility Close Call that occurred on
July 16, 2013, and defined the Board's responsibilities.

The investigation was conducted in accordance with NPR 8621.1 "NASA Procedural Requirements
for Mishap and Close Call Reporting, Investigation, and Recordkeeping." Enclosed is the final report
for the Mishap Investigation Board' s activities, finding, and recommendations.

Christopher P. Hansen
Chairman, Mishap Investigation Board
ISSP Chief Engineer
Johnson Space Center

3 Enclosures

1 Written Report
2 Electronic Files of Written Report

7
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

1.3 Appointment Letter

National Aeronautics and Space Administration

Headquarters
Washington, DC 20546-0001
July 22, 2013

Roply to Attn or:

Human Exploration and Operations Mission Directorate

TO: Distribution

FROM: Associate Administrator for Human Exploration and Operations

SUBJECT: Appointment of Mishap Investigation Board for International Space Station

(ISS) EVA Suit Water Intrusion, High Visibility Close Call on
July 16,2013

This memorandum is in reference to NPR 8621.1 "NASA Procedural Requirements for

Mishap Reporting, Investigating, and Recordkeeping" and establishes the ISS EVA Suit
Water Intrusion, High Visibility Close Call Mishap Investigation Board (MTB) and sets
forth its responsibilities and membership. The chairperson and members of the Board are
listed in the enclosure.

ln accordance with the NPR, I am establishing the ISS EVA Suit Water Intrusion, High
Visibility Close Call MIB to gather information; analyze the facts; identify the proximate
causes, root causes, and contributing factors relating to the mishap; and to recommend
appropriate actions to prevent a similar mishap from occurring again.

The Board chairperson will report to me on all aspects regarding this investigation.
The MIB will complete the following actions:

Obtain and analyze whatever evidence, facts, and opinions it considers relevant
including past operational and maintenance performance.
Establish a positive working relationship with the NASA investigative team to
ensure that tasks are not duplicated and that the NASA investigation proceeds
promptly. This team should not interfere with the activities already underway by
the ISS team.
Conduct tests and any other activity it deems appropriate.
Interview witness and receive statements from witnesses.
Impound property, equipment, and records as considered necessary (consistent with
the agreements with the international partners and contractors).
Determine the proximate causes, intermediate causes, root causes, and contributing
factors relating to the mishap.
Develop recommendations that address the problem, and are clear, verifiable, and
achievable in order to prevent similar mishaps.
Develop lessons learned for potential application to all NASA human space flight
programs and projects.

8
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Provide a final written report to me that will conform to all requirements in the
referenced NPR.
The Board chairperson will complete the following actions:

Conduct Board activities in accordance with the requirements in NPR 8621.1.

Establish and document, as necessary, rules and procedures for organizing and
operating the Board, including any subgroups, and for the format and content of
oral or written reports to and by the Board.
Designate any additional representatives, advisors, consultants, experts, liaison
officers, or other individuals who may be required to support the activities of the
Board and define the duties and responsibilities of those persons. Note that the
NASA Safety Center (NSC) Mishap Investigation Supp01t Office (MISO) Regional
Field Specialists are available to boards as advisors for initial MIB orientation, and
should be used as a resource to ensure timely conduct ofthe investigation, and that
all deliverables are completed in accordance with Agency expectations. This
includes, but is not limited to the following: NASA Root Cause Analysis process
and software guidance; technical writing assistance; and critique of the draft and
final report submissions. Additional technical writing or editing resources are also
available throughout the investigation and report process. Please ensure that the
NSC representative is included in the deliberations of this Board and call on him or
her to assist or advise the Board if there is process or other technical concerns.
Designate another voting member of the Board to act as chairperson in his or her
absence.
Document meetings and retain records.

The Board will begin its investigation during the week of29 July 2013 and will provide a
report with their findings and recommendations within 75 workdays (or fewer, if directed).
There may be the potential for futme analysis after the Extravehicular Mobility Unit
(EMU) is returned to Earth in early 2014. I also recognize that this activity will be different
as the effected crew and hardware will not be directly available. The lack of directly
available physical evidence will require this board to work closely with the ISS program
and its ongoing activities. This information will be used to substantiate any board
findings, but should not delay any Board proceedings. The Board members are released
from all other duties until completion of the investigation and out brief to endorsing
officials.

I will dismiss the Board when it has fulfilled its requirements.

William H. Gerstenrnaier

Enclosure

9
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

10
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

11
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

1.4 Executive Summary

On July 16, 2013, two US crew members (referred to here as EV1 and EV2) exited the International
Space Station (ISS) US Airlock to begin U.S. Extravehicular Activity (EVA) 23. Roughly 44 minutes
(Phase Elapsed Time) into EVA 23, EV2 reported water inside his helmet on the back of his head. The
EVA ground team and EV2 were unable to identify the waters source. As EV2 continued to work, the
amount of water in the helmet increased and eventually migrated from the back of his head onto his face.
EVA 23 was terminated early and the crew safely ingressed the airlock. The nominal rate was used to re-
pressurize the airlock followed by an expedited suit doffing. The water quantity introduced into the
helmet was estimated at 1 to 1.5 liters.

During the post-EVA debrief, EV2 reported impaired visibility and breathing with water covering his
eyes, nose, and ears. In addition, EV2 had audio communication issues because of the water. When
returning to the airlock, EV2 had to rely on manual feel of his safety tethers cable for pathway directions.
The event was classified as a High Visibility Close Call and entered into the NASA Incident Reporting
Information System (IRIS) as record number S-2013-199-00005.

A related concern occurred during a post-EVA 23 suit dry-out procedure. A vacuum cleaner was used and
unexpectedly suctioned O2 from the suits secondary high pressure oxygen tank, causing a potentially
hazardous mix of electricity and pure O2, which could have ignited flammable materials in and around the
vacuum cleaner. Fortunately, no incident of this nature was detected.

Figure 1-1. Extravehicular Mobility Unit (EMU) With Water in Helmet During Post-EVA
23 Screening Test

The International Space Station (ISS) Extravehicular Activity (EVA) 23 Suit Water Intrusion High
Visibility Close Call (HVCC) Mishap Investigation Board (MIB) was appointed on July 22, 2013 to
gather information, analyze facts, identify the proximate cause(s), intermediate cause(s), and root causes
that resulted in the mishap. In addition, the MIB was asked to comment on observations and contributing
factors related to the mishap, and make recommendations that could be implemented within NASA to
prevent a similar mishap from occurring in the future. It was not the intent of the MIB to place blame or

12
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

to determine legal liability for the mishap but only to act as an Independent Investigation Authority in
compliance with the guidelines of NASA Procedural Requirements (NPR) 8621.1B.

The MIB identified the underlying causes of the event based on reviews of audio and video recordings of
the EVA and all associated ground and Space to Ground communications (comm) loops; interviews with
many of the individuals (including EV1 and EV2) involved in the EVA planning and execution;
interviews with contractors involved in the manufacture and refurbishment of the EMU; the use of the
Root Cause Analysis Tool (RCAT); interactions and participation with the ISS Investigation Team
investigating the hardware failure; data mining of numerous NASA sources; performance of a Human
Factors analysis; and participation in the performance of onboard testing. An Event and Causal Factor
Tree (ECFT) diagram (Appendix I and Section 0) was developed which identifies the root, proximate, and
intermediate causes.

In summary, the causes for this mishap evolved from (1) inorganic materials causing blockage of the
drum holes in the EMU water separator resulting in water spilling into the vent loop; (2) the NASA
teams lack of knowledge regarding this particular failure mode; and (3) misdiagnosis of this suit failure
when it initially occurred on EVA 22.

The source of the inorganic materials blocking the water separator drum holes had not been experienced
during an EVA before and is still undergoing a concurrent investigation. The results of this investigation
will ultimately lead to resolution of this issue; however, since the concurrent investigation into the source
of the debris is expected to continue for many months, the MIB does not yet have the required data to
determine the root causes of the contamination source, which must ultimately be determined to prevent
future mishaps. Because the hardware investigation must continue, this report is divided into two unique
areas of focus. First, the report focuses on the hardware failure investigation and understanding of the
hardware involved, work completed to date, preliminary results, and future work needed to determine root
causes. Second, the report focuses on real-time operation activities that can be improved to help the
ground control teams and crew quickly recognize and react rapidly to emergencies of this type.

Since this failure had not been seen previously or anticipated, the NASA team (Engineering, Operations,
Safety, and Crew) did not know or understand that an event such as this could occur. Without this
awareness, the teams response to the failure took comparatively longer than it typically would have. The
team applied what they did know to the symptoms they saw during EVA 23. Several possible causes were
discussed in real-time between the ground team and the crew members. Ultimately, the team came to the
correct conclusion that the water in EV2s helmet was more serious than anything that could be explained
by previous experience and the EVA was terminated.

In addition, the lack of understanding of this failure mode, along with several other reasons discussed in
this report, caused the team to misdiagnose this failure when it initially occurred at the end of EVA 22.
Had the issue been discussed in more detail at the end of that EVA, the team likely would have realized
that the water experienced in EV2s helmet was out of family and needed to be investigated further
before pressing ahead to EVA 23. That investigation most likely would have discovered this failure mode
and EVA 23 would have been postponed while the issue was resolved, thus preventing this mishap.

The MIB strongly believes that EVA crewmember 2 (EV2) and the flight control team performed well
given what they knew at the time of this mishap. EV2s calm demeanor in the face of his helmet filling

13
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

with water possibly saved his life. The flight control team quickly discussed and sorted through multiple
possible explanations for the water in the helmet. The ISS Program has assembled an investigation team
which has responded to this failure with a level of concern and has applied resources that demonstrate its
awareness of both the seriousness of this event and the importance of fully understanding and correcting
the deficiencies that allowed it to happen. Many of the recommendations in this report have already been
implemented or are under discussion as a result of the involved organizations proactive response.

All voting members of the board participated in the investigation, deliberations, and development of the
findings and recommendations. Upon completion of the deliberations, all voting members were polled
and were in agreement with the findings and recommendations as written. There were no dissenting
opinions, and therefore a minority report section is not included in the report.

Summary of Findings

The appointment letter instructed the MIB to place the highest priority on determining corrective actions
necessary to prevent similar mishaps from occurring. Using the process described above, the MIB
conducted a Root Cause Analysis (RCA). Timelines and an Event and Causal Factor Tree (ECFT) were
developed, leading to the identification of one primary undesired outcome (PUO) that revealed three
proximate causes, 19 intermediate causes, 30 observations, 13 contributing factors, and 49
recommendations. Five root causes were identified for the mishap at the organizational level under the
PUO.

Primary Undesired Outcome: ECFT UO 1 - EVA crew member (EV2) exposed to potential loss of
life during EVA 23

The primary undesired outcome of this mishap was that the EV crewmember experienced a large amount
of water collecting inside his helmet which created several hazardous conditions including risk of
asphyxiation, impaired vision, and a compromised ability to communicate.

Secondary Undesired Outcome: The Crew and ISS were exposed to a potential fire hazard due to
inadvertent activation of the EMU 3011 Secondary Oxygen Pack during EMU dryout activities.

During the course of this investigation, the MIB identified an additional undesired outcome addressed
here as Secondary Undesired Outcome (SUO). Section 2.5 discusses the events involving the SUO. No
additional causes, findings, or observations were generated solely as a result of this SUO. Rather, the
causes, findings, and recommendations that centered on the PUO address the issues identified that caused
the SUO.

14
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Root Causes

A Root Cause is one of multiple factors (events or conditions, that are organizational factors) that
contributed to or created the proximate cause and subsequent undesired outcome and, if eliminated or
modified, would have prevented the undesired outcome. Typically, multiple root causes contribute to an
undesired outcome. Additional root causes to the hardware failure will be identified as the investigation of
the inorganic materials found in the Fan/Pump/Separator continues.

The MIB identified the following five Root Causes:

RC1 ECFT-1.1.1.2.1: Program emphasis was to maximize crew time on orbit for utilization.
The ISS Program must place a strong emphasis on performing utilization with the ISS; it is in fact
the very reason ISS exists. However, the strong emphasis on utilization was leading team members to feel
that requesting on-orbit time for anything non-science related was likely to be denied and therefore tended
to assume their next course of action could not include on-orbit time. The danger with that thought
process is that lower level team members were in effect making risk decisions for the Program, without
necessarily having a Program wide viewpoint or understanding of the risk trades actually being made at a
Program level.
RC2 ECFT-1.1.1.2.1: ISS Community perception was that drink bags leak.
The MIB could not identify a clear reason why the EVA community at large had the perception
that the EVA drink bags leaked. When presented with the suggestion that the crew members drink bag
leaked out the large amount of water that was found in EV2s helmet after EVA 22, no one in the EVA
community (which includes team members from Operations, Engineering, Safety, and Crew) challenged
this determination and investigated further. Had that conclusion been challenged, the issue would likely
have been discovered prior to EVA 23 and the mishap would have been avoided.
RC3 ECFT-1.1.1.3: Flight Control Teams perception of the anomaly report process as being
resource intensive made them reluctant to invoke it.
Based on interviews and MIB investigation, it was clear that several ground team members were
concerned that if the assumed drink bag anomaly experienced at the end of EVA 22 were to be
investigated further, it would likely lead to a long, intensive process that would interfere with necessary
work needed to prepare for the upcoming EVA 23, and that this issue would likely not uncover anything
significant enough to justify the resources which would have to be spent.
RC4 ECFT-3.1.2.1.1.1.1: No one applied knowledge of the physics of water behavior in zero-g to
water coming from the PLSS vent loop.
The MIB learned that while there is a significant amount of knowledge about the way water
behaves in zero-gravity, the ground teams did not properly understand how the physics of water behavior
inside the complex environment of the EMU helmet would manifest itself. The teams believed that if
significant water entered the helmet through the vent loop that it would cling to the inner surface of the
helmet rather than cling to the crew members head. They also believed that if a significant amount of
water entered the vent loop, the Fan/Pump/Separator would likely stall, as it had in 1-G when significant
water entered the vent loop. Therefore, the significant hazard it presented was not anticipated.
RC5 ECFT-3.1.2.1.2: The occurrence of minor amounts of water in the helmet was normalized.
Through interviews with ground personnel and review of data from previous EMU performance,
it was clear that some water entering the helmet was considered normal by the ground teams. Despite the
fact that water carryover into the helmet presented a known hazard of creating eye irritation due to its
interaction with anti-fog agents, and also presented a potential fogging hazard, the ground teams grew to

15
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

accept this as normal EMU behavior. Since these smaller amounts of water carryover had never caused a
significant close call, it was perceived to not be a hazardous condition. When water began entering EV2s
helmet, the ground team discussed anti-fog/eye irritation concerns and visibility concerns; however, a
more hazardous condition was not expected because the presence of water in the helmet had been
normalized.
Proximate Causes
A Proximate Cause is the event(s) that occurred, including any condition(s) that existed immediately
before the undesired outcome, directly resulted in the occurrence of the undesired outcome and, if
eliminated or modified, would have prevented the undesired outcome.
Three proximate causes were identified. Had any of these causes been eliminated or modified, neither the
Primary nor Secondary Undesired Outcomes would have occurred.
P1 ECFT-1: The ISS Program conducted EVA 23 without recognizing the EMU failure which
occurred on EVA 22
The MIB learned that on EVA 22, EV2 in suit 3011 experienced water in the helmet during
repress. This failure was misdiagnosed and not determined to be a constraint to EVA 23. The MIB has
determined that had the source of the water at the end of EVA 22 been investigated thoroughly, EVA 23
and the subsequent mishap would not have occurred.
P2 ECFT-2: EMU 3011 Helmet had a large quantity of water during EVA 23.
During EVA 23, EV2 (wearing SEMU 3011) experienced a large amount of water inside the
helmet area, originating somewhere behind the crewmembers head near the neck/lower head area. The
presence of this water created a condition that was life threatening.

P3 ECFT-3: Flight Control Team/Crew did not terminate EVA 23 as soon as water was reported in
the helmet.

The MIB determined that the time between first mention of water in EV2s helmet and the call to
terminate the EVA was roughly 23 minutes. The fact that no one on the ground or the EVA Crew
immediately recognized the severity of the hazard and terminated the EVA resulted in the crewmember
being exposed to an increased level of risk.

16
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Recommendations

The MIB developed the following recommendations. All recommendations are based on the
MIBs findings and observations seen as important to preventing future mishaps or close calls.
Discussions of the rationale behind these recommendations are contained in the body of the report.

Proximate Cause
(PC),
Intermediate
Cause(I),
Recommendation Contributing
Factor (CF), Root
Cause(RC),
Observation (O)

Recommendation The ISS Program must reiterate to all team members that, if they RC-1
R-1: feel that crew time is needed to support their system, a request
and associated rationale must be elevated to the ISS Program for
an appropriate decision.
Recommendation ISS Program should ensure that the FMEA/CILs are updated I-7
R-2: and maintained and MOD should make them required
reading/study for all EVA Systems instructors and Flight
Controllers up to and including FCR operators as well as their
proficiency flows. EVA safety and Engineering MER support
personnel should also include this in their training flows
Recommendation MOD SSTF instructors should ensure that training includes use I-7
R-3: of the FMEA/CIL to develop failure scenarios for use in
integrated and stand-alone simulations.
Recommendation The ISS Program should ensure that updates are made to the I-16
R-4: EMU hazard reports to reflect the possibility of water in the
helmet resulting in a catastrophic event due to asphyxiation.
Recommendation The ISS Program should ensure that the FMEA/CIL is updated I-19
R-5: and reviewed thoroughly from end-to-end every two years to
ensure currency with participation by Engineering, MOD,
Safety, Medical, and appropriate contractor personnel.
Recommendation The ISS Program should ensure that all instances of free water RC-5
R-6: and contamination in the EMU are documented and
investigated, with corrective action taken, if appropriate.
Recommendation MOD must lead the development of appropriate flight rules and I-15
R-7: procedures to address the course of action to take in the event of
water in the helmet.
Recommendation The ISS Program should investigate alternate materials that O-2
R-8: effectively perform the helmet anti-fogging function without the
risk of eye irritation.
Recommendation The ISS Program should investigate alternate CO2 sensor O-3
R-9: designs that eliminate the sensitivity to moisture.

17
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Recommendation MOD should evaluate how personnel who are located in the O-4
R-10: POCC facility and not part of the active flight control team
interact with the active flight control team and ensure that lines
of communication and the decision making chain is not
compromised.
Recommendation The ISS Program should perform testing and analysis to verify O-5
R-11: that use of the Helmet Purge Valve to remove free water from
the helmet is safe and effective. Results of this testing should be
made clear to the EVA community, including the flight control
team and documented in hazard reports, flight rules and
procedures.
Recommendation ISS Program and Safety and Mission Assurance should review O-6
R-12: and update the process as defined in JSC 28035 to resolve the
conflict of interest of the EVA Office in initiating FIARs.
Recommendation Safety and Mission Assurance with the assistance of the EVA O-6
R-13: Office should initiate a review of all non-conformances
contained in the PRACA database for the EMU and review the
assignment of FMEA associated with each one and update as
required.
Recommendation The ISS Program should commission an independent technical O-7
R-14: review of the EMU 6-year certification plan which should
identify all deficiencies or weaknesses in the certification and re-
establish the true life expectancy of the EMU, and then plan
appropriate use and logistic strategies commensurate with the
results of the review.
Recommendation The EVA Office should ensure that all EMU procedures are O-8
R-15: consistent between all teams that perform operations with the
EMU, and require that all contamination found during ground
processing be evaluated by the Engineering and Quality teams.
Recommendation MOD should ensure that simulations of specific, fast-paced O-9
R-16: failure scenarios (visiting vehicles, on-board emergency
response, software transition issues, and serious system
hardware failures) should include all phases of the teams
response to ensure that the response can be fully performed from
end-to-end in a quick, proficient manner.
Recommendation The ISS Program should ensure that FMEA/CILs related to fast- O-9
R-17: paced failure scenarios (visiting vehicles, on-board emergency
response, software transition issues, and serious system
hardware failures) are regularly updated, studied, and used in
training for flight controllers as well as engineering and safety
personnel.

18
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Recommendation As the success of the ISS Program continues, the ISS Program O-9
R-18: must institute requirements and behaviors that combat the
tendency towards complacency by requiring regular training by
all teams in the safety critical aspects of failures related to fast-
paced scenarios (visiting vehicles, on-board emergency
response, software transition issues, and serious system
hardware failures).
Recommendation The ISS Program must ensure that full root cause determination O-9
R-19: of failures related to specific, fast-paced failure scenarios
(visiting vehicles, on-board emergency response, software
transition issues, and serious system hardware failures) must be
pursued and verified by ISS Program managers and the
Engineering and Safety Technical Authorities.
Recommendation The ISS Program should institute a systematic process of O-10
R-20: monitoring water quality and chemistry aboard ISS to track
changes that can affect critical ISS systems including the EMU,
crew health, and multiple ISS Systems that use water and are
sensitive to its chemical makeup (The Oxygen Generation
System, The Water Processor Assembly, the Common Cabin Air
Assembly, etc.). This process should include consideration of
onboard monitoring capability. It should also include return of
any removed hardware to the ground for evaluation.
Recommendation The ISS Program should develop a system that allows high rate O-11
R-21: data telemetry to be received by ground teams during an EVA to
allow near instantaneous monitoring of critical system
parameters.
Recommendation The ISS Program should develop a flexible system that allows O-12
R-22: multiple short EMUs, as well as EMU components such as the
PLSS, to be launched or returned on multiple vehicles.
Recommendation The ISS Program and the EVA Project Office should put O-13
R-23: schedules and processes in place to ensure access to flight
hardware to the broader EVA community including the
Astronaut Office, MOD EVA, and S&MA personnel.
Recommendation The ISS Program and the EVA Project Office should require O-13
R-24: close out photos be taken of all hardware with the participation
of operations personnel to document the precise configuration of
what is flying as well as accurate configuration records
maintained and made available to real-time support personnel to
facilitate effective communication between the ground and crew
in flight.
Recommendation The ISS Program and the EVA Project Office should ensure that O-13
R-25: all procedures are validated on flight hardware if the procedure
requires a functioning system versus a fit check.
Recommendation For critical external tasks, the ISS Program should provide at 0
R-26: least one viable and proven dissimilar backup EVA capability
(known candidates include dexterous robotics or Russian EVA)

19
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Recommendation With the help of MOD, the EVA Office should review all O-15
R-27: existing EVA knowledge databases and combine them into a set
of databases that are complete, accurate, kept up-to-date, and
easily accessible to the entire EVA community
Recommendation The ISS Program should ensure that the EMU Requirements O-15
R-28: Evolution Book is routinely updated to capture the maturing
design and design rationale of the EMU and include material
originally intended for the placeholder sections in the 1994
version.
Recommendation The ISS Program should ensure that the EVA community uses O-15
R-29: the EMU Requirements Evolution Book and the improved
knowledge capture databases, once developed, to improve
ground team training requirements throughout the EVA
community for better depth of EMU system knowledge and
attention to design and failure history.
Recommendation The Agency, Centers, and Programs should improve O-16
R-30: requirements for root cause determination and subsequent
training and provide the training for Engineering and Safety
personnel to better ensure root cause determination of critical
and reoccurring failures.
Recommendation MOD should provide integrated EVA sims with the possibility O-18
R-31: of ending the sim early. These sims must be scheduled for the
full duration, but allowed to end early if required by the actions
taken by the flight control team. Additionally, airlock ingress
and repress should be routinely included as part of simulations
that involve terminating an EVA with an EMU in an off nominal
configuration.
Recommendation MOD should review all procedures with a MCC step and O-19
R-32: verify that rationale exists to explain the required actions to be
taken by the flight control team if this step is reached.
Recommendation The ISS Program should ensure appropriate connectivity O-20
R-33: between all relevant parties who participate in EVA activities to
support real-time operations including talk/listen access to MCC
Audio Loops.
Recommendation MOD should strengthen training to emphasize the physiological O-22
R-34: effects of a rapid repress on the crew to aid in the decision
making process in real-time.
Recommendation The ISS Program and JSC EVA Office should improve technical O-23
R-35: and management coordination between their two organizations
and ensure that all strategic and tactical decisions that are made
by either organization are quickly and effectively understood,
and officially accepted by both.

20
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Recommendation The government officials and contract managers must put in O-24
R-36: place expectations and create a board environment that allows
the EVA contractors to freely challenge technical decisions
made by the governing boards when appropriate and encourage
proactive participation.
Recommendation To reinforce the independence of safety and recognize the O-25
R-37: unique criticality of EVA in the safety community, consider
altering the ISS CSOs office to more closely mirror that of the
ISS Chief Engineers Office by creating a deputy CSO for EVA
position to more closely work with the EVA safety community
and help integrate them into the ISS Program and aid the CSOs
and Program Managers understanding of EVA risks in the
context of the ISS Program.
Recommendation JSC Safety and Mission Assurance should provide additional O-25
R-38: EVA training and integration activities to the MER Safety
Officer training syllabus.
Recommendation JSC Safety and Mission Assurance should institute a training O-26
R-39: program for all of its EVA personnel that includes a subset of
MOD EVA task and EVA systems training flows to gain the
requisite training on EVA hardware and tasks it is being used
on. This training should be supplemented by observing EMU
vacuum chamber runs, NBL runs, hardware reviews, and ground
testing both at SGT and UTAS Windsor Locks and studying the
EMU Requirements Evolution document should be mandatory.
Recommendation JSC Safety and Mission Assurance should routinely advocate for O-26
R-40: and lead the periodic review of FMEA/CILs and Hazard reports
and be intimately familiar with their content.
Recommendation ISS Program should augment, at least temporarily, MOD EVA O-27
R-41: personnel to allow the existing backlog of work to be completed
in a fairly short order by bringing on, through rotational
opportunities, personnel that can provide valuable technical
assistance that will not add to the training and certification
burden already faced by the organization.
Recommendation ISS Program should provide additional long term resources to O-27
R-42: augment current EVA community staffing to support the coming
increased frequency of ISS maintenance and contingency EVAs.
Recommendation The ISS Program must define The Roles and Responsibilities of O-28
R-43: the MER and the FCT to a level whereby each position (FCT
and MER) on either side clearly understands their role and the
role of their counterparts and mutual expectations must be
established and agreed to. As part of this effort, the Program
needs to reinforce the understanding that it is the FCT that is
authorized to accept risk on behalf of the Program in real-time
operations requiring best engineering judgment.

21
National Aeronautics and Space Administration EVA 23 Suit Water Intrusion

Recommendation The ISS Program must establish a protocol whereby whenever O-28
R-44: conflicts arise between the MER and FCT concerning roles and
responsibilities or one partys performance during a particular
event, the appropriate management from each side must meet to
discuss the conflict and revise the roles and responsibilities or
expectations accordingly.
Recommendation The ISS Program should develop proficiency requirements for O-28
R-45: MER Managers by event they are certified to support, as well as
on a time basis (e.g. annually) to maintain currency.
Recommendation The ISS Program should provide training to the MER Managers O-28
R-46: to deepen their systems and vehicle knowledge to ensure proper
subsystem and situational awareness during real-time
operations.
Recommendation The ISS Program should immediately modify the contractual O-28
R-47: clauses that may prevent the recommendations contained in this
report from being implemented within the contractor
community.
Recommendation NASA real-time operations community should work with the O-29
R-48: JSC Human Factors team to assess areas where human factors
processes can be better trained and implemented in operations
and develop specific training to reduce the impact of human
factors in future mishaps.
Recommendation The ISS Program should commission an independent study team O-30
R-49: to identify options to ensure an ISS EVA capability through
2028 that trades improvements to the current single fault tolerant
suit via options such as additional on-orbit diagnostics and
preventative/corrective maintainability, redesign to separate
water and vent loops, and/or implementation of an advanced
suit.

22
 National Aeronautics and Space Administration EVA23Water Intrusion

2.0 Narrative Description and Facts

2.1 Overview
The International Space Station (ISS) is located in low Earth orbit about 400 km (250 mi) above the
Earths surface. It serves as a microgravity and space environment research laboratory in which crew
members conduct experiments in biology, physics, astronomy, materials science and many other fields.
The station is suited for the testing of spacecraft systems and equipment required for future missions to
the Moon and Mars.
The ISS Program is tied together by a complex set of legal, political, and financial agreements among 15
nations involved in the project, governing ownership of the various components, rights to crewing and
utilization, and responsibilities for crew rotation and station resupply. These intergovernmental treaties
and agreements are the basis of the joint project among five participating space agenciesNASA, the
Russian Federal Space Agency (Roskosmos), Japan Aerospace Exploration Agency (JAXA), European
Space Agency (ESA), and Canadian Space Agency (CSA)and their respective ISS Programs.
Agreements govern daily interaction among the agencies teams to maintain station operations, from
traffic control of spacecraft to and from the Station, to utilization of space and crew time. ISS
Extravehicular Activities (EVAs), or spacewalks, are major activities performed outside an Earth-orbiting
craft. Astronauts or cosmonauts conduct spacewalks to build and maintain the orbital laboratory. They
install new components; re-wire systems, modules, and equipment; monitor, install, and retrieve scientific
experiments; and also provide a contingency capability to assure ISS viability and crew safety.

2.2 Background
The current ISS Extravehicular Mobility Unit (EMU), a complex spacesuit that provides protection from
extreme conditions of space, is a mobile life support system with an oxygen supply, electrical power,
water-cooling equipment, ventilating fan, and an in-suit drink bag. The EMU was originally developed
for use on the U.S. Space Shuttle to mitigate failure scenarios in which the Shuttle payload bay doors
failed to close and lock properly prior to atmospheric re-entry. An additional postulated failure scenario
involved achieving rescue of a disabled orbiter by EVA crewmembers entering a depressurized vehicle
and accessing the flight deck. This particular risk mitigation approach required that the EVA suit and the
Portable Life Support System (PLSS) assembly be sizedwidth and depthto pass through the Shuttle
hatch openings to the flight deck. The EMU has since evolved from a suit to mitigate Shuttle failure
scenarios to one capable of deploying, capturing, and repairing satellites, and enabling astronauts to
assemble, repair, and maintain the ISS.

As mission objectives expanded, the once single-mission EMU certification was incrementally extended
to an operational life of multiple years on the ISS. The evolution of the suit over the years resulted in a
long history of issues that led to many modifications to EMU components as noted in Appendix H.

The Quest Joint Airlock module in the U.S. segment of the ISS maintains the habitable environment when
astronauts are exiting or entering the spacecraft for EVA operations. It consists of two main parts: the
equipment lock and the crew lock. The equipment lock is where the EMUs are stored and preparations for
spacewalks are carried out. The crew lock is depressurized during spacewalks.

23
 National Aeronautics and Space Administration EVA23Water Intrusion

Continuous flight of the ISS requires spacesuits to be left on-board for longer periods of time than the
suits original Shuttle certification allowed. At the beginning of the ISS Program, EMUs were launched
on a Shuttle and a complement of suits was left on ISS when the Shuttle un-docked. On subsequent
Shuttle missions, the suits were replaced and returned to the ground for maintenance and refurbishment.

To support continuous operation of the ISS, the period of EMU maintenance cycles was extended from
the 1-3 EVAs of a Shuttle mission to one year and 25 EVAs. Then in 2002, the maintenance interval was
extended again to 2 years. In 2007, the certification was further extended to 3 years. The current
operational certification is 6 years. NASAs decision to retire the Shuttle fleet in 2011 required another
change to the EMU operations concept. The complement of EMUs on ISS was increased from three to
four. For the EMU hardware to meet the longer 6-year maintenance interval on-orbit in the ISS, it is
required to go through additional ground processing. This processing includes cleaning or replacing water
filters along with the stripping and recoating of areas with known susceptibility to corrosion (i.e., water
tank walls, Sublimator flange, and so on).

2.2.1 Mission Control Center Team

The Mission Control Center (MCC), located at the NASA Johnson Space Center (JSC) manages the ISS
vehicle missions. The MCC team consists of a staff of flight controllers and other support personnel who
monitor all aspects of the mission using telemetry, send commands to the vehicle using ground stations,
and speak with the crew members on orbit. Personnel supporting the mission from MCC include
representatives of the attitude control system, power, propulsion, thermal, life support, flight dynamics,
operations planning, computer systems, EVA, Robotics, and other subsystem disciplines. The Flight
Director (FD) has responsibility for the entire team and is the decision-maker and risk manager for
operations including EVA. The flight controllers training, which includes extensive rehearsals with
simulated scenarios in the MCC, is also the responsibility of the FD.

The Missions Operations Directorate (MOD) console in MCC is manned by a senior manager who acts as
the liaison between the Flight Control Team and ISS Program management. In the MCC, the FD and all
flight controllers communicate with one another on dedicated internal voice loops, which are recorded
and archived for later review. To this end, Flight Control Room (FCR) operators wear headsets to listen to
the FD loop as well as the Space-to-Ground (S/G) loop to hear the crew on-orbit and be ready to respond
to questions, and by tracking the crews actions and vehicle status, respond proactively. Likewise FCR
operators monitor additional comm loops with a back room or Multi-Purpose Support Room (MPSR)
occupied by personnel who are also on headsets to monitor the FD loop, the appropriate FCR support
loop, and the S/G loop and provide additional systems and procedures monitoring assistance.

For questions that require a deeper understanding of the ISS systems, Engineering Directorate personnel
staff the Mission Evaluation Room (MER) and monitor the appropriate communications loops including
FD, S/G and select FCR and MPSR loops. Additional engineering personnel support EVA from the
Central Operations Room for EVA (CORE) located in JSC Building 7 as well as the EMU contractor
plant, United Technologies Aerospace Systems (UTAS) in Windsor Locks, CT. In each location one
person at the most is on headset, with others monitoring over a speaker. MER personnel communicate
with CORE through telephone contact and similarly, CORE communicates with UTAS over a speaker
phone located in an adjacent office to CORE. Figure 2-1 illustrates the MCC teams communication
paths.

24
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 2-1. EVA Communication Paths

Table 2-1 summarizes the teams that conduct and support EVAs, team members, and their roles and
responsibilities

Table 2-1. Mission Control Center Teams/Positions

MCC Teams Team members Roles and Responsibilities

ISS EV1 EVA lead crewmember
Crewmembers EV2 EVA support crewmember
IV Intravehicular crewmember supporting crewmembers outside
of vehicle
Representative of the Liaison between the Flight Control Team and ISS Program
Director of MOD Management
Flight Control (In this case it was the
Team Chief of the Flight
Director Office.)
Flight Director (FD) Flight Control Team leader responsible for safety of crew and
vehicle

25
 National Aeronautics and Space Administration EVA23Water Intrusion

Flight Control MOD EVA Leads for FCR EVA console support for EVAs 22 and 23
Team (cont.) EVAs 22 and 23
MOD EVA Airlock MPSR position to monitor Airlock systems
MOD EVA Task MPSR position to monitor EVA tasks; expert on all EVA tasks
providing insight to MOD EVA Lead on hardware fit and
functions; manages logistics of tools and other EVA hardware
as well as EVA timeline
MOD EVA Systems MPSR position to monitor EVA systems including the EMU.
MER EVA MER EVA and Crew MER console position primarily following EVAs from the
Systems engineering perspective; staffed by engineering personnel who
manage EVA hardware including tools and EMUs; consultant
to FCT if a deeper understanding of EVA hardware engineering
needed
MER Manager Manages the engineers in the MER covering all ISS systems
including EVA and specialty engineering areas
MER EVA Safety MER position providing safety monitoring and support for
EVA operations
Astronaut Office CAPCOM Primary voice to crew for everything but EVA operations
CB Ground IV Primary voice to crew for EVA operations; sits next to
CAPCOM in Mission Control
CORE Crew and Thermal Engineering personnel providing expertise in EVA hardware
Systems Division particularly the EMU; primarily supports the EVA MER
personnel
UTAS UTAS EMU personnel United Technologies Aerospace Systems (UTAS) is the prime
contractor for the EMU who provide plant level support to
EVA community during an EVA; primarily supports the CORE
Space and Life Flight Surgeons Monitors crew health during EVAs
Sciences Biomedical Engineers
Directorate (SA)

2.3 Events Immediately Leading Up to Mishap

Prior to EVA 23, EV crewmember 1 (EV1) had completed five EVAs, totaling 29 hours and 43 minutes.
EVA crewmember 2 (EV2) had completed one EVA (EVA 22), which was 6 hours and 7 minutes.

On May 12, 2013, ISS crewmembers conducted U.S. EVA 21. An EVA crewmember on this EVA wore
Short Extravehicular Mobility Unit (SEMU) 3011the SEMU that experienced the close call on EVA
23. The crewmember did not experience water in the suit during EVA 21.

On July 9, 2013, ISS crewmembers conducted US EVA 22. During EVA 22, EV1 and EV2 wore the
same EMUs that were later worn on EVA 23. When EV2s helmet was removed post-EVA 22, 1/2 to 1
liter of water was discovered in the helmet. EV1 reported that when EV1 was face-to-face with EV2 at
the airlock hatch prior to ingress, no sign of water was evident in EV2s helmet. Therefore, the crew
concluded that the water must have entered the helmet during repress. Also, during EVA 22 repress, EV2
was looking down and leaning forward and likely had pressed on the drink bag with his chest and could
have pinched the bite valve open with his chin, releasing water into his helmet. The ground team accepted

26
 National Aeronautics and Space Administration EVA23Water Intrusion

the crews drink bag leak assessment and the presence of excessive water in the helmet was not
investigated further. The crew cleaned up the residual water, and the ground team sent up procedure
deltas for EMU stowage to help the equipment dry out. The ground team instructed the crew to use a new
drink bag for the upcoming EVA 23, which they did. There was no discussion of water in the helmet
during EVA 23 pre-briefs on July 11, 2013 or July 15, 2013.

2.4 Events at Time of Mishap

On July 16, 2013, roughly PET (Phase Elapsed Time) 38 minutes into U.S. Extravehicular Activity EVA
23, EV2 (wearing SEMU 3011) had a CO2 Sensor Bad alarm in his suit. At PET 44 minutes, EV2
reported feeling water inside his helmet on the back of his neck and head. Less than 10 minutes later, EV2
reported an increase of water behind his head. The ground team and EV2 were unable to identify the
waters source. The pooling water was visually confirmed by EV1. As EV2 continued to work, the
amount of water increased. After 23 minutes of elapsed time, the Mission Control Team called for an
EVA termination and directed EV2 to translate back to and ingress the airlock. An EVA termination
allows for a crewmember with an EMU issue to return to the safety of the airlock while the other
crewmember safes the worksite followed by a nominal and orderly re-pressurization of the airlock.

During his translation to the Airlock, the water behind EV2s head began to migrate onto his face. Also
while translating, EV2 experienced intermittent communication difficulties with the ground. Following
EV1s ingress into the airlock, the nominal rate was used to re-pressurize the airlock followed by an
expedited suit doffing for EV2. The water quantity introduced into the helmet was estimated at 1 to 1.5
liters. Video downlink confirmed significant water covering the helmet interior when the helmet was
removed. EV2s Liquid Cooling and Ventilation Garment (LCVG) was relatively dry, however his
communications cap and helmet vent pad were completely soaked with water.

During a post-EVA debrief, EV2 reported having impaired visibility and breathing with water covering
his eyes, nose, and ears during his translation to the Airlock. In addition, EV2 mentioned having audio
communication issues due to water in his comm cap. His visibility was so poor due to the water that,
while returning to the airlock, EV2 had to rely on the manual feel of his safety tethers cable for pathway
directions. Because of the seriousness of this event, specifically the possibility that the water in the helmet
could have caused asphyxiation of EV2, a High Visibility Close Call was documented resulting in the
formation of this Mishap Investigation Board.

27
 National Aeronautics and Space Administration EVA23Water Intrusion

2.5 Events Following the Mishap including Secondary Undesired Outcome

Initial troubleshooting was performed the day after the EVA on July 17, 2013. Crew inspected EV2s
drink bag and determined no visible cuts, holes or other damage existed. The crew then filled the drink
bag and applied pressure. No leaks were noted. An inspection of the suit revealed no visible water (water
had all been cleaned up post-EVA). Water recharge and ullage dump were performed nominally and 3.84
lbm (1.75 Liters) of water went into the suit. This correlated with the crew report of 1 to 1.5 L of water in
the helmet post-EVA in addition to what would nominally be expended by the suit during a 1:40 PET
EVA. After water recharge, the crew executed water leak troubleshooting steps: The PLSS fan was turned
on with the SOP Check Out Fixture (SCOF) installed to cover the Vent port and O2 Actuator in IV setting
for approximately 14 minutes. No off-nominal conditions were noted and inspection of the suit revealed
no visible water. When the SCOF was removed per procedure, the crew reported hearing a sucking
sound and the fan stopped operating. The crew was directed to turn off the suit fan and move the O2
Actuator to OFF. The crew then turned the suit fan back ON and again set the O2 Actuator to IV. The fan
briefly began spinning and then shut down almost immediately, with the crew reporting a water sucking
or gurgling sound. Subsequent inspection revealed water in the METOX canister outlet and suit inlet
ports (a few drops on the METOX side and about a spoonful of water on the suit side). Crew additionally
reported seeing droplets of water in the neck vent port (T2). While the crew was awaiting further direction
from the ground, the Infrared (IR) CO2 transducer in the suit began to show an increase in its reading and
eventually went off-scale high, most likely due to moisture in the vent loop near the CO2 transducer.

After the fan in the EMU water separator pump was flooded, the ground team submitted a procedure for
drying out EMU 3011s Vent Loop using a wet/dry vacuum at the crews next available opportunity.
After the crew executed the EMU 3011 Vent Loop Wet/Dry Vacuum and Dryout activity, the EVA
officer in MCC noticed a nearly 500 psi reduction in SOP pressure had occurred during a Loss of Signal
(LOS), a time period where there is no telemetry available between the ground and ISS. Pressure in the
SOP dropped from an initial value of 5580 to 5081 psia. The crew reported the SOP pressure gauge read
~5200 psia which matched the telemetered data within the allowable +/- 490 psi range for the gauge. On
July 26, 2013, during EMU 3011 troubleshooting, EVA reported the SOP pressure reading had increased
to 5271 psia; the increase being due to the warming of the bottle contents after the expansion cooling
experienced during the inadvertent flow initiation. The use of a wet/dry vacuum during this
troubleshooting procedure was an off-nominal operation. The EMU team did not fully appreciate that the
SOP would engage and flow with the EMU O2 Actuator in the OFF position. The transmitted procedure
was not fully validated on the ground, and once implemented aboard ISS, caused a secondary undesired
outcome (SUO) involving exposing the crew and ISS to a potential fire hazard during inadvertent
activation of the EMU SOP.

A mechanical design review that was conducted and the vendor information that was obtained after this
incident indicate that a vacuum force is enough to partially open but not fully open the SOP regulator
valve. The vacuum produced by the vacuum cleaner in the vent loop provided enough delta-P to cause the
bellows of the SOP regulator to overcome the spring force designed to keep the regulator closed. When
this occurred, it created a potentially hazardous mix of electricity and pure O2, which could have ignited
flammable materials in and around the vacuum cleaner.

During discussions of the post-EVA 23 SOP issue, interviewees indicated that there was at least a
perceived pressure to perform the dry out procedure at the next available crew opportunity rather than

28
 National Aeronautics and Space Administration EVA23Water Intrusion

take the time to perform a proper Procedure Verification (PV) on the ground using high-fidelity flight
hardware. Only a fit check of the wet/dry vacuum on a non-functional Class-3 EMU was performed. This
perceived pressure was the result of programmatic emphasis to maximize on-orbit crew utilization time.

No additional causes, findings, or observations were generated solely due to this SUO. Rather, the causes,
findings, and recommendations that are centered on the close call also envelop the SOP issue.

29
 National Aeronautics and Space Administration EVA23Water Intrusion

3.0 Data and Analysis

3.1 Approach
The MIB investigation approach was challenged by the circumstances of the close call event. The EVA
took place onboard ISS which is an operational facility on orbit. The ISS has multiple failure scenarios
that can lead to it becoming zero-fault tolerant to the loss of the ISS vehicle. If any of these failures occur,
the ISS is at great risk for a total loss of vehicle event until the failed component is replaced. Most of
these failures require an EVA to remedy. Therefore, it is possible that an EVA could be required to save
the ISS at any time, even prior to the determination of the root cause of the EMU water close call. [Note:
at the time of submission of this report, prior to root cause determination, a series of EVAs required to
safe the ISS was underway]. This possibility required the ISS Program to quickly establish the ISS
Investigation Team which was tasked with determining the root cause of the failure as well as
determining what safety measures would need to be in place in the event an emergency EVA was
required. Because this team was in place at the time the MIB was convened, the MIB was directed to
work very closely with the ISS Investigation Team. To facilitate this, the MIB established a positive
working relationship with the ISS Investigation Team and attended team meetings as well as participated
in testing. In addition, the MIB obtained the services of a consultant with many years of experience as a
former subsystem manager for the EMU to consult both for the MIB as well as the ISS Team. His
participation allowed for easy knowledge transfer among all of the teams. The MIB was responsible for
ensuring that further EVA close calls were avoided, therefore, the MIB was involved in all decisions
related to preparations for a potential emergency EVA.

An additional challenge for the MIB was the fact that the failed hardware was on orbit so access to it was
difficult. Hence, much of the data gathered and used by the MIB to develop findings was gathered during
privileged interviews conducted with individuals from a broad spectrum of organizations that were
involved with EVA operations. This included individuals who were directly involved in the events of
EVA 23 as well as managers, engineering specialists, and contractors who support the Agencys efforts to
perform EVAs. There were also many less formal discussions that took place with both individuals and
groups that helped the MIB better understand organizations and their inter-relationships to look for areas
that worked well and areas that had weaknesses. The MIB also collected and reviewed many historical
documents to determine the information available to the EVA teams prior to and during the events being
investigated. The MIB also participated in many meetings held by the ISS Program related to
investigating the hardware failure and developing methods to protect future EVA crew members. In
addition, the MIB evaluated telemetry data from the EMUs themselves to better understand the nominal
performance of the suits as well as off nominal behavior due to water intrusion.

3.2 Type of Data Gathered: List of Documents

Data gathered during the Mishap Investigation includes video and audio feeds, transcripts of
communications between MCC and ISS (during EVA 23, EVA 22 debrief), audio voice loop
communications within MCC, witness statements, console logs, previous reports and investigations,
drawings, photographs, technical materials, and meeting minutes from a multitude of sources. For the
complete list of documents and data gathered, refer to Appendix B: List of Documents.

30
 National Aeronautics and Space Administration EVA23Water Intrusion

3.3 Timeline
The following timeline recaps events noted by the MIB that occurred around the time of the mishap.
Quotations shown are transcribed from Space-to-Ground audio loops. A more detailed timeline with a
sense of the suits long history is attached as Appendix E and the details of the EMU evolution are
documented in Appendix E:

Table 3-1. Short Timeline of EVA 23 Real-time Events

Date Time Description
7/9/2013 06:50:00 GMT A lot of water was reported in EV2's helmet post EVA 22 repress
7/16/2013 11:57:00 GMT EVA 23 started (PET=0:00)
7/16/2013 12:35:00 GMT EV2's CO2 Sensor went off scale high.
7/16/2013 12:41:00 GMT EV2 reported "I feel a lot of water on the back of my head, but I dont think it is
from my bag."
7/16/2013 12:44:00 GMT EVA suggested to EMU that water may be from vent port (ground conversation)
7/16/2013 12:45:00 GMT Ground IV asked EV2 to try and identify water source and if it is increasing
7/16/2013 12:46:00 GMT EV2 reported "I still feel it and I cannot tell you the source. My only guess is that
it came out of my bag and then found its way over there in the back, but I dont
have any water in the front of the helmet."
7/16/2013 12:51:00 GMT EV2 reported "the leak is not from the water bag and it is increasing",
unprompted.
7/16/2013 12:51:30 GMT EVA Officer asked EVA Backroom if the EVA should be terminated
7/16/2013 12:52:00 GMT EV2 drank remainder of water in Drink Bag
7/16/2013 12:53:00 GMT EV1 visually estimated 1/2 L of water at back of EV2's head
7/16/2013 12:54:00 GMT Flight Director discussed with EVA Officer the fact that EV2s drink bag was
empty, and that appeared to be the likely source of the water.
7/16/2013 12:57:00 GMT EV2 stated "Im thinking that it might not be the water bag", unprompted.
7/16/2013 12:57:30 GMT EV1 questioned sweat or urine as leak source.
7/16/2013 12:58:00 GMT EVA Officer discussed with FD the water tanks and sweat as other possible water
sources.
7/16/2013 13:00:00 GMT EV2 questioned LCVG as a possible leak source
7/16/2013 13:02:00 GMT EV2 stated the water is mostly at the back of his head
7/16/2013 13:05:00 GMT Ground IV called Terminate EVA
7/16/2013 13:06:00 GMT EV2 headed towards Airlock
7/16/2013 13:08:00 GMT Intermittent comm with EV2 began.
7/16/2013 13:09:00 GMT Start of night-pass (sunset)
7/16/2013 13:26:00 GMT Airlock hatch was closed and locked
7/16/2013 13:29:00 GMT EVA 23 ended (nominal repress started)
7/16/2013 13:35:00 GMT Crew lock hatch opened
7/16/2013 13:38:00 GMT Helmet off

31
 National Aeronautics and Space Administration EVA23Water Intrusion

3.4 Extravehicular Mobility Unit (EMU) Operational Description

The unanticipated introduction of a large quantity (1 to 1-1/2 liters) of water into EV2s helmet focused
immediate attention upon the operation of the EMU PLSS subsystems, as well as other elements of the
system, including the water-bearing components of the Space Suit Assembly (SSA). In order to facilitate
understanding of what items were investigated; how the investigations were carried out; and the meaning
of results, some basic understanding of the EMU and its operation is necessary. The following sections
seek to provide this understanding. The description of the EMU, as well as its overall performance
requirements, are presented first. Subsequent sections delve into the subsystems and individual
components themselves, with an emphasis on those featuring interaction with the various sources of water
and potential points of entry into the ventilation loop.

3.4.1 System Level EMU Description

The Extravehicular Mobility Unit (EMU) (Figure 3-1 and Figure 3-2) is an independent anthropomorphic
system that provides environmental protection, mobility, life support and communication for ISS
crewmembers to perform Extravehicular Activity (EVA) in Earth orbit. EVA is defined for EMU design
considerations as activity that occurs in environmental pressure below 4.0 psia.

The EMU system is comprised of two main assemblies, the pressure garment (also known as the Space
Suit Assembly or SSA) and the Portable Life Support System (PLSS) with the attached Secondary
Oxygen Package (SOP). As seen in Figure 3-1, the two assemblies are covered in an outer garment (the
Thermal Micro-meteoroid Garment or TMG) that acts as a barrier both to the thermal extremes of space
and to impacts due to micro-meteoroids, cuts, and punctures. The SSA provides the pressurized
environment, thermal management, and pressurized mobility for the astronaut wearing the suit. This
assembly is comprised of layers of materials which provide several functions. Innermost is a coated nylon
bladder which retains the pressurized gas inside the suit. Surrounding the bladder is a pressure restraint
garment which carries the load of the suit pressure. Outside the restraint are five layers of rip-stop scrim
and aluminized Mylar, which provide thermal isolation. Finally, the TMG surrounds these inner layers.
The thickness of these layers is approximately half an inch, and less on the gloves, which enables
astronauts to perform complex tasks like the intricate and delicate repairs to the Hubble Space Telescope
or replacement of large modules such as a Pump Module on the ISS.

The PLSS and SOP provide the life support, power and communication systems. The main components
that are found in the PLSS/SOP assembly are the space-to-space radio, the high-pressure primary and
secondary oxygen tanks, the primary and secondary water tanks for cooling, the fan/pump/separator, the
METOX canister for CO2 removal, and the water sublimator for cooling. These systems are monitored by
the Enhanced Caution and Warning System (ECWS) and controlled by the EVA crewmember using the
Display and Control Module (DCM).

Astronaut Electrocardiogram (ECG) and EMU performance parameter data are telemetered to the ground
via a Real Time Data System (RTDS). The EMU data signals are provided once every two minutes, at the
end of the ECG transmission. This arrangement was a compromise, since the EMU data items were added
after the system was designed to provide continuous ECG data.

32
 National Aeronautics and Space Administration EVA23Water Intrusion

The PLSS is equipped to support astronauts for a seven hour EVA with an hour contingency; however,
the actual maximum length of the EVA is determined by the individual metabolic rate of the astronauts
and the thermal environment of the EVA.

The EMU is designed to accommodate an EVA mission consisting of the following characteristics:

a) Total duration of 7 hours maximum under nominal solar exposure or 6 hours maximum under the
worst case solar exposure.
b) An average crewmember metabolic rate of 1000 Btu/hr for 7 hours or 850 Btu/hr for 8 hours.
c) Peak crewmember metabolic rates of 2000 Btu/hr for 15 minutes and 1600 Btu/hr for 1 hour at
any time within the EVA.
d) Minimum rate of 350 Btu/hr for 30 minutes after an average work rate of 1000 Btu/hr and
followed by a rate of 700 Btu/hr for up to 30 minutes.
The EMU major subsystems, the Primary Life Support Subsystem (PLSS), the SOP and the Space Suit
Assembly (SSA), and other associated support and ancillary equipment are shown schematically in Figure
3-3.

Figure 3-1. NASA EMU High-Level Description Illustrating Components of EMU

Pressure Garment and Life Support Assemblies

33
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-2. Extravehicular Mobility Unit Components

The Portable Life Support System (PLSS) (Figure 3-3) is made up of four distinct circuits:

a) Oxygen Ventilation Circuit.

b) Primary Oxygen Circuit
c) Feedwater Circuit
d) Liquid Transport Circuit
These circuits provide suitable breathing environment and thermal control for the crewmember inside the
Space Suit enclosure. The Primary Oxygen Circuit supplies oxygen at regulated pressure to the Oxygen
Ventilation Circuit for crewmember breathing, SSA pressurization and expulsion of water from storage
tanks in the PLSS. The fan portion of the fan/pump/separator, located in the Oxygen Ventilation Circuit,
circulates the oxygen and gas exhaled by the crewmember through connecting ducts and items. The
exhaled CO2 is removed from the breathing environment by a contaminant control cartridge. Cooling and
humidity removal occurs as the oxygen circulates through the sublimator. Additionally, primary oxygen is
also directed through ducting to pressurize the water storage tanks located in the Feedwater Circuit. The
water storage tanks supply makeup water to the coolant loop as well as expendable water at regulated
pressure to the sublimator. The Feedwater Circuit in the sublimator is open to atmosphere (vacuum) and,
as the expendable water is fed into the sublimator, it freezes and then sublimates to space. This
sublimation process removes heat from the Liquid Transport Circuit. The pump circulates cooling water
through the Liquid Transport Circuit which includes the Liquid Cooling and Ventilation Garment of the
SSA, worn by the crewmember. The water removes heat from the crewmember and is cooled by the
sublimator.

34
 National Aeronautics and Space Administration EVA23Water Intrusion

The Secondary Oxygen Pack (SOP) (Figure 3-3) attaches to the bottom of the PLSS and provides 30
minutes of back-up pressure regulated oxygen in the event that

1) The primary oxygen supply is exhausted.

2) Primary pressure regulation is lost via a regulator failure or excessive oxygen demand
3) One of the Purge Valves is opened during the following events: the PLSS fails to remove
contaminants, or heat or humidity in the ventilation circuit or ventilation flow drops below
acceptable levels.
The Display and Controls Module (DCM) contains the controls required to operate the EMU, as well as
status and warning displays (Figure 3-3). It also is the point of functional interface with the vehicle
systems via the ISS EMU Umbilical (IEU) for recharging and draining the PLSS water storage system;
recharging the PLSS primary oxygen tanks; providing electrical power and communications prior to and
following EVA; and providing cooling water supply and return from the ISS environmental control
system prior to and following EVA.

In addition to providing the functions listed above, the IEU contains a load-bearing tether to prevent
loading of connectors.

The following sections will concentrate on the subsystems and components of the EMU Space Suit
Assembly (SSA) and Life Support System (LSS) which are involved in the specific investigative steps
discussed in succeeding sections and Figure 3-3 should be consulted throughout the following discussion.
Item numbers will be cited frequently throughout this section of the report, and they are listed beside their
identified components in (Figure 3-3).

35
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-3. Annotated EMU Schematic

36
 National Aeronautics and Space Administration EVA23Water Intrusion

3.4.2 Ventilation Circuit

37
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-4. Item 140, Sublimator Vent Flow Path

38
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-5. Ventilation pad

Figure 3-6. HUT Water and Ventilation Lines

39
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-7. Liquid Cooling Ventilation Garment (LCVG) Ducts

3.4.3 Cooling Loop

40
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-8. Sublimator Cooling Loop Cross Section

Figure 3-9. Sublimator Slurper

41
 National Aeronautics and Space Administration EVA23Water Intrusion

3.4.4 Water Separator Loop

Figure 3-10. Fan Pump Separator (I-123) Cross Section

42
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-11. Water Separator Operation

43
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-12. Gas Trap Assembly (I-141)

3.4.5 ISS EMU Coolant Loop

44
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-13 Simplified Schematic of ISS Cooling Loop and EMU

3.4.6 Airlock Cooling Loop Recovery (ALCLR) Kit

Although the SPCU heat exchanger was replaced by one incorporating a double braze to mitigate
corrosion, the ALCLR kit was developed to scrub the circulating water to provide an extra measure of
security for the EMU cooling water loop by keeping the water loop system free of ionic, organic and
particulate contamination. It was launched aboard STS-114 in July of 2005.

The kit is designed to be installed at the Multiple Water Connector (MWC) of the EMU. The kit uses
activated carbon, ion exchange resin, filtration, and iodination to control contamination formation. Two
configurations are used: the scrubbing version is shown in Figure 3-14 and the special kit for iodinization
is shown in Figure 3-15. The scrubbing configuration incorporates an ion filter that uses activated carbon
to remove organic contaminants and an ion exchange resin to remove ionic contaminants. It also uses a 3-
micron filter to remove particulate contaminants. The iodinization uses an ion exchange resin to iodinate
the coolant loops to ~4 ppm iodine.

Current procedures involve an ALCLR scrub no earlier than 4 weeks before an EVA, and after every
EVA. A scrub is also required if a SEMU is idle for 90 days.

45
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-14 ALCLR Kit

Figure 3-14 shows the ALCLR Components EMU Water Processing Jumper (Shown with Ion and
Micron Filters Attached).

Figure 3-15 Iodination Kit

3.4.7 Maximized EMU Ground Activity (MEGA) Certification Process

Faced with the imminent prospect of limited access to on-orbit EMUs for comprehensive ground-based
maintenance after Shuttle retirement, the Maximized EMU Ground Activity (MEGA) program was
instituted in 2006 to permit orbital stay times of 6 years and 25 EVAs. All EMUs that saw ISS use after
Shuttle retirement, including those currently aboard the ISS as well as those planned for future ISS use,
have been refurbished using the MEGA process.

As part of the MEGA process, the aluminum water tank structure is stripped and recoated, the valve
module is dipped in a solvent for cleaning, and the sublimator is thoroughly cleaned and refurbished. The
remaining components are checked for limited-life and operation, and the assembled suit goes through a
vacuum certification process (24 hours of unmanned operation in a vacuum chamber, using simulated
metabolic loading, and 6 hours of manned metabolic-challenge in a vacuum chamber).

The significance of the MEGA process is that it means that EMU hardware will be left on-orbit for up to
6 years without teardown inspections or detailed component testing.

46
 National Aeronautics and Space Administration EVA23Water Intrusion

3.5 Concurrent Investigations

Assessment of the suit water intrusion event required that a careful investigation of the space suit
hardware be carried out, and this section of the report will focus on the ground-based and on-orbit tests
and evaluation performed in support of the ISS Investigation Team and MIB investigations.

3.5.1 Initial Activities

Following the water intrusion event on EVA 23, an ISS Investigation Team was formed. Members of the
team included the Crew and Thermal Systems Divisions (CTSD) EMU Subsystem Manager (SSM),
members from United Technologies Aerospace Systems (UTAS), Johnson Space Center (JSC) Safety
and Mission Assurance, JSCs Mission Operations Directorate (MOD), and JSCs EVA Office.

The ISS Investigation Team determined that there are several potential sources which could result in
water entering the ventilation circuit. The Disposable In-suit Drink Bag (DIDB) (Figure 3-16) is an
obvious candidate, since it contains up to 32 ounces of water and is in the proximity of the helmet.
Another potential source is the waste collection pull-up absorbent garment. A third possibility is leakage
from the cooling water side to the ventilating gas stream side of the sublimator heat rejection component
(Figure 3-9). A fourth potential leak source is the LCVG connector or the tubing itself (Figure 3-17).
Leakage of the cooling water transfer lines through the HUT is a fifth potential source (Figure 3-6).

47
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-16. Disposable In-suit Drink Bag (DIDB)

48
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-17. LCVG Tubing and Connector

A sixth possibility is water leakage from the water storage tankage, out through the pressurizing bladders,
and into the Item 120 Dual Mode Relief Valve circuit, through an orifice and eventually into the suit at
port T11 (Figure 3-3 and Figure 3-6).

A seventh possible source is the water separator circuit, which consists of the spinning drum-and-pitot
components (Figure 3-18 showing the path of water leaking into the ventilation loop via the fan inlet),
the Item 134 check valve and filter (Figure 3-19), the Item 125 pump priming valve (Figure 3-20), the
Item 141 gas trap (Figure 3-12) and the water flow passages linking the various elements of the circuit.

The possible leak sources discussed above, as well as any others identified, were used by the ISS
Investigation Team (IIT) and the Flight Safety Office of JSCs Safety and Mission Assurance Directorate
to construct a detailed fault tree (Appendix G). A systematic investigative program was then developed to
eliminate non-contributors to the leakage failure and ultimately identify the proximate cause.

49
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-18. Water Separator Spinning Drum and Pitot Components

Figure 3-19. Check Valve and Filter I-134

50
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-20. Pump Priming Valve I-125

3.5.2 Initial Investigative Steps

51
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-21. EMU SOP Check Out Fixture (SCOF)

52
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-22. SOP Regulator Shown in OFF Position

53
 National Aeronautics and Space Administration EVA23Water Intrusion

3.5.3 Condition of the Other EMUs On Orbit

Considering the potentially severe consequences of the type of failure experienced by EMU 3011, the
community determined that it would be prudent to ascertain the current status of other EMUs on orbit at
this time. Four EMUs are currently aboard the ISS: 3005, 3010, 3011 and 3015. EMU 3015 is not
considered operable, due to the inability of its sublimator to remove the required amount of heat during
EVA 18 on 8-30-12. The cause of the problem was found by ground-based analysis of a water sample
taken from SEMU 3015 to be contamination of the sublimator porous plate. This cause is merely a

54
 National Aeronautics and Space Administration EVA23Water Intrusion

proximate cause and no intermediate or root causes for this failure are yet known. The exact composition
and source of this sublimator contamination have not been determined. Furthermore, there is no capability
for on-orbit replacement of an EMU sublimatorsuit was not designed for this replacement on-orbit, it
has not been attempted before and no spare sublimators exist onboard ISS. EMUs 3005, 3010 and 3011
were considered at low risk of the sublimator problem, based on satisfactory thermal performance during
prior EVAs. The question arose, however, as to the status of EMUs 3005 and 3010 with respect to the
leakage problem exhibited by EMU 3011. A screening test was devised to subject the water separator
circuits of these units to the stress of operation under IVA conditions. The severity of the IVA condition
arises from the fact that the ventilating circuit load on the fan motor is greater than that during EVA
because of the increased density of the ventilating stream during the higher pressure of IVA (a factor of
over 3.5). This results in a slower motor speed (17,300 rpm during IVA versus around 19,300 rpm during
EVA) which means that the water collection drum spins more slowly. This translates to lower dynamic
pressure at the pitot, and therefore less ability to pump water.

Accordingly, on 8-14-13, sequential screening testing was carried out on EMUs 3005 and 3010. The
configuration of each EMU involved full-up suited unmanned operation at IVA pressure (approximately
one psi above ambient) with helmets installed. TP Bs were closed, and the Item 125 pump priming
valves were manually actuated (Figure 3-20) in order to assure that the water separator circuits were
operating. Each EMU was left operating in this condition, and an examination for any quantity of water
was carried out by the crew. No water was found at either the helmet vent pad (outlet of T2) or at the
CCC outlet (fan inlet.) No change occurred in either the fan speed or the current, nor was any CO2 sensor
fault noted, which indicated that the fan impeller and CO2 sensor were probably not exposed to water.
The conclusion reached was that neither EMU 3005 nor EMU 3010 exhibited the failure condition
present on EMU 3011 at the time of the test.

3.5.4 Specific Testing of the EMU 3011 Water Separator Circuit

In looking at probable offenders in the water separator circuit, one candidate immediately suspected was
the filter just upstream of the Item 134 check valve. This selection was due in large part to previous
experiences with this filter becoming clogged as well as its location supporting a likely proximate cause
scenario. Partial obstruction of this filter would prevent adequate flow through the Item 134 valve, and the
result would be inability of the pitot pump to circulate all of the flow coming from the gas trap through
the Item 125 valve (Figure 3-3). The excess water would then flood the water separator drum and enter
the ventilation loop at the fan inlet (Figure 3-19).

Failure of the Item 134 itself was a less likely possibility since check valves typically fail open. An open
valve would not have caused the observed condition since flow through the valve is the normal
operational condition. If the valve were to fail fully or partially closed, it could then have caused the
failure, but the design of the valve (Figure 3-19) would make either of these failure conditions unlikely. In
view of this, it was decided to develop a procedure for on-orbit removal and replacement of the Item 134
check valve and filter, and ship replacement items to the crew on orbit for a component substitution
followed by retest.

At this point, a decision was made to perform a repeat screening test of EMU 3011 using on-board high-
speed data acquisition and display capabilities not previously used. The rationale for repeating the test
was not to check again for the failure condition, but to acquire a better signature for the failure, and in

55
 National Aeronautics and Space Administration EVA23Water Intrusion

so doing, help provide a finer-grained observation of operation during the failed condition. With the
improved data in hand, a repeat of the screening test after removal and replacement of the Item 134 check
valve and filter could provide a better means of determining the status of the EMU. Previous testing had
been performed using a sample rate of one reading every two minutes, while the high-rate data provided a
sample rate of 10 Hertz (Hz) which is stored on an onboard laptop to be downloaded at a later time. There
was a concern about imposing yet another wetting of the fan motor bearings, which have been known to
suffer long-term degradation by exposure to water, but the value of obtaining better data was deemed
more important than the potential for damaging the bearings, since they had been wetted several times
already and the fan motor appeared to keep functioning normally.

Accordingly, on 8-27-13, a screening testing similar to that carried out on EMUs 3005 and 3010 was
performed. A chart of results is presented as Figure 3-23.

FAN SPEED

Figure 3-23. EMU 3011 Screening Test Data

The plot of particular interest is the one for fan speed. Just after 13:39, the Item 125 was manually
actuated, and almost immediately the fan speed went into a steep decline, indicating that the fan was
experiencing severe loading, presumably from a quantity of liquid water. At approximately 13:42, water
was noticed in the helmet. At about 13:51, the Display and Control Module (DCM) purge valve (Item 314
in Figure 3-3) was opened, effectively reducing the ventilation circuit pressure. As can be seen from the
data curves, fan speed immediately increased as pressure was lowered. At around 13:58, the oxygen
supply was shut off, which resulted in venting of the pressurizing oxygen in the water system to ambient.
As can be seen from the plot of fan speed, the fan was rapidly cleared of water, which allowed the fan
speed to increase back to normal. The reason for this effect occurring before the time that the water tank

56
 National Aeronautics and Space Administration EVA23Water Intrusion

pressurization was reduced is that the water which leaked into the ventilation loop was being replaced by
the pressurized stored water in the EMU tankage. The location tying the water separator loop to the
coolant loop is through the Item 141 gas trap, with its 8 to 10 lbm/hour (hr) flow rate to the water
separator through the Item 125 and 134 (Figure 3-3). Depletion of this flow due to leakage is
compensated by water flowing from the tanks into the coolant loop through the Item 171 solenoid valve
(open when the fan is operating) (Figure 3-3). The significance of the test was to establish a good baseline
for subsequent testing with the Item 134 valve and filter replaced.

3.5.5 Removal and Replacement of the Item 134 Valve and Filter, and Subsequent Retest
Detailed procedures for on-orbit removal and replacement of the Item 134 valve and filter had been
developed by NASA and ground processing contractor Stinger Ghaffarian Technologies (SGT).
Engineering and technician personnel working at the SGT facility in Houston coordinated with UTAS in
Windsor Locks, CT. Along with the replacement Item 134 valve and filter, special tools had been placed
aboard Progress flight 52P, and received aboard the ISS. On 8-31-13, the Item 134 valve and filter were
removed. The condition of the filter is shown in Figure 3-24. As may be seen in the photo, small amounts
of gray and white material are present, but not in sufficient quantities to block flow and cause the leakage
into the suit loop. (A discussion of the makeup of the contamination found in these items is presented in
Section 3.5.7 below.)

Figure 3-24. Item 134 Valve and Filter from EMU 3011 Post-Anomaly

Following replacement of the valve and filter with the new items, a repeat of the screening test was
performed, and results are shown in Figure 3-25. Although specific annotation is lacking, it has been
verified that the Item 125 pump priming valve was manually actuated at about 11:20, and from there
forward, the fan speed trace is similar to that produced in the preceding test, indicating that the problem
still existed.

57
 National Aeronautics and Space Administration EVA23Water Intrusion

FAN SPEED

Figure 3-25. EMU 3011 Screening Test Data Post I-134 R&R

While the Item 134 and filter were removed from the EMU, a syringe filled with water was inserted into
the location of the Item 134 valve bore and water was injected to determine if the flow path was
obstructed. Clogging of the flow passage downstream of the Item 134 was another possible cause that
could result in water in the vent loop, and although this exercise did not yield any definitive evidence of
blockage or contamination, final resolution would await the removal and replacement of the Item 123
FPS.

3.5.6 Removal and Replacement of the Item 141 Gas Trap Cartridge, and Subsequent
Retest
The next step taken was the removal and replacement of the Item 141 gas trap cartridge. This item is
meant to be replaced on-orbit, so no special procedure development was required. Rationale for removal
of this item for return to earth was based on two desires: first and foremost to capture any water residing
in the filter for analysis; second, to analyze any residue on the 20 micron filter screen (Figure 3-27). Since
a missing or damaged O-ring could cause leakage through the gas trap, the condition of these seals was of
interest. Examination of the seals on the gas trap revealed no O-rings missing or damaged (blue O-rings in
Figure 3-26).

58
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-26. Item 141 Gas Trap from EMU 3011 Post-Anomaly

Figure 3-27. Item 141 Gas Trap from EMU 3011 Post-Anomaly Inspection

Following the replacement of the cartridge, the screening test was repeated, with results as shown in
Figure 3-28. Lack of annotation notwithstanding, it may be seen that the results are similar to the other
two tests. Thus, the gas trap was removed from consideration as a cause or contributor to the leakage
condition.

59
 National Aeronautics and Space Administration EVA23Water Intrusion

FAN SPEED

Figure 3-28. EMU 3011 Screening Test Data Post I-141 R&R

3.5.7 Ground-based Analysis of the Item 134 Filter and Valve, and the Item 141 Gas Trap
The removed items from EMU 3011, the Item 134, its filter, and the Item 141 cartridge, along with lint-
free wipes used to swab the parts were packed for return and analysis. The hardware items were placed in
non-powdered, nitrile surgical gloves and the wipes in individual plastic baggies. They were returned to
UTAS for detailed analysis of any contamination or anomalous conditions on 9-19-13.

By 9-26-13, preliminary findings were presented by UTAS. The Item 134 filter was found to be ~90%
clear. The particulate contamination was made up of what is considered to be normal ferrous and
aluminum corrosion products, as well as fluorinated organics from lubricants. Some zinc and silica were
noted, along with one cellulose fiber. The amounts and types fell within allowable limits which the
program had deemed acceptable via a contamination book. This practice was deemed to be an example of
unwarranted acceptance of what really amounts to off-nominal situations. Recommendation R-6 was
generated to include investigation of instances of contamination, along with instances of water in the
helmet.

The Item 134 valve itself passed all leakage, cracking pressure and flow tests with values well within
specification.

60
 National Aeronautics and Space Administration EVA23Water Intrusion

The Item 141 gas trap had contained only about 1 cc of water, which was diluted 50:1 and sent out for
analysis, which showed 3 parts per million (ppm) manganese and 3.8 ppm silica, based on the original 1
cc of water. Contamination makeup similar to the Item 134 filter was found, along with a few silver
iodide particles; the source of which is still not known definitively. Microbial growth cultures were taken,
and readings after about 8 weeks showed small quantities of microorganisms which were not considered
to be contributory to the observed problem. Overall, the condition of the gas trap after some 148 hours of
operation was deemed excellent (Figure 3-26).

3.5.8 Removal and Replacement of the Item 123 Fan-Pump-Separator

61
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-29. Wire Insertion from TP B

62
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-30. Absorbent Plug in Bore

Figure 3-31. Areas of Concern for Fan Strikes

63
National Aeronautics and Space Administration EVA23Water Intrusion

64
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-32. Fan Outlet to Sublimator Inlet Adaptor

65
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-33. Face Seal O-ring

3.5.9 Post R&R Verification and SEMU 3011 Screening

On 10-25-13, after the return to configuration of the SEMU, the pump was primed using the procedures
previously described. The system was brought into operation with the FPS operating and the water
storage system pressurized. TP B was opened, and the Item 125 pump priming valve depressed, which
allowed water to flow through the Item 141 gas trap and into the FPS water separator circuit. The
expected golf-ball size mix of air and water was observed at TP B, and the crew reported that the fan

66
 National Aeronautics and Space Administration EVA23Water Intrusion

sounded normal. TP B was closed, and no water leaks were observed. Cooling water flow was detected by
noting a coolant temperature drop at the sublimator water flow outlet, displayed on the DCM. The helmet
was installed and an internal suit pressure of about 0.8 psid was noted. A globule of water was observed
near TP B, but the consensus was that this was left over from the golf ball effluent previously noted.
The water was absorbed onto wipes, and no recurrence was noted over the remainder of the operation.

The system was allowed to run for a total of about an hour and ten minutes with no leakage evident at any
location. After the helmet was removed, another inspection was performed of the suit interior and, again,
no evidence of leakage was found.

The conclusion drawn was that the leakage problem encountered during EVA 23 was caused by a failure
condition residing somewhere in the FPS water separator components. Figure 3-34 presents a comparison
of fan speed before any of the replacement activities, and after the I-134, I-141 and I-123 replacements.
This figure graphically illustrates the effect of water carryover on fan speed. The constant fan speed
observed after the I-123 R&R is in stark contrast to the decrease in speed experienced while the failure
condition was still present.

Figure 3-34. Comparison of Fan Motor Speed and UIA Current Before and After I-134, I-
141, and I-123 Replacement

67
 National Aeronautics and Space Administration EVA23Water Intrusion

3.5.10 Post-flight Evaluation Planning

With the isolation of the failure condition to the FPS assembly, plans were formulated to package and
return the FPS and the associated contamination samples and wipes to the ground for detailed evaluation.

The evaluation plan consisted of four parts:

a. NDE X-ray CT scanning of the FPS without the Item 127 filter and 128 check valve, and
scanning of those items separately.
b. Teardown and examination of the FPS itself.
c. Separate N-ray CT scanning of the water separator drum and pitot independently.
d. Examination and analysis of all contamination and wipes, including any material found during the
disassembly.
A pathfinder exercise using a correctly configured FPS was outfitted with some representative
contamination, and both X-ray and N-ray Computer Tomography were used to determine sensitivity and
best orientations for scanning. Results were positive, and these techniques were planned for use on the
discrepant flight article, which was returned from the ISS in mid-November.

3.5.11 Post-flight Evaluation of the FPS and ALCLR Filters

68
National Aeronautics and Space Administration EVA23Water Intrusion

69
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-35. Pitot Detail

Figure 3-36. Contamination Site at Drum

70
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-37. Contamination in Drum

71
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-38. Typical of 8 Blocked Holes

Figure 3-39 Residue on Pump Housing and Corroded Braze Joint

72
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-40. Contamination At and In Drum Hole

73
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-41. N-Ray of Pitot

74
National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-42. FPS Performance Test Schematic

75
 National Aeronautics and Space Administration EVA23Water Intrusion

3.5.12 Forward Planning

A Technical Interchange Meeting (TIM) was held at UTAS in Windsor Locks, CT, on 12- 4 and 12-5-13.
JSC Engineering, Safety, Mission Operations, and Flight Crew were represented. The participants
discussed what had been learned, what remained to be investigated, and what modifications to the
hardware and operations should be implemented. They also discussed various means of achieving a
greater degree of confidence in the EVA hardware left aboard ISS.

76
 National Aeronautics and Space Administration EVA23Water Intrusion

Before a down-select and task definitions could occur, the EMU program was overtaken by events. The
coolant loop pump package problem experienced by ISS during mid-December, 2013, emphasized the
urgency of not only finding the root causes of the suit water intrusion incident, but also to determine
means of achieving safe conduct of EVA before root cause identification and corrective action
implementation. The coolant loop is one of several ISS systems which depend on EVA availability for
risk mitigation.

The replacement of the FPS and other components of SEMU 3011, and the ALCLR ion exchange bed
analyses described above, along with the operational histories of SEMUs 3005 and 3010 were evaluated
by a special team of JSC and UTAS experts, and their conclusions were that all three SEMUs were
acceptable for at least three EVAs the projected number required to change out a defective pump
module package in the ISS coolant loop. SEMU 3011 was estimated to have slightly more margin than
SEMU 3005, but either would be acceptable to be used in concert with SEMU 3010.

The ISS Investigation Team will continue the hardware failure investigation to determine root causes and
will identify corrective actions, along with a plan and schedule for implementation. The MIB Appointing
Official will determine the requirements for closure of this investigation and will determine the
continuing involvement of the Mishap Investigation Board.

77
 National Aeronautics and Space Administration EVA23Water Intrusion

3.6 Root Cause Analysis

This investigation followed two distinct paths: 1) what caused the failure in the hardware and 2), how the
operations community dealt with the failure during the EVA. First, the hardware failure has been traced to
inorganic materials blocking the EMU water separator drum holes. This failure had not been experienced
during EVA before and is still undergoing a concurrent investigation. The results of this investigation will
ultimately lead to resolution of this issue, however, since the concurrent investigation into the source of
the debris is expected to continue for many months, the MIB does not yet have the required data to
determine the root causes of the contamination source. The root causes of this issue must ultimately be
determined to prevent future mishaps. The focus on the hardware failure investigation centers on
understanding of the hardware involved, work that has been completed to date, preliminary results, and
future work needed to determine root causes. Discussion of the hardware investigation can be found in
ECFT-2 of our Event and Causal Factor Tree. The latest version of the Engineering Fault Tree which
details the possible causes of this EMU failure is included as Appendix G. Second, the report then focuses
on the real-time operations activities that can be improved to help the ground control teams more quickly
recognize and react as quickly as possible to emergencies of this type. Discussions of the operations
investigation can be found in ECFT-1, ECFT-3 and ECFT-4 of our Event and Causal Factor Tree.

To determine causes and contributing factors of the event, the MIB applied the NASA Root Cause
Analysis (RCA) method. The Undesired Outcome (UO) was identified as EVA crew member (EV2)
exposed to potential loss of life during EVA 23 to reflect the severity of the HVCC related to not just
water in the helmet but also the response to that event. Next, the MIB established a Timeline of events
and conditions that were relevant to this investigation, capturing significant historical events related to the
suit development, management decisions, and analyses up to and immediately following the mishap. The
Timeline of Events is shown in Table 3-1.

Concurrent with Timeline development, the MIB identified key events directly before the UO and
brainstormed possible causes. Proximate Causes, the events or conditions that occurred immediately
before or existed at the time of the UO, were established and the NASA Root Cause Analysis Tool
(RCAT) Fault Tree model was used to identify and capture possible causes. The MIB employed two aids
to ensure a broad scope was covered in our brainstorming. SHELL-D, which stands for Software,
Hardware, Environment, Liveware (Team), Liveware (Individual), and Documents from the NASA RCA
training and PPPEE, or Paper, People, Part, Equipment, and Environment, from our consultants Failure
Recovery Planning training.

As data was gathered, elements on the Fault Tree (FT) were ruled out with disputing data or ruled in
where there was sufficient data to support causal logic. All of the substantiated causal events, conditions,
and contributing factors that were ruled in were reflected on an Event Causal Factor Tree (ECFT)
(Appendix I). The ECFT tree was expanded (discussed in the following sections) by continually asking
why for the elements above until a logical endpoint emerged. The RCA Tool produced an .rca file for
the mishap. The .rca file for this mishap RCA will be maintained in IRIS along with this report.

Items that were identified as Significant but not causal were captured and are discussed throughout
Section 3.6.2 as Contributing Factors, and in Section 3.7 Observations.

78
 National Aeronautics and Space Administration EVA23Water Intrusion

The MIB generated recommendations to address causal factors, contributing factors and observations
identified in this HVCC to avoid recurrence of this or similar mishaps in the future.

During the course of this investigation, the MIB identified an additional undesired outcome which was
addressed as a Secondary Undesired Outcome (SUO). The SUO, Inadvertent activation of EMU 3011
SOP during Vent Loop Wet/Dry Vacuum and Dryout activity (SUO), occurred during the EMU 3011
Vent Loop troubleshooting and dryout activity. See Section 2.5 for details of this SUO.

The MIB ECFT shown throughout Section 3.6 and in Appendix I correspond to the following legend:

Orange Rectangle Undesired Outcome

Green Rectangle Event
Blue Oval Condition
White Crosshatch Contributing Factor

79
 National Aeronautics and Space Administration EVA23Water Intrusion

3.6.1 Proximate Causes

The MIB defined the three Proximate Causes and one contributing factor (Figure 3-43), which led to the
PUO and are discussed in the following sections.

EVA crew member (EV2) exposed to

potential loss of life during EVA 23.

UO-1

ISS Program performed EVA 23 EMU 3011 Helmet had large quantitiy of Flight Control Team/Crew did not Flight Control Team sent EV2 back to
w ithout recognizing EMU failure on w ater during EVA 23. terminate EVA as soon as w ater w as Airlock unaccompanied during EVA
EVA 22. reported in the helmet. termination.
ECFT-1 ECFT-2 ECFT-3 ECFT-4

Figure 3-43. ECFT Proximate Causes

ECFT-1 ISS Program performed EVA 23 without recognizing EMU failure on EVA 22.
(Proximate Cause 1) (Figure 3-44)

Supporting Evidence: The MIB learned through interview, audio recordings and transcript review that
on EVA 22, EV2 in suit 3011 experienced water in the helmet during repress. This failure was
misdiagnosed and not determined to be a constraint to EVA 23. The MIB has determined that had the
source of the water at the end of EVA 22 been investigated thoroughly, EVA 23 and the subsequent
HVCC would not have occurred. The reasons why ECFT-1 occurred are discussed in the following
sections.

ECFT-2 EMU 3011 Helmet had large quantity of water during EVA 23. (Proximate Cause
2)(Figure 3-45)

Supporting Evidence: During US EVA 23 on July 16, 2013, EVA Crewmember 2 (EV2) (wearing
SEMU 3011) experienced an abnormal and large amount of water inside the helmet area, originating
somewhere behind the crewmembers head near the neck/lower head area.

The MIB confirmed through video, transcripts of EVA and ground communication loops, and interviews
that the quantity of water in EV2s helmet was estimated to be 1 to 1.5 liters. Combined with the other
proximate events and given the behavior of the water in the limited volume of the helmet, this condition
was life threatening.

In the course of troubleshooting the suit failure, several components were removed from SEMU 3011 and
replaced with new or recently refurbished parts. Replacing the Fan/Pump/Separator component in SEMU
3011 ultimately corrected the water intrusion issue. Due to the late return of the failed
Fan/Pump/Separator from ISS, the root cause for the hardware failure has yet to be determined. Once the
subsequent investigation (NDE, teardown and evaluation, and analysis of all contamination and suspect
material) is carried out, further analyses toward root cause may be pursued.

ECFT-3 Flight Control Team/Crew did not terminate EVA 23 as soon as water was reported in
the helmet. (Proximate Cause 3)(Figure 3-46 through Figure 3-67)

80
 National Aeronautics and Space Administration EVA23Water Intrusion

Supporting Evidence: The MIB determined through interview and transcript review that the Phased
Elapsed Time between first mention of water in EV2s helmet and the call to terminate the EVA was
roughly 23 minutes. The fact that no one on the Flight Control Team (FCT) or the EVA Crew,
immediately recognized the severity of the hazard and terminated the EVA is discussed in the sections
below.

ECFT-4 Flight Control Team sent EV2 back to Airlock unaccompanied during EVA termination.
(Contributing Factor 1)(Figure 3-68)

Supporting Evidence: The MIB determined from interviews, audio recordings and transcript reviews
that, when the decision to terminate the EVA was made, EV2 was sent back to the Airlock
unaccompanied which contributed to the severity of the event. At that time, it was still daylight and EV2
was not yet reporting difficulties seeing due to the water accumulating in his helmet. In addition, EV2s
translation to the Airlock from his location near Node1 was fairly short. Conversely, EV1s safety tether
was routed such that his translation back to the Airlock needed to be via a different, less direct route than
EV2s. This Safety Tether arrangement was specifically planned to avoid EV1 and EV2 tangling tethers.
EV1 commented during a post-flight public interview on NASA TV that, as EV2 started his translation
toward the Airlock, he felt he should probably have gone with him. However, this would have
necessitated him dropping his safety tether and tethering to EV2 resulting in his safety tether being left
outside and several clean up steps being left undone. EV1 accompanying EV2 back to the Airlock
would also have necessitated further discussions with the Flight Control Team ultimately delaying EV2
from getting there as expeditiously as possible, which was the intention.

81
 National Aeronautics and Space Administration EVA23Water Intrusion

3.6.2 Intermediate Causes, Root Causes and Contributing Factors

The following sections discuss the Intermediate, Root Causes, and Contributing Factors that led to the
Proximate Causes described above. Figure 3-41 shows the Intermediate and Root Causes under ECFT-1.

ISS Program performed EVA 23

w ithout recognizing EMU failure on
EVA 22.
ECFT-1

FCT/Crew incorrectly attributed Water

in helmet during EVA 22 to the DIDB.

ECFT-1.1

Flight Control Team did not perform

investigation of w ater source.

ECFT-1.1.1

Flight Control Team accepted the Ground team allow ed time pressures FCT's perception of the anomaly report
explanation that the w ater w as from of impending EVA to influence actions. process as being resource intensive
the drink bag. made them reluctant to invoke it.
ECFT-1.1.1.1 ECFT-1.1.1.2 ECFT-1.1.1.3

ISS Community Perception w as that Program Pressure w as to maximize

drink bags leak. crew time on orbit for utilization.

ECFT-1.1.1.1.1 ECFT-1.1.1.2.1

Figure 3-44 Intermediate and Root Causes under ECFT-1

ECFT-1.1 Flight Control Team and Crew incorrectly attributed the water in the helmet during
EVA 22 to the drink bag. (Intermediate Cause 1)

Supporting Evidence: MIB determined from EVA debrief and crew member interviews that EV2 had
indicated that he saw some water escape past his bite valve during EVA 22 repress. This might have
happened because his chin was resting on the bite valve while he was in a tucked position. Audio logs and
interview transcripts revealed that the ground team accepted the crews determination of cause for the
EVA 22 water leak. The team perceived that additional investigation of the water leak was not necessary,
because the source had already been identified and agreed upon. They also considered that a lengthy
investigation could impede preparations for EVA 23 for a reason likely would likely turn out to be
inconsequential.

Crew and ground team training did not prepare the team with an adequate understanding of how the EMU
could fail with respect to water in the helmet. Had this been done, the crew and ground team may not
have attributed water in the helmet to just the drink bag. This branch was causal or contributory to several
key events and conditions of this HVCC.

82
 National Aeronautics and Space Administration EVA23Water Intrusion

The Human Factors Analysis (Appendix A) findings support the fact that the team was predisposed to
determine the drink bag was the cause of the water. See Appendix A.1.1 PC 214 Response Set for more
detail on ECFT-1.1.

ECFT-1.1.1 Flight Control Team did not perform investigation of the water source following EVA
22. (Intermediate Cause 2)

Supporting Evidence: After the conclusion of EVA 22, the team perceived that the water in EV2s
helmet probably came from EV2s drink bag and that the solution was to provide a new drink bag prior to
the next EVA, per routine. The team also perceived that lengthy investigations were unnecessary and
could impede preparations for EVA 23, which was scheduled for the following week. Audio logs and
interview transcripts revealed that the team, as a whole, did not perform a thorough real-time risk
assessment; instead, the crews determination of cause for the EVA 22 water leak was accepted by the
ground team (Engineering, Safety, Operations). The Human Factors Analysis (Appendix A) findings
support the fact that the team was predisposed to determine the drink bag was the cause of the water and
that further investigation would be time-consuming and unproductive. See Appendix A.1.1 PC 214
Response Set for more detail.

ECFT-1.1.1.1 Flight Control Team accepted the explanation that the water during EVA 22 was
from the drink bag. (Intermediate Cause 3)

Supporting Evidence: Based on operator console audio loops and interviews, it appears this theory was
first suggested by EV1 shortly after the conclusion of EVA 22 after EV2 had indicated that he thought he
saw some water escape past the bite valve on his drink bag. After the conclusion of EVA 22, the EVA
team perceived that all the water in EV2s helmet probably came from EV2s drink bag and that the
solution was to provide a new drink bag prior to the next EVA, per routine.

Additionally, after EVA 22, some of the team perceived that lengthy investigations were unnecessary and
could impede preparations for EVA 23, which was scheduled for the following week

Crew and ground team training did not prepare the team with an adequate understanding of how the EMU
could fail with respect to water in the helmet. Had this been done, the crew and ground team may not
have accepted the explanation that the water was from the drink bag.

See Appendix A.1.1 PC 214 Response Set for more detail.

ECFT-1.1.1.1.1 ISS Community Perception was that drink bags leak. (Root Cause 2)

Supporting Evidence: After EVA 22 ended, the crew concluded that the water which collected in EV2s
helmet had likely leaked from his drink bag inadvertently. Based on MIB interviews, it was determined
that one reason that ground team members did not challenge the crews conclusion was the perception
that drink bags leak. Additionally, the way in which the drink bag was supposed to have leaked made
sense, according to the ground teams understanding of its design. However, the MIB found no recorded
evidence of a drink bag leak during an actual EVA other than small amounts of water (droplets)
introduced into the helmet due to inadvertent actuation of the bite valve. The MIB has concluded that the

83
 National Aeronautics and Space Administration EVA23Water Intrusion

perception that drink bags leak, especially as a frequent occurrence, is false. The MIB further found that
this drink bag failure conclusion was not questioned by any of the other ground teams, supporting the
contention that a drink bag leak was considered a normal event that could be corrected by simply
replacing the failed drink bag.

No further analysis was done on this ECFT legRoot Cause was reached. See Appendix A.1.2
Perceptions of Equipment for more detail.

ECFT1.1.1.2 Ground Team allowed time pressures of impending EVA to influence actions.
(Intermediate Cause 4)

Supporting Evidence: After EVA 22, the team perceived that lengthy meetings were not possible,
because of the high ops tempo involved in preparing for EVA 23. Essentially, the Flight Control Team
accepted the crews assessment of the EVA 22 water leak and chose not to investigate further.

According to post mishap interview transcripts, more than one team member indicated that they wished
they had called a time-out. However, EVA 23 was scheduled for the following week, which left little
time to prepare.

There was also a perception that if the question concerning the source not being the drink bag was raised,
it would invoke a fairly resource intensive and potentially cumbersome process involving Engineering
and Safety for what most felt would likely turn out to be a non-issue. This would have an impact on EVA
23 preparations. In hindsight, however, it is now apparent that EVA 23 should not have commenced until
the EVA 22 issue had undergone a more adequate evaluation. That is not to say that a lengthy formal risk
assessment was required (that may, or may not be the case), just that the EVA 22 water leak deserved a
more refined assessment of risk. Had that been done, the EVA 23 HVCC might not have occurred.

See Appendix 0

84
 National Aeronautics and Space Administration EVA23Water Intrusion

AE201 Risk Assessment During Operation and Appendix A.1.7 OP001 Ops Tempo/Workload for more
detail.

ECFT1.1.1.2.1 Program emphasis was to maximize crew time on orbit for utilization. (Root Cause
1)

Supporting Evidence: From interviews with team members across multiple disciplines, and MIB Board
experience, it is clear that the ISS Program strongly emphasizes that crew time on orbit should be used to
maximize utilization (performance of science). Due to this knowledge, team members felt that requesting
on-orbit time for anything non-science related was likely to be denied and therefore tended to assume
their next course of action could not include on-orbit time. The danger with that thought process was that
lower level team members were in effect making risk decisions for the Program, without necessarily
having a Program wide viewpoint or understanding of the risk trades actually being made at a Program
level. The MIB recognizes the need to emphasize utilization of the ISS Program; it is the very reason for
its existence. However, the implementation of that emphasis is having the undesired effect of removing
the risk assessment process from the Program and performing it at a lower level.

No further analysis was done on this ECFT legRoot Cause was reached.

Recommendation R-1: The ISS Program must reiterate to all team members that, if they feel that
crew time is needed to support their system, a request and associated rationale must be elevated to
the ISS Program for an appropriate decision.

See Appendix A.1.6 Organizational Influences for more detail.

ECFT-1.1.1.3 Flight Control Teams perception of the anomaly report process as being resource
intensive made them reluctant to invoke it. (Root Cause 3)

Supporting Evidence: Based on interviews, there was a perception that if the question concerning the
source of water at the end of EVA 22 not being the drink bag was raised, it would invoke a fairly resource
intensive and potentially cumbersome process involving Engineering and Safety for what most likely
would turn out to be a non-issue. This would have an impact on EVA 23 preparations. This also points to
a significant cultural issue between Operations and Engineering teams. The MIB learned during
interviews that the Operations team feels that sometimes the support they receive from the Engineering
teams is not conducive to real-time operations where decisions often must be made quickly using the best
data available at the time and good engineering judgment. The Engineering Teams feel that the Flight
Control Teams sometimes make decisions too quickly or that they do not feel comfortable relying on
judgment calls. This lack of trust and understanding about roles and responsibilities between the two
teams and associated recommendations are discussed in Section 3.7 Observations.

No further analysis was done on this ECFT legRoot Cause was reached.

The Human Factors Analysis (Appendix A) Appendix 0

85
 National Aeronautics and Space Administration EVA23Water Intrusion

AE201 Risk Assessment During Operation and Appendix A.1.6 Organizational Influences has more
detail.

EMU 3011 Helmet had large quantitiy of

w ater during EVA 23.

ECFT-2

Large quantity of w ater from vent loop

leaked into helmet.

ECFT-2.1

Figure 3-45 Intermediate Cause Directly Under ECFT-2

86
 National Aeronautics and Space Administration EVA23Water Intrusion

ECFT-2.1 Large quantity of water from vent loop leaked into helmet during EVA 23. (Intermediate
Cause 5)

Supporting Evidence: The MIB determined through video transcripts of EVA and ground
communication loops, and interviews that the water in EV2s helmet was originating somewhere behind
the crewmembers head near the neck/lower head area. Post EVA, water was noted in and around the T2
vent port which is at the base of the crewmembers head at the back of his neck (reference Figure 3-6).

The MIB confirmed through video, that the quantity of water in the Astronauts helmet was estimated to
be 1 to 1.5 liters. Given the limited volume of the helmet and the behavior of the water, this condition was
life threatening.

The MIB in conjunction with the ISS Investigation Team evaluated multiple sources of water in the suit.
A significant effort was put into developing a detailed Fault Tree for the hardware failure (see Appendix
G). This fault tree was developed and controlled by the ISS Investigation Team, but the MIB participated
with the ISS Investigation Team and concurred with its accuracy and sufficiency. Investigations are
continuing to resolve the cause(s) of the water in the vent loop. Additional root causes will be found.

Proximate Cause ECFT-3 addresses the fact that no one called to terminate EVA 23 as soon as water was
identified in the helmet which contributed to the severity of the event. The discussion below addresses the
events and conditions relative to each team, starting with the EVA 23 Crew (Figure 3-46).

Flight Control Team/Crew did not

terminate EVA as soon as w ater w as
reported in the helmet.
ECFT-3

EVA 23 Crew did not immediately Ground Team did not immediately
recognize the severity of the event recognize severity of the event.

ECFT-3.1 ECFT-3.2

Figure 3-46 Intermediate Causes Directly Under ECFT-3

ECFT-3.1 EVA 23 Crew did not immediately recognize the severity of the event. (Intermediate
Cause 6)

Supporting Evidence: The MIB learned through discussions captured on audio loops between the
ground control teams and the EVA crew members that neither of the EVA 23 crew initially recognized
the severity of the water pooling at the back of EV2s head. EV2 did not indicate a sense of urgency with

87
 National Aeronautics and Space Administration EVA23Water Intrusion

respect to the water in his helmet. EV1 recognized the fact that water pooling in the helmet was not a
nominal behavior of the EMU and participated in the questioning of EV2 about the source. The EVA 23
crew concurred with the ground teams initial recommendation to continue the EVA since they believed
the source of the water was not increasing. Prior to getting a direct visual on EV2, EV1 only had the
information coming to him via the loop communications and his own experience with the behavior of the
EMU. By GMT 12:51 (10 minutes after first call of water), EV1 had a direct visual of EV2s face and saw
water pooling on the side of EV2s head. At this time, EV2 also expressed concern about the quantity of
water in his helmet and, within minutes, the decision was made to terminate the EVA. The terminate
decision is a less rapid response than an abort call and EV1 concurred with the decision to terminate.
EV1 was concerned about EV2s condition as he watched EV2 navigate towards the airlock and started
suspecting that the condition of EV2 with respect to the water in his helmet was more severe than first
believed.
EVA 23 Crew did not immediately
recognize the severity of the event

ECFT-3.1

FCT focused on the drink bag as the Crew Member Training did not include Critical Information w as not Airflow Contamination Procedure did EV2 did not have the experinence
source of the w ater. this failure mode. communicated betw een Crew and not address the failure mode. base to recognize the severity of the
Ground Team situation.
ECFT-3.1.1 ECFT-3.1.2 ECFT-3.1.3 ECFT-3.1.4 ECFT-3.1.5

Figure 3-47 Direct Intermediate Causes and Contributing Factor Under ECFT-3.1

Figure 3-48 expands the causes under ECFT 3.1.1 Team focused on the drink bag as the source of the
water.

88
 National Aeronautics and Space Administration EVA23Water Intrusion

FCT focused on the drink bag as the

source of the w ater.

ECFT-3.1.1

Team had Channelized Attentioned on Team's set of responses led to a delay

the drink bag as the primary source of in identifying the vent loop as the
w ater in the helmet source of the w ater leak.
ECFT-3.1.1.1 ECFT-3.1.1.2

FCT/Crew incorrectly attributed Water

in helmet during EVA 22 to the drink
bag.
ECFT-3.1.1.1.1

Flight Control Team did not perform

investigation of w ater source.

ECFT-3.1.1.1.1.1

Figure 3-48 Causes Under ECFT 3.1.1

ECFT-3.1.1 Flight Control Team focused on the drink bag as the source of the EVA 23 water.
(Intermediate Cause 8)

Supporting Evidence: In reviewing the Space-to-Ground communication loops during EVA 23, it is
clear that despite some conversations about other possible sources of water between the ground and the
EV crew, the conversation between EV1 and EV2 initially discussed the drink bag as the possible source
of water. The Flight Control Team was also focused on the possibility that the drink bag was the problem.

ECFT-3.1.1.1 Team had Channelized Attention on the drink bag as the primary source of water
in the helmet. (Intermediate Cause 12)

Supporting Evidence: Based on interviews and comm loop recordings, it was found that the ground
team (Engineering, Safety, Operations) primarily focused on EV2s drink bag as the possible source of
water in his helmet. Other suggestions included accumulation of sweat and leakage from the LCVG, but
both were quickly dismissed. Channelizing on the drink bag may have prevented the team from
continuing to ask questions to come up with a different answer or ask new and more specific questions
that would have pointed to something other than the drink bag, such as the temperature of the water.

When the CO2 sensor failed early in the EVA at GMT 12:35, most of the team believed that it failed due
to a nominal accumulation of moisture in the vent loop. Since nominal water carryover only results in a
limited/manageable amount of water in the helmet, the significance of the CO2 sensor failure was quickly

89
 National Aeronautics and Space Administration EVA23Water Intrusion

disregarded, despite the fact that this type of failure almost always occurred near the end of a long EVA.
No one on the team recognized the relationship between the early failure of the CO2 sensor and an
abnormally large amount of water in the vent loop until much later.

See Appendix A.2.3 PC102 Channelized Attention for more detail.

ECFT-3.1.1.1.1 FCT/Crew incorrectly attributed water in helmet during EVA 22 to the drink
bag. (Intermediate Cause 1)

Supporting Evidence:

The MIB learned from the EVA 22 Debrief and interviews that the crew and ground attributed the water
in the EMU 3011 helmet at the end of EVA 22 to a leak in the crew members drink bag.

EV2 had indicated that he saw some water escape past his bite valve during repress and that maybe this
had happened because his chin was resting on the bite valve while he was in a tucked position. Audio logs
and interview transcripts revealed that the ground team accepted the crews determination of cause for the
EVA 22 water leak.

After EVA 22, the team perceived that additional investigation of the water leak was not necessary,
because the source had already been identified and agreed upon. They also considered that a lengthy
investigation could impede preparations for EVA 23.

Crew and ground team training did not prepare the team with an adequate understanding of how the EMU
could fail with respect to water in the helmet. Had this been done, the crew and ground team may not
have attributed water in the helmet to just the drink bag.

The causes for this event were discussed previously in ECFT-1.1 branch: FCT/Crew attributed Water
in the helmet during EVA 22 to the drink bag. The Human Factors Analysis (Appendix A) findings
support the fact that the team was predisposed to determine the drink bag was the cause of the water. See
Appendix A.1.1 PC 214 Response Set for more detail on ECFT-1.1.

ECFT-3.1.1.2 Team's set of responses led to a delay in identifying the vent loop as the source of
the water leak. (Intermediate Cause 13)

Supporting Evidence: After the conclusion of EVA 22, the team erroneously perceived that the water in
EV2s helmet came from EV2s drink bag. Based on the audio logs and interview data, we find that early
in the course of EVA 23 events, the team began to inquire about the source of the water and the one
question that was repeated by multiple team members was: is the water coming from his drink bag?
This line of questioning was reinforced by the framework of expectations that arose from EVA 22.

The early failure of the CO2 sensor provided another opportunity to suspect something other than the
drink bag as the source of the water. The CO2 sensor sometimes fails during EVA ops, a fact that is
generally known (and attributed to sublimator carry-over) by all ground team personnel who monitor it. It
failed earlier than usual during EVA 23, because of excessive water in the vent loop. This was an off
nominal event, yet the team missed its significance because of normalization of deviance.

See Appendix 0

90
 National Aeronautics and Space Administration EVA23Water Intrusion

PC214 Response Set for more detail.

Crew Member Training did not include

this failure mode.

ECFT-3.1.2

MOD did not understand the failure

mode.

ECFT-3.1.2.1

EMU Hazard Report did not identify the Minor amounts of w ater in the helmet Engineering Team did not understand Safety Team did not understand the
hazard. w as normalized. the failure mode. failure mode.

ECFT-3.1.2.1.1 ECFT-3.1.2.1.2 ECFT-3.1.2.1.3 ECFT-3.1.2.1.4

Figure 3-49 Intermediate Causes Under ECFT-3.1.2

ECFT-3.1.2 Crew Member Training did not include this failure mode. (Intermediate Cause 7)

Supporting Evidence: Overall, EVA training is very extensive and comprehensive, however there are
areas that can and should be improved. From the transcripts of the MCC loops and subsequent discussions
with flight control, engineering, and contractor personnel, it is evident that no one really considered that
the water was coming from the PLSS, especially in the quantity experienced. As discussed in the Human
Factors section of this report (Appendix A), this was due to a combination of factors. If one reviews the
failure history of the EMU as well as the FMEA/CIL one will find ample evidence of problems with the
EMU vent loop/water loop interface causing anomalous water in the vent loop which could lead to all of
the PLSS water flooding the helmet in zero-G.

The Board performed an analysis of the training flows of MOD, Safety, and Engineering personnel
involved in the incident. Although crew and flight controller training is extensive in the area of EMU
operations, from discussions with personnel involved with the incident, awareness of these failure modes
seems to be missing. In looking at the training flows for EVA flight controllers, EVA MER personnel,
and EVA safety personnel, it is evident that relative to the rest of the curriculum there is not much time
spent explicitly learning the various failure modes or history of the EMU. The EVA flight control training
is often used by the rest of the community as the basis of their training as well, so deficiencies in the
MOD training curriculum has impacts across the community.

MOD personnel are initially exposed to the EMUs failure history as a new EVA operator or instructor in
the EVA core training flow where they must study a variety of sources including the EMU mini-data
book, the EMU Specification and Assembly Drawing Document (SAD), the EMU Requirements
Evolution Document, EVA Lessons Learned, and the FMEA/CIL, and hazard reports. However, for the
most part, these references are only studied during this initial training on the road to becoming a full EVA
Systems instructor while one is still relatively new and without experience and while absorbing a lot of
new information. These sources are not required reading for proficiency or as part of subsequent training

91
 National Aeronautics and Space Administration EVA23Water Intrusion

after one has significant experience. Overt study of the FMEA/CIL is also very limited in the training
flows for MER Engineering and non-existent for EVA Safety personnel. In particular, it was found that
the only area requiring study of the FMEA/CIL was in support of the MER 1 Level 2 Specialist position
for the specific components of the METOX, ALCLR, IEU, and FPU. Otherwise the training flows do not
require reference to the FMEA/CIL.

Recommendation R-2: ISS Program should ensure that the FMEA/CILs are updated and
maintained and MOD should make them required reading/study for all EVA Systems instructors
and Flight Controllers up to and including FCR operators as well as their proficiency flows. EVA
safety and Engineering MER support personnel should also include this in their training flows.
Recommendation R-3: MOD SSTF instructors should ensure that training includes use of the
FMEA/CIL to develop failure scenarios for use in integrated and stand-alone simulations.

ECFT-3.1.2.1 MOD did not understand the failure mode. (Intermediate Cause 11)

Supporting Evidence: MOD is responsible for writing procedures and conducting crewmember training.
The MIB learned through interview and discussions with EVA crew members, crew trainers, and flight
controllers that MOD relies on a variety of resources to learn about the functionality of the ISS
subsystems including the EMU and Airlock. This knowledge and collaboration with the Engineering and
Safety communities forms the basis for flight rules, procedures and training materials. Fundamentally,
four factors led to MOD not understanding this failure mode to the extent necessary to identify its
severity. These factors included the hazard reports did not address the hazard of water in the helmet to the
extent experienced, water in the helmet had been normalized over time, and neither did the engineering
nor safety communities understand the failure.

EMU Hazard Report did not identify the

hazard.

ECFT-3.1.2.1.1

FMEA/CIL did not effectively describe FMEA/CIL did not effectively quantify
the behavior of w ater entering the the amounts of w ater entering the vent
vent loop from the PLSS. loop from the PLSS.
ECFT-3.1.2.1.1.1 ECFT-3.1.2.1.1.2

No one applied our know ledge of the FMEA/CIL did not undergo thorough
physics of w ater behavior in zero g to review and update periodically.
w ater coming from the PLSS vent loop
ECFT-3.1.2.1.1.1.1 ECFT-3.1.2.1.1.1.2

ISS program FMEA/CIL requirement did Program cut funding.

not require complete FMEA/CIL periodic
update.
ECFT-3.1.2.1.1.1.2.1 ECFT-3.1.2.1.1.1.2.2

Community had lost sight of the value

of the FMEA/CIL effort.

ECFT-3.1.2.1.1.1.2.2.1

92
 National Aeronautics and Space Administration EVA23Water Intrusion

Figure 3-50 Causes and Contributing Factors under ECFT 3.1.2.1.1

ECFT-3.1.2.1.1 EMU Hazard Report did not identify the hazard. (Intermediate Cause 16)

Supporting Evidence: The MIB conducted a thorough review of EVA hazard reports. The EVA hazard
report, EMU-13, that address water in the helmet is entitled Contamination Control. Loss of Visibility.
Hazard Cause: Loss of visibility due to fogging of lenses (Helmet, DCM)

Section E of EMU-13 hazard report applies to this failure: Water carryover into the re-circulating vent
circuit and loss of SOP/Purge operation. The hazard report then goes on to list all of the relevant failure
modes identified in the FMEA that can cause this hazard. The hazard level is listed as Critical. It then
goes on to list all of the controls on the hazard.

Nowhere in the hazard report does it mention the possibility of excessive water in the helmet resulting in
a catastrophic event due to asphyxiation.

Recommendation R-4: The ISS Program should ensure that updates are made to the EMU
hazard reports to reflect the possibility of water in the helmet resulting in a catastrophic event due
to asphyxiation.

ECFT-3.1.2.1.1.1 FMEA/CIL did not effectively describe the behavior of water entering the vent
loop from the PLSS. (Intermediate Cause 19)

Supporting Evidence:
The EMU FMEA/CIL lists the following failure modes causing PLSS water in the vent loop that could
potentially enter the helmet:

102FM22 (LCVG Multiple Water Connector HUT half)

123FM04 (Fan/Pump/Water Separator)
123FM07 (Fan/Pump/Water Separator)
125FM02 (Pitot Actuated Valve or Pump Priming Valve)
134FM02 (Condensate Water Relief Valve)
140FM04 (Sublimator)
140FM05A (Sublimator)
141FM05 (Gas Trap)

All are considered 2/1R failures and thus are considered critical items. In each case the result of the
failure mode is listed as water carryover in the vent loop and possible helmet fogging. Inconsistently,
various FMEA/CILs mention the fact that there are 9 lbm of water in the PLSS and nearly all have the
following or similar wording for the operational response (this particular quote is from 140FM04): If
helmet fogging occurs or significant amounts of water detected exiting helmet vent duct, terminate EVA
per EVA Cuff Checklist. Open helmet purge valve to defog helmet and provide cooling to helmet area.

93
 National Aeronautics and Space Administration EVA23Water Intrusion

Additionally, 123FM04 states under GFE Interface: Water carryover into the space suit assembly.
Discharge of up to 10 lbm water from the water tanks into the suit. But then goes on to say, Potential
helmet fogging. Once again, nowhere does the FMEA specifically mention that the result is the water
will be in the helmet and would present a more serious hazard such as asphyxiation due to water
inhalation.

The reasons for this are unclear, but from anecdotal evidence from those involved in the initial
development of the EMU and its operations, there was a belief by the general EVA community that if a
significant amount of water was introduced into the vent loop it would stall the fan. This was supported
by ground testing experience. A failed fan leads to lack of cooling in the suit and loss of humidity control
in the vent loop which leads to helmet fogging. If this is the rationale for the conclusion that one should
open the purge valve and terminate the EVA, then that assumption should have been stated in the FMEA.
It is not there.

Additionally, the physics and behavior of the water once in the vent loop was not well understood in this
case. In the past it was thought that significant water in the vent loop, if it would make it past the fan,
would manifest itself as a kind of rain in the helmet being propelled by the force of the air coming out
of the helmet vent pad at the back of the helmet. This was supported by experience in vacuum chamber
runs on the ground and the occasional slurper carry-over experienced in small quantities on-orbit. The
conclusion in the case of significant water in the helmet should have been that water will not rain in the
helmet and if it does shortly thereafter will clump together and stick to the nearest surface vent pad, com
cap, scalp, or visor. Why these facts were not put together for this case is not known.

Finally, the FMEA/CIL is out of date. The EVA community, MOD, Engineering, Safety and the
Astronaut Office should be able to use this tool to capture predicted possible failure modes and failures
that have been well documented in the past. This is not to say that this would have caught this particular
failure mode, but there is a chance that someone looking at this for the first time might have caught some
of the logical inconsistencies noted above.

Recommendation R-5: The ISS Program should ensure that the FMEA/CIL is updated and
reviewed thoroughly from end-to-end every two years to ensure currency with participation by
Engineering, MOD, Safety, Medical, and appropriate contractor personnel.

ECFT-3.1.2.1.1.1.1 No one applied knowledge of the physics of water behavior in zero-g to water
coming from the PLSS vent loop. (Root Cause 4)

Supporting Evidence: Through interviews and review of flight rules and procedures, the MIB learned
that while there is a significant amount of knowledge about the way water behaves in zero-gravity, the
ground teams did not properly understand how the physics of water behavior inside the complex
environment of the EMU helmet would manifest itself. Engineering teams had 1-g experience showing
that a significant amount of water in the vent loop would stall the vent fan. However, during this HVCC,
an amount of water that would normally stall the fan during ground testing was allowed to pass by the fan
and enter the helmet. The Engineering teams now believe that, in zero-g, the water can cling to the
interior walls of the fan housing and be passed through the fan assembly without stalling it, which they
presume happened during this HVCC. The MIB concurs with this explanation. In addition, the
Engineering teams informed the MIB that they believed that if a significant amount of water entered the

94
 National Aeronautics and Space Administration EVA23Water Intrusion

helmet, the air flow from the vent loop would force the water to streak over the top of the helmet and
down the front of the visor, possibly affecting the crew members visibility. From evaluation of this
event, the MIB and ground teams now know that the water entered the helmet, but did not streak over the
top and down the visor. Instead, surface tension forced the water to form near the outlet of the vent line
until the quantity was sufficient enough to contact the back of EV2s head. At that point, surface tension
brought the large amount of water to the back of EV2s head and it eventually made its way to the front of
EV2s head, covering his eyes and nostrils.

No further analysis was done on this ECFT legRoot Cause was reached.

ECFT-3.1.2.1.1.1.2 FMEA/CIL did not undergo thorough review and update periodically.
(Contributing Factor 9)

Supporting Evidence: From a review of the EMU FMEA/CIL documentation and interviews, it was
determined that the last re-baselining of the EMU FMEA/CIL was after the Challenger accident in 1986.
As configuration changes are made to the hardware, appropriate changes to the FMEA are made and the
CIL is reviewed and updated. During the CoFR process changes in dash numbers are noted and it is
confirmed that no FMEA updates are required. The review and updates to the FMEA are not
comprehensive and usually only the minimum necessary changes are made. The MIB noted several
instances where the failure history associated with a FMEA was either incomplete or in error. See section
ECFT-3.1.2.1.1.1 for more detail. The MIB determined that lack of FMEA/CIL review may not be
determined causal but increased the likelihood that the FMEA/CIL did not reflect the failure mode seen in
this HVCC.

ECFT-3.1.2.1.1.1.2.1 ISS program FMEA/CIL requirements did not require complete FMEA/CIL
periodic review. (Contributing Factor 11)

Supporting Evidence: From reviews of applicable ISS Safety and Mission Assurance documentation
(SSP-30234, Failure Modes and Effects Analysis and Critical Items List Requirements for Space Station)
and interviews with personnel, the MIB can find no requirement within the ISS Program to periodically
perform a comprehensive review of FMEAs. See ECFT-3.1.2.1.1.1 for more detail.

ECFT-3.1.2.1.1.1.2.2 Program cut funding. (Contributing Factor 12)

Supporting Evidence: From interviews and discussions with personnel, it was determined that an effort
was underway to bring the EMU FMEA/CILs and hazard reports into compliance with the ISS Program
requirements which should have resulted in a major review of the all of the documentation. This work
was suspended due to budget cuts this year.

ECFT-3.1.2.1.1.1.2.2.1 Community had lost sight of the value of the FMEA/CIL effort.
(Contributing Factor 13)

Supporting Evidence: From SSP-30234 on the purpose of the FMEA post design phase:

The FMEA provides documentation of the failure modes present in the system, the effects of failure
mode occurrence, the methods of detecting the failure, and corrective action taken to prevent effects of
failure (including restoration of function). For critical failure modes, and when required to be submitted

95
 National Aeronautics and Space Administration EVA23Water Intrusion

for risk acceptance, the retention rationale justifies use of the critical item in the system. The retention
rationale drives the inspection, process control, and test/verification requirements for the critical items;
influences operations planning (including mission planning, procedure development, and logistical and
maintenance support requirements); and reports failure history. Logistical and maintenance support
requirements could be impacted by acceptance of a critical item; therefore, consideration of the following
should precede formal acceptance of each critical item: total crew maintenance time allocations,
logistical capabilities of the system, and microgravity (probability of success) requirements.

In order to be effective in fulfilling its purpose, it is essential that the FMEA be kept current with the ISS
design and operational use.

From interviews and discussions with personnel, it was determined that updating the FMEA is primarily
viewed by many as a paperwork exercise and a tool to be mainly used by the S&MA community. This is
further evidenced in practice by the lack of time and effort taken to update and review the information
when it is deemed necessary to update as well as its lack of involvement in engineering risk discussions or
training. See Recommendation R-5 above.

ECFT-3.1.2.1.1.2 FMEA/CIL did not effectively quantify the amounts of water entering the vent
loop from the PLSS. (Contributing Factor 7)

Supporting Evidence: As discussed in ECFT 3.1.2.1.1.1, PLSS water in the vent loop is discussed in the
FMEA but does not directly address the quantity of water reaching the helmet.

MOD did not understand the failure

mode.

ECFT-3.1.2.1

EMU Hazard Report did not identify the Minor amounts of w ater in the helmet Engineering Team did not understand Safety Team did not understand the
hazard. w as normalized. the failure mode. failure mode.

ECFT-3.1.2.1.1 ECFT-3.1.2.1.2 ECFT-3.1.2.1.3 ECFT-3.1.2.1.4

Figure 3-51. Causes and Contributing Factors under ECFT 3.1.2.1 for reference

ECFT-3.1.2.1.2 Minor amounts of water in the helmet was normalized. (Root Cause 5)

Supporting Evidence: Through interviews with ground personnel and review of data from previous
EMU performance, it was clear that some water entering the helmet was considered normal by the ground
teams. EMU 3005 was even referred to as the wet EMU since its acceptance test data indicated that its
sublimator slurper was less efficient and led to a larger amount of water carryover into the vent loop.
Despite the fact that water carryover into the helmet presented a known hazard of creating eye irritation
due to its interaction with anti-fog agents, and also presented a potential fogging hazard, the ground teams
grew to accept this as normal EMU behavior. Since these smaller amounts of water carryover had never
caused a significant close call, it was perceived to not be a hazardous condition. When water began
entering EV2s helmet, the ground team discussed anti-fog/eye irritation concerns, and visibility

96
 National Aeronautics and Space Administration EVA23Water Intrusion

concerns, a more hazardous condition was not expected because water in the helmet behavior had been
normalized.

No further analysis was done on this ECFT legRoot Cause was reached.

Recommendation R-6: The ISS Program should ensure that all instances of free water and
contamination in the EMU are documented and investigated, with corrective action taken, if
appropriate.

Engineering Team did not understand

the failure mode.

ECFT-3.1.2.1.3

No one applied know ledge of the

physics of w ater behavior in zero g to
w ater coming from the PLSS vent loop
ECFT-3.1.2.1.3.1

Figure 3-52 Cause under ECFT 3.1.2.1.3

ECFT-3.1.2.1.3 Engineering Team did not understand the failure mode. (Intermediate Cause 17)

Supporting Evidence: MIB learned through interviews and discussions with Engineering team members
and flight controllers that, until this HVCC occurred, previous analysis of water in the helmet did not
describe the outcome that was experienced during this mishap.

ECFT-3.1.2.1.3.1 No one applied knowledge of the physics of water behavior in zero-g to water
coming from the PLSS vent loop. (Root Cause 4)

Supporting Evidence: See ECFT-3.1.2.1.1.1.1

97
 National Aeronautics and Space Administration EVA23Water Intrusion

Safety Team did not understand the

failure mode.

ECFT-3.1.2.1.4

No one applied know ledge of the Minimal Formal Training on EMU

physics of w ater behavior in zero g to function existed for the Safety Team.
w ater coming from the PLSS vent loop
ECFT-3.1.2.1.4.1 ECFT-3.1.2.1.4.2

Requirement for specific EVA/EMU

trianing of Safety personnel did not
exist.
ECFT-3.1.2.1.4.2.1

Figure 3-53 Root Cause and Contributing Factors under ECFT 3.1.2.1.4

ECFT-3.1.2.1.4 Safety Team did not understand the failure mode. (Intermediate Cause 18)

Supporting Evidence: MIB learned through interviews and discussions with Safety team members and
flight controllers that, until this HVCC occurred, previous analysis of water in the helmet did not describe
the outcome that was experienced during this mishap.

ECFT-3.1.2.1.4.1 No one applied knowledge of the physics of water behavior in zero-g to water
coming from the PLSS vent loop. (Root Cause 4)

Supporting Evidence: See ECFT-3.1.2.1.1.1.1

ECFT-3.1.2.1.4.2 Minimal Formal Training on EMU function existed for the Safety Team.
(Contributing Factor 8)

Supporting Evidence: From interviews of Safety and Mission Assurance personnel as well as a review
of their training records the MIB determined that NASA Safety and Mission Assurance personnel receive
no formal training on EVA systems and the S&MA contractor, SAIC, received only some basic systems
training as part of those personnel being certified to support the MER.

98
 National Aeronautics and Space Administration EVA23Water Intrusion

ECFT-3.1.2.1.4.2.1 Requirement for specific EVA/EMU training of Safety personnel did not exist.
(Contributing Factor 10)

Supporting Evidence: Interviews with Safety and Mission Assurance management and personnel as well
as a review of relevant documentation indicated that there are no requirements for formal systems training
set forth by S&MA. The S&MA organization depends largely on the existing skill and knowledge base of
the personnel hired into the organization from other jobs such as in Engineering or Mission Operations
for their systems knowledge. See Observation 26 in Section 3.7 for more discussion.

Critical Information w as not

communicated betw een Crew and
Ground Team
ECFT-3.1.3

EV1 initially focused on drink bag as

source in EVA 23.

ECFT-3.1.3.1

Crew incorrectly attributed w ater in Crew w as not trained to recognize

EV2's helmet after EVA 22 to the drink this failure mode.
bag.
ECFT-3.1.3.1.1 ECFT-3.1.3.1.2

Figure 3-54 Causes under ECFT 3.1.3

ECFT-3.1.3 Critical Information was not communicated between Crew and Ground Team.
(Intermediate Cause 9)

Supporting Evidence: Good communication is dependent on at least three factors: transmission,

reception, and comprehension. Based on audio loops and post HVCC interviews, it was determined that
there was a failure to receive critical information that had been transmitted during EVA 22 and 23.

The MIB learned through review of Space to Ground Comm Loop and interview that one of the
statements made by EV2 (shortly after EVA 22 ended) was understood differently by various team
members. Some thought he stated I didnt drink at all, while others thought he stated I didnt drink it

99
 National Aeronautics and Space Administration EVA23Water Intrusion

all. Those who understood his words to mean that he didnt drink at all would have less difficulty
believing that all the water in EV2s helmet could have come from his drink bag, which held
approximately one liter of water. That amount was similar to what was estimated to have collected in his
helmet toward the very end of EVA 22: to 1 liter of water.

During EVA 23, EV2 called down over the S/G loop (at PET 44 minutes) to say that he had a lot of water
at the back of his head and he didnt think it was from his drink bag. This was EV2s first call about water
and it was clearly transmitted. The call was acknowledged by the ground, but it seems that some on the
ground understood only the first half of the transmission. The team did not comprehend the critical
information that the water had not come from his drink bag. The amount of activity and communication
traffic occurring around the time of the call may have contributed to the confusion. In reaction to EV2s
first call, the team questioned EV2 as to whether or not the water had come from his drink bag. In
response, EV2 second-guessed his first assumption, so he made a second call soon after his first, at PET
49 minutes, to state that the drink bag may be suspect. Five minutes after the second call, EV2 made a
third call, at PET 54 minutes, in which he emphatically states that the leak was not from the water bag.
At this time, most of the team was heavily engaged with the issue of water in EV2s helmet, but they did
not clearly understand how much water was there, if it was increasing in quantity, or what temperature it
was. It is worth noting that water temperature was probably not discussed by the team, because they were
not focusing on the PLSS as the source of water. As discussed above and in following sections, this was a
failure mode that had not been adequately covered in previous training.

The comm loops were very busy with team members trying to resolve the problem, and there were many
conversations occurring off the loops as well. The MIB also noted that several times the ground team had
to ask the crew to repeat themselves, because critical communications were missed, or stepped on by
other transmissions. This may explain why some of this critical information remained unclear to most of
the team members and certainly to the Flight Director, who, at PET 57 minutes, states sounds like the
water source was the drink bag. This was in contradiction to what EV2 stated three minutes earlier, in
which he clearly transmitted that the water was not from his drink bag.

The examples above indicate there was a breakdown in communicating critical information, despite no
failure in the ability to make clear voice transmissions over the comm loops between the time when EV2
first declared water in his helmet and when the team elected to terminate the EVA.

See Appendix 0

100
 National Aeronautics and Space Administration EVA23Water Intrusion

PP106 Communicating Critical Information for more detail.

ECFT-3.1.3.1 EV1 initially focused on drink bag as source in EVA 23. (Intermediate Cause 14)

Supporting Evidence: In reviewing the Space-to-Ground communication loops during EVA 23, it is
clear that despite some conversations between the ground and the EV crew about other possible sources
of water, the conversation between EV1 and EV2 initially focused on the drink bag as a possible source
of water.

101
 National Aeronautics and Space Administration EVA23Water Intrusion

ECFT-3.1.3.1.1 Crew incorrectly attributed water in EV2's helmet after EVA 22 to the drink bag.
(Intermediate Cause 1)

Supporting Evidence: After EVA 22 ended, the audio logs and interview transcripts revealed that EV2
had indicated that he saw some water escape past his bite valve during repress and that maybe this had
happened because his chin was resting on the bite valve while he was in a tucked position. This
conclusion was consistent with their understanding of how the drink bag operated. See ECFT-1.1

ECFT-3.1.3.1.2 Crew was not trained to recognize this failure mode. (Intermediate Cause 7)

Supporting Evidence: See ECFT-3.1.2

Airflow contamination Procedure did

not address the failure mode.

ECFT-3.1.4

Flight Rule to address this FM did not

exist.

ECFT-3.1.4.1

MOD did not understand the failure

mode.

ECFT-3.1.4.1.1

Figure 3-55 Causes under ECFT 3.1.4

ECFT-3.1.4 Airflow contamination procedure did not address the failure mode. (Intermediate
Cause 10)

Supporting Evidence: The Air Flow Contamination procedure that has been in the EMU cuff checklist
since prior to 1989 lists steps to take in the event of LiOH dust or LiOH contaminated water in the EMU
helmet--open the helmet purge valve and terminate the EVA. One section of the procedure that would
have applied in this case, If excessive water in the helmet leads to a step that just states MCC with
no further instructions. This means contact Mission Control and await further guidance. No mention is

102
 National Aeronautics and Space Administration EVA23Water Intrusion

made of water coming from the PLSS in any quantity other than as flowing through the LiOH cartridge.
This procedure, as written, was inadequate to handle this failure.

ECFT-3.1.4.1 Flight Rule to address this failure mode did not exist. (Intermediate Cause 15)

Supporting Evidence: A review of the ISS Flight Rules in effect at the time of the HVCC showed that
there were no flight rules that addressed large quantities of water in the helmet.

The EMU FMEA/CIL has a number of failure modes that can result in water in the vent loop culminating
in water in the helmet. Water in the helmet in at least small quantities was considered nominal and caused
by sublimator carry-over the result of inefficiency of the sublimator slurper to remove condensate from
the vent loop.

Recommendation R-7: MOD must lead the development of appropriate flight rules and
procedures to address the course of action to take in the event of water in the helmet.

ECFT-3.1.4.1.1 MOD did not understand the failure mode (ECFT-3.1.2.1). (Intermediate Cause
11)

Supporting Evidence: See ECFT-3.1.2.1 branch.

EVA 23 Crew did not immediately

recognize the severity of the event

ECFT-3.1

Team focused on the drink bag as the Crew Member Training did not include Critical Information w as not Airflow Contamination Procedure did EV2 did not have the experinence
source of the w ater. this failure mode. communicated betw een Crew and not address the failure mode. base to recognize the severity of the
Ground Team situation.
ECFT-3.1.1 ECFT-3.1.2 ECFT-3.1.3 ECFT-3.1.4 ECFT-3.1.5

Figure 3-56 Causes and Contributing Factors under ECFT 3.1 for reference

ECFT-3.1.5 EV2 did not have the experience base to immediately recognize the severity of the
situation. (Contributing Factor 2)

Supporting Evidence: From EVA historical documentation and interviews, the MIB found that this was
EV2s second EVA. His first EVA was EVA 22 which occurred just one week prior to EVA 23. Prior to
spaceflight, EV2 had conducted one vacuum chamber run in an EMU on the ground. His chamber run
was uneventful but his first EVA resulted in water in his helmet. The water in his helmet during EVA 22
started appearing at the end of the spacewalk during Airlock re-pressurization. Since it was EV2s first
EVA, he assumed that water droplets and fog on his visor were normal Airlock re-pressurization
phenomena that no one had told him about in advance. When EV2s helmet was removed, EV1 saw what
he considered a lot of water (estimated at one-half to one liter) and asked EV2 where it all had come
from. In discussing it between themselves and with the ground team, the water was attributed to sweat
and water from the drink bag. EV2 didnt have the experience base or training to determine if this was a
serious issue or not and didnt want to cause the EVA to be terminated for a non-issue.

103
 National Aeronautics and Space Administration EVA23Water Intrusion

Ground Team did not immediately

recognize severity of the event.

ECFT-3.2

Flight Control Team did not understand Ground Team focused on the drink bag Critical Information w as not
the failure mode. as the source of the w ater. communicated betw een Crew and
Ground Team
ECFT-3.2.1 ECFT-3.2.2 ECFT-3.2.3

Figure 3-57 Causes and Contributing Factors under ECFT 3.2

ECFT-3.2 Ground Team did not immediately recognize the severity of the event. (Intermediate
Cause 6)

Supporting Evidence: It is clear through discussions captured on audio loops between the ground control
teams and the EV crew that the Ground Team did not initially recognize the severity of the water pooling
at the back of EV2s head. The Ground Team did recognize the fact that water pooling in the helmet was
not a nominal behavior of the EMU and participated in the questioning of EV2 (through the CAPCOM)
about the source. The ground teams initial recommendation was to continue the EVA since they believed
the amount of the water was not increasing. Prior to getting a direct visual on EV2 by EV1, the Ground
Team only had the information coming to them via the loop communications and their own experience
with the behavior of the EMU. By GMT 12:51, EV1 had a direct visual of EV2s face and saw water
pooling on the side of EV2s head. At this time, EV2 also expressed concern about the quantity of water
in his helmet and within minutes the Ground Team made the decision to terminate the EVA for EV2.

Flight Control Team did not understand

the failure mode.

ECFT-3.2.1

EMU Hazard Report did not identify the Engineering Team did not understand Safety Team did not understand the
hazard. the failure mode. failure mode.

ECFT-3.2.1.1 ECFT-3.2.1.2 ECFT-3.2.1.3

Figure 3-58 Causes under ECFT 3.2.1

ECFT-3.2.1 Flight Control Team did not understand the failure mode. (Intermediate Cause 11)

Supporting Evidence: See ECFT-3.1.2.1.

104
 National Aeronautics and Space Administration EVA23Water Intrusion

EMU Hazard Report did not identify the

hazard.

ECFT-3.2.1.1

FMEA/CIL did not effectively describe FMEA/CIL did not effectively quantify
the behavior of w ater entering the the amounts of w ater entering the vent
vent loop from the PLSS. loop from the PLSS.
ECFT-3.2.1.1.1 ECFT-3.2.1.1.2

No one applied our know ledge of the FMEA/CIL did not undergo thourough
physics of w ater behavior in zero g to review and update periodically.
w ater coming from the PLSS vent loop
ECFT-3.2.1.1.1.1 ECFT-3.2.1.1.1.2

ISS program FMEA/CIL requirement did Program cut funding

not require complete FMEA/CIL periodic
update.
ECFT-3.2.1.1.1.2.1 ECFT-3.2.1.1.1.2.2

Community had lost sight of the value

of the FMEA/CIL effort.

ECFT-3.2.1.1.1.2.2.1

Figure 3-59 Causes and Contributing Factors under ECFT 3.2.1.1

ECFT-3.2.1.1 EMU Hazard Report did not identify the hazard. (Intermediate Cause 16)

Supporting Evidence: See ECFT-3.1.2.1.1 branch, EMU Hazard Report did not identify the hazard,
for discussion of this and lower level causes.

105
 National Aeronautics and Space Administration EVA23Water Intrusion

Engineering Team did not understand

the failure mode.

ECFT-3.2.1.2

No one applied know ledge of the

physics of w ater behavior in zero g to
w ater coming from the PLSS vent loop
ECFT-3.2.1.2.1

Figure 3-60 Causes and Contributing Factors under ECFT 3.2.1.2

ECFT-3.2.1.2 Engineering Team did not understand the failure mode. (Intermediate Cause 17)

Supporting Evidence: See ECFT-3.1.2.1.3 branch for discussion of this and lower level causes.

Safety Team did not understand the

failure mode.

ECFT-3.2.1.3

No one applied our know ledge of the Minimal Formal Training on EMU
physics of w ater behavior in zero g to function existed for the Safety Team.
w ater coming from the PLSS vent loop
ECFT-3.2.1.3.1 ECFT-3.2.1.3.2

Requirement for specific EVA/EMU

trianing of Safety personnel did not
exist.
ECFT-3.2.1.3.2.1

Figure 3-61 Causes and Contributing Factors under ECFT 3.2.1.3

106
 National Aeronautics and Space Administration EVA23Water Intrusion

ECFT-3.2.1.3 Safety Team did not understand the failure mode. (Intermediate Cause 18)

Supporting Evidence: See ECFT-3.1.2.1.4 branch for discussion of this and lower level causes.

Ground Team did not immediately

recognize severity of the event.

ECFT-3.2

Flight Control Team did not understand Ground Team focused on the drink bag Critical Information w as not
the failure mode. as the source of the w ater. communicated betw een Crew and
Ground Team
ECFT-3.2.1 ECFT-3.2.2 ECFT-3.2.3

Figure 3-62 Causes and Contributing Factors under ECFT 3.2.for reference

ECFT-3.2.2 Ground Team focused on the drink bag as the source of the water. (Intermediate
Cause 8)

Supporting Evidence: Although shown as a Contributing Factor in this leg of ECFT-3.2, this was an
Intermediate Cause in ECFT-3.1.1. The Intermediate Cause 8 label is carried here to avoid duplication
and confusion. Audio logs and interviews indicate that the ground team focused primarily on the drink
bag as the source of the water leak. Because the drink bag was known to hold about one liter of water, it
would be understandable for the team to think that the leak would be self-limiting to a liter, or less. It was
reasonable to assume that the leak would be less if EV2 were to consume whatever water remained in the
drink bag, after the leak was made known, as he eventually did. Since the team focused on the drink bag,
and water leaking from the drink bag could be managed, the true severity of the condition went
unrecognized, that being the possibility of aspiration of much more water (over four liters) leaking from
the PLSS.

See ECFT-3.1.1(channelized attention), ECFT-1.1(water from drink bag in EVA 22), and ECFT-3.1.2
(response set)

107
 National Aeronautics and Space Administration EVA23Water Intrusion

Ground Team focused on the drink bag

as the source of the w ater.

ECFT-3.2.2

The Team had Channelized Attention Team's set of responses led to a delay Ground team did not identify PLSS as Cognitive Task Oversaturation
on the drink bag as the primary source in identifying the vent loop as the actual source of w ater. contributed to the teams delayed
of w ater in the helmet source of the w ater leak. identification of the actual source of
ECFT-3.2.2.1 ECFT-3.2.2.2 ECFT-3.2.2.3 ECFT-3.2.2.4

Figure 3-63. Contributing Factors under ECFT 3.2.2

The Team had Channelized Attention

on the drink bag as the primary source
of w ater in the helmet
ECFT-3.2.2.1

FCT/Crew incorrectly attributed Water

in helmet during EVA 22 to the DIDB.

ECFT-3.2.2.1.1

Flight Control Team did not perform

investigation of w ater source.

ECFT-3.2.2.1.1.1

Flight Control Team accepted the Ground team allow ed time pressures Flight Control Team's perception of the
explanation that the w ater w as from of impending EVA to influence actions. anomaly report process as being
the drink bag. resource intensive made them
ECFT-3.2.2.1.1.1.1 ECFT-3.2.2.1.1.1.2 ECFT-3.2.2.1.1.1.3

ISS Community Perception w as that Program emphasis w as to maximize

drink bags leak. crew time on orbit for utilization.

ECFT-3.2.2.1.1.1.1.1 ECFT-3.2.2.1.1.1.2.1

Figure 3-64. Contributing Factors under ECFT 3.2.2.1

108
 National Aeronautics and Space Administration EVA23Water Intrusion

ECFT-3.2.2.1 The Team had Channelized Attention on the drink bag as the primary source of
water in the helmet. (Intermediate Cause 12)

Supporting Evidence: Although shown as a Contributing Factor in this leg of ECFT-3.2.2.1, this was an
Intermediate Cause in ECFT-3.1.1.1. The Intermediate Cause 12 label is carried here to avoid duplication
and confusion. The ground team relied on its recent EVA 22 water leak experience with the drink bag
being the source of water leak in the helmet. This is a contributory factor and not an intermediate cause
because, in this segment of the ECFT, it refers to the actions of the ground team focusing on the drink bag
as the source of the water. Eliminating channelized attention from the scenario would not by itself have
resulted in the ground team recognizing the severity of the event; ultimately, the team did not understand
the PLSS failure mode, which is causal.

Other suggestions included accumulation of sweat and leakage from the LCVG, which were quickly
dismissed. Channelization may have prevented the team from continuing to ask questions to come up
with a different answer or ask new and more specific questions that would have pointed to something
other than the drink bag, such as the temperature of the water.

The sudden CO2 sensor failure led some of the team to believe that it failed due to a nominal
accumulation of moisture in the vent loop. Since nominal water carryover only results in a
limited/manageable amount of water in the helmet, the significance of the CO2 sensor failure was quickly
disregarded, despite the fact that this type of failure almost always occurred near the end of a long EVA.
No one on the team recognized the relationship between the early failure of the CO2 sensor and an
abnormally large amount of water in the vent loop until much later.

Previously discussed in ECFT-3.1.1.1. (Team focused on the drink bag as the source of the water)

ECFT-3.2.2.1.1 FCT/Crew incorrectly attributed water in helmet during EVA 22 to the DIDB.
(Intermediate Cause 1)

Although shown as a Contributing Factor in this leg of ECFT-3.2.2.1.1, this was an Intermediate Cause in
ECFT-1.1. The Intermediate Cause 1 label is carried here to avoid duplication and confusion.

See ECFT-1.1 branch.

ECFT-3.2.2.1.1.1 Flight Control Team did not perform investigation of the water source following
EVA 22. (Intermediate Cause 2)

Supporting Evidence: Although shown as a Contributing Factor in this leg of ECFT-3.2.2.1.1.1, this was
an Intermediate Cause in ECFT-1.1.1. The Intermediate Cause 2 label is carried here to avoid duplication
and confusion. After EVA 22, the team perceived that lengthy investigations were unnecessary and could
impede preparations for EVA 23, which was scheduled for the following week. Audio logs and interview
transcripts revealed that the team, as a whole, did not perform a thorough real-time risk assessment;
instead, the crews determination of cause for the EVA 22 water leak was accepted by the ground team.

See ECFT-1.1 branch.

109
 National Aeronautics and Space Administration EVA23Water Intrusion

ECFT-3.2.2.1.1.1.1 Flight Control Team accepted the explanation that the water during EVA 22
was from the drink bag. (Intermediate Cause 3)

Supporting Evidence: Although shown as a Contributing Factor in this leg of ECFT-3.2.2.1.1.1.1, this
was an Intermediate Cause in ECFT-1.1.1.1. The Intermediate Cause 3 label is carried here to avoid
duplication and confusion. After EVA 22, some of the team perceived that lengthy investigations were
unnecessary and could impede preparations for EVA 23, which was scheduled for the following week.
Audio logs and interview transcripts revealed that the ground team did not perform a thorough real-time
risk assessment; instead, they accepted the crews determination of cause for the EVA 22 water leak.

Crew and ground team training did not prepare the team with an adequate understanding of how the EMU
could fail with respect to water in the helmet. Had this been done, the crew and ground team may not
have accepted the explanation that the water was from the drink bag.

See ECFT-1.1 branch.

ECFT-3.2.2.1.1.1.1.1 ISS Community Perception was that drink bags leak. (Root Cause 2)

Supporting Evidence: Although shown as a Contributing Factor in this leg of ECFT-3.2.2.1.1.1.1.1, this
was a Root Cause in ECFT-1.1.1.1.1. The Root Cause 2 label is carried here to avoid duplication and
confusion. After EVA 22 ended, the crew concluded that the water which collected in EV2s helmet had
inadvertently leaked from his drink bag. Based on post HVCC interviews, it was determined that one of
the reasons that ground team members did not challenge the crews conclusion was because there was a
perception that drink bags leaked.

See ECFT-1.1 branch.

ECFT3.2.2.1.1.1.2 Ground Team allowed time pressures of impending EVA to influence actions.
(Intermediate Cause 4)

Supporting Evidence: Although shown as a Contributing Factor in this leg of ECFT-3.2.2.1.1.1.2, this
was an Intermediate Cause in ECFT-1.1.1.2. The Intermediate Cause 4 label is carried here to avoid
duplication and confusion. According to post mishap interview transcripts, more than one team member
indicated that they wished they had called a time-out. However, EVA 23 was scheduled for the
following week, which left little time to prepare.

There was also a perception that if the question concerning the source not being the drink bag was raised,
it would invoke a fairly resource intensive and potentially cumbersome process involving Engineering
and Safety for what most felt would likely turn out to be a non-issue. This would have an impact on EVA
23 preparations. In hindsight, however, it is now apparent that EVA 23 should not have commenced until
the EVA 22 issue had undergone a more adequate evaluation. That is not to say that a lengthy formal risk
assessment was required (that may, or may not be the case), just that the EVA 22 water leak deserved a
more refined assessment of risk. Had that been done, the EVA 23 HVCC might not have occurred.

See ECFT-1.1 branch.

110
 National Aeronautics and Space Administration EVA23Water Intrusion

ECFT3.2.2.1.1.1.2.1 Program emphasis was to maximize crew time on orbit for utilization. (Root
Cause 1)

Supporting Evidence: See ECFT-1.1 branch.

ECFT-3.2.2.1.1.1.3 Flight Control Teams perception of the anomaly report process as being
resource intensive made them reluctant to invoke it. (Root Cause 3)

Supporting Evidence: See ECFT-1.1 branch.

Ground Team focused on the drink bag

as the source of the w ater.

ECFT-3.2.2

The Team had Channelized Attention Team's set of responses led to a delay Ground team did not identify PLSS as Cognitive Task Oversaturation
on the drink bag as the primary source in identifying the vent loop as the actual source of w ater. contributed to the teams delayed
of w ater in the helmet source of the w ater leak. identification of the actual source of
ECFT-3.2.2.1 ECFT-3.2.2.2 ECFT-3.2.2.3 ECFT-3.2.2.4

Figure 3-65 Contributing Factors under ECFT 3.2.2

ECFT-3.2.2.2 Team's set of responses led to a delay in identifying the vent loop as the source of
the water leak. (Intermediate Cause 13)

Supporting Evidence: See ECFT-3.1.1.2

111
 National Aeronautics and Space Administration EVA23Water Intrusion

Ground team did not identify PLSS as

actual source of w ater.

ECFT-3.2.2.3

Critical Information w as not

communicated betw een Crew and
Ground Team
ECFT-3.2.2.3.1

EV1 initially focused on drink bag as

source in EVA 23.

ECFT-3.2.2.3.1.1

Crew incorrectly attributed w ater in Crew w as not trained to recognize

EV2's helmet after EVA 22 to the drink this failure mode.
bag..
ECFT-3.2.2.3.1.1.1 ECFT-3.2.2.3.1.1.2

Figure 3-66 Contributing Factors under ECFT 3.2.2.3

ECFT-3.2.2.3 Ground team did not identify PLSS as actual source of water. (Contributing Factor
4)

Supporting Evidence: Based on the review of the actual documents, it was determined that the
FMEA/CILs ineffectively addressed issues dealing with water in the helmet, therefore that information
was not incorporated into ISS/EVA training. This is discussed in more detail in ECFT-3.1.2.1.1.1.

ECFT-3.2.2.3.1 Critical Information was not communicated between Crew and Ground Team.
(Intermediate Cause 9)

Supporting Evidence: See ECFT-3.1.3

ECFT-3.2.2.3.1.1 EV1 initially focused on drink bag as source in EVA 23. (Intermediate Cause
14)

Supporting Evidence: Although shown as a Contributing Factor in this leg of ECFT-3.2.2.3.1.1, this was
an Intermediate Cause in ECFT-3.1.3.1. The Intermediate Cause 14 label is carried here to avoid
duplication and confusion. At the end of EVA 22 the crew and ground team had concluded that the water
which had leaked into EV2s helmet had come from his drink bag. This, as well as the perception that
drink bags on-orbit have had a history of leaking primed EV1 with the perception that EV2s drink bag
might have been leaking again during EVA 23.

112
 National Aeronautics and Space Administration EVA23Water Intrusion

See ECFT-3.1.3.1

ECFT-3.2.2.3.1.1.1 Crew incorrectly attributed water in EV2's helmet after EVA 22 to the drink
bag. (Intermediate Cause 1)

Supporting Evidence: See ECFT-1.1

ECFT-3.2.2.3.1.1.2 Crew was not trained to recognize this failure mode. (Intermediate Cause 7)

Supporting Evidence: See ECFT-3.1.2

Ground Team focused on the drink bag

as the source of the w ater.

ECFT-3.2.2

The Team had Channelized Attention Team's set of responses led to a delay Ground team did not identify PLSS as Cognitive Task Oversaturation
on the drink bag as the primary source in identifying the vent loop as the actual source of w ater. contributed to the teams delayed
of w ater in the helmet source of the w ater leak. identification of the actual source of
ECFT-3.2.2.1 ECFT-3.2.2.2 ECFT-3.2.2.3 ECFT-3.2.2.4

Figure 3-67. Contributing Factors under ECFT 3.2.2 for reference

ECFT-3.2.2.4 Cognitive Task Oversaturation contributed to the teams delayed identification of the
actual source of the water leak. (Contributing Factor 5)

Supporting Evidence: Based on MIB previous experience, emergencies which are not covered by
training have the potential to open a plethora of technical resources and information that must be
deciphered in a very short amount of time. The information that the individual must process may exceed
their cognitive resources, given a limited amount of time. In reviewing transcripts and interviews from
several team members, it appears that some individuals were cognitively task oversaturated by the events
of EVA 23.

In addition to processing technical information, the team members also had to process verbal information,
which in itself can lead to cognitive oversaturation if an individual is simultaneously trying to listen and
respond to multiple conversations. From interviews, the MIB learned that due to the multiple
communications going on at the same time, some team members didnt have sufficient time to work
through the suspicion that the drink bag was not the source of water.

There were many tasks required of controllers at each position within mission control. They were
mentally engaged in EVA operations, dealing with off-nominal events, communicating on the comm
loops, accessing flight rules, referencing flight notes, and, at times, navigating a host of other
informational databases. Multi-tasking of this nature can lead to cognitive task over-saturation see Human
Factors section (Appendix A).

113
 National Aeronautics and Space Administration EVA23Water Intrusion

ECFT-3.2.3 Critical Information was not communicated between Crew and Ground Team.
(Intermediate Cause 9)

Supporting Evidence: See ECFT-3.1.3 branch.

Flight Control Team sent EV2 back to

Airlock unaccompanied during EVA
termination.
ECFT-4

Flight Control Team did not recognize

the severity of the situation.

ECFT-4.1

EV2 did not communicate severity of

his condition after terminate call.

ECFT-4.1.1

EV2's comm system failed due to

w ater

ECFT-4.1.1.1

Figure 3-68. Contributing Factors under ECFT 4 for reference

ECFT-4.1 Flight Control Team did not understand the severity of the situation. (Intermediate
Cause 6)

Supporting Evidence: Although shown as a Contributing Factor in this leg of ECFT-4.1, this was an
Intermediate Cause in ECFT-3.2. The Intermediate Cause 6 label is carried here to avoid duplication and
confusion. Audio logs and interviews indicate that the ground team focused primarily on the drink bag as
the source of the water leak. Because the drink bag was known to hold about one liter of water, it would
be understandable for the team to think that the leak would be self-limiting to a liter, or less. It was
reasonable to assume that the leak would be less if EV2 were to consume whatever water remained in the
drink bag, after the leak was made known, as he eventually did. Since the team focused on the drink bag,
and water leaking from the drink bag could be managed, the true severity of the condition went

114
 National Aeronautics and Space Administration EVA23Water Intrusion

unrecognized, that being the possibility of aspiration of much more water (over four liters) leaking from
the PLSS.

See ECFT-3.2 for additional information

ECFT-4.1.1 EV2 was unable to communicate the severity of his condition after the terminate call.
(Contributing Factor 3)

Supporting Evidence: The MIB learned from speaking to EV2 that he was continuing to make status
calls during his translation back to the Airlock but this was not heard over the comm loops. He started to
have issues with visibility shortly after the decision to terminate the EVA was made.

ECFT-4.1.1.1 EV2s comm system failed due to water. (Contributing Factor 6)

Supporting Evidence: The MIB was told by EV2 that he made calls during his translation to the Airlock
but apparently the water in his helmet was affecting his communications system because some calls were
not heard. His comm was, at best, intermittent because his call reporting that he was back at the Airlock
was heard by the Flight Control Team on the ground.

115
 National Aeronautics and Space Administration EVA23Water Intrusion

3.7 Observations

O-1 Based on what they knew at the time, the Ground Team performed admirably.
Supporting Evidence: From the time EV2 reported over S/G loop water on the back of my head until
the Flight Director called for a Termination of the EVA was 23 minutes. In that time, a great deal of
discussion went on between EVA front and back room controllers, CORE and Windsor Locks engineers
and Surgeon. Water had been seen in EMU helmets before but this specific failure hadnt been seen
before and resulted in much more water than had ever been experienced. The team quickly worked
through the known possible failures and came to the conclusion that the EVA needed to be terminated.
The teams quick actions may have saved the crewmembers life.

O-2 Helmet Anti-fog material is problematic.

Supporting Evidence: Liquid soap is used as an Anti-Fogging agent on the inside of the EMU helmets
visor before crews go EVA. The MIB learned through review of failure reports that there are at least 5
prior incidents of reported eye irritation during EVAs. Many times this irritation is caused by the anti-
fog material (soap and glycerin mixture) flaking off and floating into the eyes or a drop of water (sweat or
drink bag water) hits the visor and picks up some of the anti-fog then floats into the crewmembers eye(s).
This irritation has been reported by astronauts as extremely painful resulting in temporary impaired vision
and work stoppage until the irritation can be washed out through tearing and blinking. This could lead to
potential hazards while in an EVA environment.

Recommendation R-8: The ISS Program should investigate alternate materials that effectively
perform the helmet anti-fogging function without the risk of eye irritation.
O-3 The EMU CO2 sensor has a history of failing during EVA due to excess moisture in the
event loop.
Supporting Evidence: Through interviews and a review of EMU failure histories, the MIB has
determined that the CO2 sensor is sensitive to moisture in the vent loop, which has occurred multiple
times in the past. It happens often enough that the Flight Control Team has standard procedures to deal
with it and it is not considered a reason to terminate an EVA. In this mishap, the CO2 sensor failed off-
scale high which was likely due to the moisture that the failed EMU was dumping into the vent loop. If
the CO2 sensor was not subject to frequent failure, this might have given the Flight Control Team an
earlier warning that an abnormal condition was being experienced. In addition, if the function of the CO2
sensor is considered important for physiological monitoring, it should be reliable for performing that
function.

Recommendation R-9: The ISS Program should investigate alternate CO2 sensor designs that
eliminate the sensitivity to moisture.
O-4 EVA POCC oversight of EVA flight control team may be detrimental.
Supporting Evidence: From interviews, discussions, and MCC audio, it is the perception of the MIB that
the interaction of the POCC with the Flight Control Team may be detrimental to the functioning of the
Flight Control Team by leaving the impression that they are second guessing the actions of the team.
There is clearly some benefit to having experienced EVA personnel, who are not part of the flight control
team, following along with on-orbit activities. It offers the opportunity for them to help train less

116
 National Aeronautics and Space Administration EVA23Water Intrusion

experienced flight controllers and also offers a second set of experienced eyes watching a very complex
activity. However, when issues arise during the EVA, their involvement in flight loop conversations can
add confusion.

Recommendation R-10: MOD should evaluate how personnel who are located in the POCC
facility and not part of the active flight control team interact with the active flight control team
and ensure that lines of communication and the decision making chain is not compromised.
O-5 Use of the Helmet Purge Valve (HPV) was questioned by some flight controllers after the
terminate call was made
Supporting Evidence: From interviews and audio recordings, the MIB learned that some individuals on
the flight control team discussed the use of the HPV to remove free water from the helmet of the crew
member. However, there was concern that the HPV could freeze if water was passed through it and
possibly damage the valve. This has evidently been a concern for some time. The HPV is referenced
frequently in various malfunction procedures any of which could have led to water being intentionally or
unintentionally evacuated through it from sweat or the drink bag, etc. If there were doubts about the
efficacy of using the HPV at any time in the past, these should have been investigated and put to rest at
that time.

Recommendation R-11: The ISS Program should perform testing and analysis to verify that use
of the Helmet Purge Valve to remove free water from the helmet is safe and effective. Results of
this testing should be made clear to the EVA community, including the flight control team and
documented in hazard reports, flight rules and procedures.
O-6 The process by which on-orbit EMU non-conformances are initiated and ultimately closed
is inconsistent with best practices and seems to be implemented inconsistently.
Supporting Evidence: During this investigation, from discussions and interviews with Engineering,
S&MA, and EVA Office personnel as well as reviewing the documentation governing the process by
which non-conformances are documented and the PRACA GFE database itself, the MIB found that the
process by which on-orbit EMU non-conformances are initiated and ultimately closed is inconsistent with
best practices and seems to be implemented inconsistently.

For in-flight anomalies JSC 28035, Program Problem Reporting and Corrective Action Requirements for
Johnson Space Center Government Furnished Equipment allows the EVA Office to determine when to
open a FIAR on the GFE controlled by that office. This is in contrast to the system used by the rest of ISS
GFE where it is up to the ISS MER S&MA Console to determine if a FIAR is to be initiated. Although
the Board could find no evidence of this being a problem, this appears to the Board to be a conflict of
interest.

In review of a relatively small subset of EMU non-conformances, it was fairly easy to find inconsistencies
in how the non-conformance reporting criteria were applied over time including reports that were issued
under the criteria that were excepted from reporting. Also it was noted that assignment of related FMEA
failure mode was not consistent or precise. By that it is meant that similar non-conformances had either
no FMEA assigned to one of several occurrences or were incorrectly assigned. The latter may be due to
the preliminary assignment of a presumed failure mode being witnessed and later after investigation not
being updated. In any case it prevents the ability to correctly monitor trends in failures.

117
 National Aeronautics and Space Administration EVA23Water Intrusion

Recommendation R-12: ISS Program and Safety and Mission Assurance should review and
update the process as defined in JSC 28035 to resolve the conflict of interest of the EVA Office in
initiating FIARs.
Recommendation R-13: Safety and Mission Assurance with the assistance of the EVA Office
should initiate a review of all non-conformances contained in the PRACA database for the EMU
and review the assignment of FMEA associated with each one and update as required.
O-7 Achievement of 6-year on-orbit certification of the EMU appears inadequate.
Supporting Evidence: During the Mishap Investigation, the MIB reviewed the documentation related to
the 6-year certification of the EMU. The MIB also conducted interviews with various technical personnel
about the process and results of the 6-year certification effort. While an exhaustive review of the
certification plan by the MIB could not be performed, even a high level review identified that there are
likely weaknesses in the certification that could lead to premature failures of the EMU. In addition, there
are known components that could not be certified for 6-years; however this fact was not widely known at
the higher levels of management where risk decisions are made. For example, the Item 134 check
valve/filter assembly was readily admitted not to be certified for 6 years by UTAS WL and EC SSM,
however, no on-orbit procedure to check and clean/replace this assembly existed prior to this Mishap.

Looking back on the suits water related issues from 2004 through today, a number of experts from both
within the MIB and from outside organizations raised the possibility of shortcomings in the routine care
available onboard ISS, on the ground and the ground tests and analysis used for life extension. While the
life extension tests relied on the best available information at the time for water quality and flight duration
parameters and aimed to minimize the need for onboard crew intervention, subsequent real ISS flight
experience indicates that unanticipated variable water conditions exist. Reliance on ground-based fleet
leader hardware is similarly suspect because the conditions of the on-orbit environment, particularly water
quality, are not well understood. As confirmation of the suspect reliance on life extension practices, Table
3-2 shows comparison of hoped for life versus actual life shows that at least 50% of current onboard flight
suits are failing to meet the hoped for goals of total years and total sorties.

Table 3-2 Status of EMUs currently on ISS

ISS Arrival PLSS # Current Status (Aug 2013) Generic Life Goal
Date

Passed on-orbit internal use test after ~ 4 years

Nov 2009 3010
on-orbit and 8 uses

Passed on-orbit internal use test after ~ 3.5

6 years on-orbit
Feb 2010 3005 years on-orbit and 7 uses, but during increment
24 in Aug 2010 had golf ball of water in 25 uses
helmet and failed CO2 sensor.

May 2010 3011 Failed (since July 2013) due to water in helmet
after ~ 3 years on-orbit and 6 uses
July 2011 3015 Failed (since Aug 2012) due to sublimator loss
after ~ 1 year on-orbit and 1 use

118
 National Aeronautics and Space Administration EVA23Water Intrusion

Recommendation R-14: The ISS Program should commission an independent technical review of
the EMU 6-year certification plan which should identify all deficiencies or weaknesses in the
certification and re-establish the true life expectancy of the EMU, and then plan appropriate use
and logistic strategies commensurate with the results of the review.
O-8 Differences noted between EMU plant and field procedures indicate a lack of two-way
feedback and procedure control.
Supporting Evidence: During the timeframe of the MIB investigation, it was found that the UTAS plant
and field procedures differ on whether or not to open Test Port B during certain tests of the water
separator loop. This was revealed during the PLSS 1017 flooding investigation that occurred in August
2013. In addition, it was discovered by the MIB during discussions that some amount of contamination is
considered acceptable by the field technicians but no process is in place to verify that the contamination
found is understood and a nominal condition. Given the changing water environment the EMU is
subjected to, this behavior could allow new issues to remain undiscovered.

Recommendation R-15: The EVA Office should ensure that all EMU procedures are consistent
between all teams that perform operations with the EMU, and require that all contamination
found during ground processing be evaluated by the Engineering and Quality teams.
O-9 The pace and potential hazards associated with EVAs on ISS are similar to other activities
that should receive similar scrutiny by the ISS Program.
Supporting Evidence: The majority of the activities performed while operating the ISS do not have
immediate hazards associated with them. EVAs are different than most, however, in that they require
extreme focus from the ground teams, have hazards associated with them that can quickly lead to injury
or death, and require decisions to be made quickly and efficiently. This report has focused on issues
related to performing EVAs, however, the MIB was asked to identify activities on board ISS that have
similar characteristics to EVA (immediate hazards, the need for quick decision making, etc.) and
determine which recommendations should be applied to these other activities.

To determine other areas which share characteristics with EVAs, per the direction of the Appointing
Official, the MIB interviewed several experienced ISS Flight Directors, and used the experience of the
MIB members. Several areas were identified:

Visiting Vehicle operations

On board emergency operations
Software transitions
Reaction to serious hardware system failures
Visiting Vehicles: Visiting vehicles subject the ISS to potential immediate hazards when they are
operating in the vicinity of ISS. Failures of their systems could lead to a collision risk with ISS. They also
present a unique challenge to the flight control team because the vehicles are operated independently by
an external flight control team, with various levels of experience. A significant focus has been placed on
commercial cargo operations to ISS. The Engineering and Flight Control teams have worked very closely
with both commercial companies to verify the safe and efficient operations of those vehicles. The MIB
does not have specific concerns with commercial cargo activities today, but care must be taken by the ISS

119
 National Aeronautics and Space Administration EVA23Water Intrusion

Program to prevent complacency as the success of these missions continues without major failures. Of
greater concern, as expressed during interviews, is the operation of Russian vehicles in proximity of ISS.
Interviewees mentioned several incidents in the past few years that made them call into question the
safety of Russian ground control team philosophy. One incident that was mentioned related to an un-
deployed antenna on a Progress cargo vehicle which presented a potential risk to the docking interface.
Despite protests from the US ground team and management, the Russians proceeded with docking. While
docking was successful, damage was done to the Progress vehicle and the risk to ISS was taken
unilaterally by an International Partner. Having an established chain of command is critical for safe ISS
operations. In addition, its clear that when commercial entities begin providing crew launch services, the
level of cooperation between NASA and the commercial companies will need to be very high.

On Board Emergency Operations: If an on board emergency, such as a rapid depressurization, fire, or

release of ammonia occurs, the response of the flight control team and onboard crew will be critical to
keep the crew safe. The response must be very rapid, practiced, and proceed with little confusion.
Interviewees indicated that a significant effort has been underway over the last two years to update and
evaluate all on board emergency procedures. Significant on board training occurs, with participation from
the ground control teams. One of the concerns expressed during interviews is the challenge of working
with the Russian partners and getting them to agree to updates in the emergency procedures. Due to the
recent updates and attention paid to emergency procedures, the MIB has no concerns, but believes that
care must be taken to remain vigilant as the Program continues.

Software Transitions: Due to the complex nature of the ISS operating systems, major software upgrades
occur periodically. When these upgrades occur, major ISS control systems are at risk of malfunctioning
due to software errors or system configuration errors. Recently during a major software transition, all
communication was lost with the vehicle for several hours. In these times, the ISS vehicle is at risk and
procedures to recover the vehicle must be clear, effective, and proceed quickly.

Reaction to serious hardware system failures: Due to the complexity of the ISS, certain failures place
the ISS in a configuration that can quickly lead to a hazardous condition or loss of the ISS vehicle. For
example, several years ago the ISS lost one of two operating Pump Modules (similar to the situation
occurring at the writing of this report) that provided cooling for the major ISS pressurized element
systems. Loss of the second unit would have quickly led to the loss of ISS. While the failed Pump Module
was replaced with a spare, the EVA necessary to replace it took some time to plan and execute. The
failure also made the ISS Program more aware of the hazards of various failures of ISS Systems.

These four identified areas represent situations that are different than the majority of ISS operations. They
require fast, well-coordinated and practiced responses to prevent a hazard. Several recommendations
developed based on the MIB investigation into the EMU failure are applicable to these areas. Specifically:

Recommendation R-16: MOD should ensure that simulations of specific, fast-paced failure
scenarios (visiting vehicles, on-board emergency response, software transition issues, and serious
system hardware failures) should include all phases of the teams response to ensure that the
response can be fully performed from end-to-end in a quick, proficient manner.
Recommendation R-17: The ISS Program should ensure that FMEA/CILs related to fast-paced
failure scenarios (visiting vehicles, on-board emergency response, software transition issues, and

120
 National Aeronautics and Space Administration EVA23Water Intrusion

serious system hardware failures) are regularly updated, studied, and used in training for flight
controllers as well as engineering and safety personnel.
Recommendation R-18: As the success of the ISS Program continues, the ISS Program must
institute requirements and behaviors that combat the tendency towards complacency by requiring
regular training by all teams in the safety critical aspects of failures related to fast-paced scenarios
(visiting vehicles, on-board emergency response, software transition issues, and serious system
hardware failures).
Recommendation R-19: The ISS Program must ensure that full root cause determination of
failures related to specific, fast-paced failure scenarios (visiting vehicles, on-board emergency
response, software transition issues, and serious system hardware failures) must be pursued and
verified by ISS Program managers and the Engineering and Safety Technical Authorities.
O-10 Water quality and chemistry in multiple onboard ISS systems is constantly changing and
many critical ISS Systems are sensitive to these changes, yet a systematic method to regularly
monitor and maintain the systems that affect the EMU and other water systems does not exist.
Supporting Evidence: Through interviews and review of past ISS problem reports, it is clear that many
systems onboard ISS have been affected by changes in water quality. For example, in recent years, the
presence of a chemical called dimethylsilanediol (DMSD) was raising the Total Organic Compound
levels in the potable water system, potentially causing or masking a crew health risk. In addition, in
August 2012, EMU 3015 failed due to cooling degradation. That suit remains onboard, not accessible for
full diagnosis. Evidence from returned water samples shows that its water is contaminated enough (at
very low concentration levels not readily traceable) to fail the sublimator porous plate. Over time, the
plumbing of the life support system is influenced by the particulates and microbes within the ISS
atmosphere and from all of the materials and accessories within the pressure garment.

For the post-Columbia onboard ISS suits, Table 3-3 indicates that with or without the latest mitigations,
the suits only remain operational on-orbit for 1-3 years (e.g. when filters of known contaminants no
longer suffice for new unknowns). It is noteworthy that 2 of the suits from the post-Columbia ISS era
were cleaned and returned to service and are now again onboard ISS.

Table 3-3 EMU failures on ISS

ISS Arrival PLSS # Current Status (Aug 2013) Generic Life Goal
Date

Failed (May 2003) due to bound water pump after ~ 1

June 2002 3013 year on-orbit and 5 uses (HEMU123-
012)(BEMU000F01)

Failed (May 2004) due to bound water pump after ~ 1.5 1-2 years on-orbit
Nov 2002 3005
years on-orbit and 0 uses 25 uses
Failed (July 2005) due to bound water pump found post
June 2002 3011 STS-114 return after 3 years on-orbit and 5 uses
(BEMU000F03)(BEMU100A035)

121
 National Aeronautics and Space Administration EVA23Water Intrusion

The SPCU Heat Exchanger which directly interfaces with the PLSS water systems has known sources of
contamination that is controlled by periodically scrubbing the system during ALCLR activities. However,
when the heat exchanger was replaced in early 2013, rather than return the hardware to the ground for
evaluation of possible contaminants, the heat exchanger was disposed of aboard a Progress vehicle.

Recommendation R-20: The ISS Program should institute a systematic process of monitoring
water quality and chemistry aboard ISS to track changes that can affect critical ISS systems
including the EMU, crew health, and multiple ISS Systems that use water and are sensitive to its
chemical makeup (The Oxygen Generation System, The Water Processor Assembly, the
Common Cabin Air Assembly, etc.). This process should include consideration of onboard
monitoring capability. It should also include return of any removed hardware to the ground for
evaluation.
O-11 During EVAs, EMU suit telemetry that provides information about certain critical
operational parameters (fan speed, battery current, water temperatures, etc.) to the ground
teams is limited to one data point every 2 minutes.
Supporting Evidence: Per review of current EMU design specification, the current design does not allow
the ability to monitor EMU performance parameters during an EVA at a high rate thus giving ground
teams the ability to monitor system performance and proactively detect problems that arise real time.
Since EVA represents a hazardous operation utilizing a system that does not contain normal ISS
redundancy, this design could be improved to prevent future mishaps.

Recommendation R-21: The ISS Program should develop a system that allows high rate data
telemetry to be received by ground teams during an EVA to allow near instantaneous monitoring
of critical system parameters.
O-12 The ISS program is developing the capability to launch and return only one full short EMU
at a time.
Supporting Evidence: During the course of this Mishap Investigation, both the MIB and the ISS
Investigation Team have identified the need to launch and return multiple components of different EMUs
to support investigation activities. Currently the ISS program is developing hardware that will allow the
launch and return of a short EMU, but has no plans to develop a system where only failed components,
such as a PLSS (or multiple PLSSs), could be launched and returned concurrently. This significantly
complicates the logistics of flying up and down EMUs and their components.

Recommendation R-22: The ISS Program should develop a flexible system that allows multiple
short EMUs, as well as EMU components such as the PLSS, to be launched or returned on
multiple vehicles.
O-13 Flight control and Engineering teams have a general inability to access flight hardware to
perform training of personnel (including flight crew) as well as validating procedures or
performing other engineering tests.
Supporting Evidence: A related recurrent theme in our discussions and interviews with EVA
community personnel is the inability to access flight hardware to perform training of personnel including
flight crew as well as validating procedures or performing other engineering tests. In many cases
personnel are reliant on pictures or mockups that may or may not represent the actual flight configuration
of hardware and crews do not have a chance to see hardware they may have to maintain on-orbit.

122
 National Aeronautics and Space Administration EVA23Water Intrusion

This puts personnel with incomplete training in positions where they are asked to make decisions based
on incomplete knowledge and incomplete procedure validation. Personnel have been turned down so
many times when they have requested access that they self-censor their requests and therefore do not ask.
(This also was a factor in personnel proceeding with the on-orbit vent-loop dry out procedure when it had
not been validated on an actual flight unit.) Rationale provided by the supporting organization is that it
would interrupt the processing, negatively impact the processing schedule, and since the hardware is so
limited, they cannot risk the potential that the hardware might be broken. This is equivalent to the
message that it is acceptable to accept the risk to hardware and personnel on-orbit over the risk to
hardware on the ground. A test like you fly philosophy should be the default position by the technical
teams.

This is not to say there needs to be unfettered access to flight hardware, but access by various members of
the EVA community, particularly operations (including crew) and safety who do not routinely have
access to flight hardware is necessary to ensure high quality training. Historically, the broader EVA
community including the Astronaut Office, MOD EVA, and S&MA personnel were given routine access
to flight hardware at events such as chamber runs and equipment bench reviews. Close out photos were
taken of all hardware to document the precise configuration of what is flying to facilitate effective
communication between the ground and crew in flight. All procedures were validated on flight hardware
if the procedure required a functioning system versus a fit check. From interviews and discussions, it has
been found that by and large these opportunities have been eliminated from the program. The reasons
cited are mainly due to budget and schedule impacts.

On a related issue, the idea that one instance of EVA hardware is identical to all others is also
fundamentally flawed. This was illustrated in the planned on-orbit changeout of the water line vent tube
assembly in the HUT of EMU 3010. The Engineering and Operations personnel supporting the changeout
were given the understanding that the fiberglass panel that covers the interface was velcroed in place. It
turned out that some units have the panel bonded in place. This wasted crew time and if nothing else was
embarrassing. Close out photos of each piece of hardware flying is vitally important. Accurate
configuration records are also necessary to be maintained and made available to all support personnel
with a real-time need. The Crew timeline was adversely impacted due to a misunderstanding of the on-
orbit configuration of the EMU hardware due to the fact that there were no closeout photos of the
hardware.

Recommendation R-23: The ISS Program and the EVA Project Office should put schedules and
processes in place to ensure access to flight hardware to the broader EVA community including
the Astronaut Office, MOD EVA, and S&MA personnel.
Recommendation R-24: The ISS Program and the EVA Project Office should require close out
photos be taken of all hardware with the participation of operations personnel to document the
precise configuration of what is flying as well as accurate configuration records maintained and
made available to real-time support personnel to facilitate effective communication between the
ground and crew in flight.
Recommendation R-25: The ISS Program and the EVA Project Office should ensure that all
procedures are validated on flight hardware if the procedure requires a functioning system versus
a fit check.

123
 National Aeronautics and Space Administration EVA23Water Intrusion

O-14 There is no backup EVA capability in the event the need for a contingency EVA arises.
Supporting Evidence: The event on EVA 23 created a crisis within the ISS Program as it meant that
there was no US contingency EVA capability in the event that a critical EVA had to be performed to
ensure the safety of the crew or vehicle. Flight Rule B1-3 C. dictates that when a function normally
provided by one partners segment is unavailable, the function shall be performed by another partners
segment when possible. Given the inherently uncertain environmental and usage conditions faced by this
legacy system and the risks of relying upon it solely and indefinitely, one or more dissimilar backup
options should exist as with other essential vehicle systems.

In the past such a capability existed. Russian EVA training suits (known as Orlan) were available and
used in the JSC NBL and cross training of basic US task skills was performed by crew in Orlan suits. The
Joint Airlock was designed with the intent that either suit could be used in that airlock. The Joint Airlock
is in service on the ISS today, however, its use by Orlans has never been validated.

Recommendation R-26: For critical external tasks, the ISS Program should provide at least one
viable and proven dissimilar backup EVA capability (known candidates include dexterous
robotics or Russian EVA)
O-15 Lessons learned databases and corporate knowledge information exist, but are not always
easily accessible, often incomplete and are not being fully utilized.
Supporting Evidence: From interviews and review of existing EVA historical databases, the MIB
learned that the loss of suit expertise that started in the early to mid-1990s continues and is uniform across
multiple EVA organizations. Some of the loss is traceable to routine personnel retirements, but other
losses were driven by decisions that declared the suit design to be sufficiently mature for ISS purposes.
Upgrades and improvements have been relatively few compared to the efforts of the 1980s and early 90s.
The depth and breadth of the work force has diminished as development and significant improvements
gave way to sustaining engineering for obsolescence and failure support. Space Shuttle retirement, ISS
completion and cancellation of the Constellation program further eroded EVA support since a number of
key personnel were sustained by directly or indirectly supporting multiple programs. Just within MODs
EVA ranks, civil servant and contractor head counts dropped from 54 to 38 after the completion of the
Shuttle program and ISS assembly. Those remaining are stretched thin to cover routine training and
multiple mission control shifts with little margin for contingency affairs without burnout. Such attrition
over the years has depleted those that remember or have direct experience with this suits legacy, its hard
earned lessons, inherent limitations or subtle messages suggesting renewed attention. These adverse labor
conditions are not unique to EVA and exist across other areas of human spaceflight.

Attempts to counteract this loss of expertise via knowledge capture and lessons learned exist, but are
limited by lack of resources and time. It is admirable that EVA mission control skills are bolstered by
innovative tactics such as staff mentoring and participation in non-EVA handovers (for currency with the
latest flight control processes and personnel) and on-the-job training (OJT) roles during this eras more
limited spacewalks. Unfortunately, departure of contractor and civil servant experts occurs faster than
information is collected and passed on. Those that remain have less time to explore history in meaningful
ways because their labor is almost fully dedicated with preparation for present and future EVAs.
Compared to the Shuttle and ISS assembly eras, hands-on hardware experience opportunities are much
reduced with fewer crewmembers flying, fewer ground training/test events and even fewer on-orbit
sorties. There is a trend toward very few vacuum chamber runs (8 ft., 11 ft., ETA, SSATA), no thermal

124
 National Aeronautics and Space Administration EVA23Water Intrusion

vacuum runs and bench reviews limited to flight hardware isolated in bubble wrap. Chamber runs are the
best available test and training grounds for flight hardware performance when on-orbit experience
opportunities are few. Attempts to compile failure and performance history summaries from cumbersome
official databases have low priority and seem to have dragged on for months/years or do not reach key
users. The EVA project offices informal and incomplete spread sheet of suit failure history has not been
finished, sustained and made broadly accessible. JSC SMAs poster of EVA incidents is missing events
and is not required reading. Alternatively, the significant EMU failure history (ground and flight)
documentation that used to exist in the vendors mini-data book was deleted. An existing means of
disseminating failure history is with the EMU Requirement Evolution book initially created in 1994, but
it has not been revised since 2005. The EMU Assured Availability report of 1991, the Red Book
performance of each suit, and the EVA checklist rationale book are all excellent resources with past EMU
history. Database software compatibility issues, user friendliness issues (e.g. OCAD is problematic for
non-EMU systems of ISS) and admin right restrictions often preclude the use of application specific
software. Until access is improved, key product masters that live only on contractor or NASA websites
(e.g. Centric, FOX), are difficult to access and less likely to be used. Unfortunately, knowledge capture
efforts sponsored by the EVA project office have been halted due to funding shortfalls and available
labor/time.

The EMU Requirements Evolution Document was originally developed in 1994 as a design knowledge
capture document towards the end of the EMUs initial development when it was recognized that there
was an impending loss of development history with future retirements. It contains information about the
rationale for the design assumptions and decisions that were made in the development of the EMU.
However, it was never completed and had several sections that were left empty for unknown reasons. The
latest edition was issued in 2005, but instead of populating the empty sections, they were deleted and
those that had data in them were updated. In MOD this book is read once during the EVA Systems Full
Instructor flow. It is not evident that Engineering or Safety refers to this book in their curricula. As
individuals progress in experience throughout their career in EVA, it would be good to periodically use
the EMU Requirements Evolution Document to refresh understanding of the assumptions under which the
EMU was designed.

The EMU Mini-data Book is a thorough compendium of drawings and specifications at the component
level of the EMU. It has had widespread use over the years and is one reference that is consistently used
by all members of the EVA community. It has been updated periodically, the last time being 2010 (Rev.
P). One shortcoming of the current edition is the elimination of failure history. As with the FMEA/CIL
this used to be a great location to find the failure history of each component without having to have access
to the formal failure databases.

There are several sources for EVA Lessons Learned. Two are managed by the EVA Project Office and
one by MOD EVA. There is overlap amongst them. The most comprehensive is the EVA CCB Lessons
Learned Database that is organized by flight/increment and category. This is a collection of issues that
arose during different EVAs during the ISS era and come from several sources including MOD post
flight/increment presentations. The descriptions are very brief without context or background and many
have actions that are not clear have been closed. High level forward actions are included. It significantly
lacks information pertaining to EVA prior to ISS assembly.

125
 National Aeronautics and Space Administration EVA23Water Intrusion

MOD EVA maintains their MOD EVA Lessons Learned Archive which consists of a series of post-
flight/increment presentations dating back to STS-88 (beginning of ISS assembly). The presentations are
very specific to the EVA being discussed and do not have enough context to allow a reader years later to
grasp the significance of the issue being discussed. Although information on EVAs prior to STS-88 is
available, none of it shows up in the MOD Lessons Learned data and is therefore not readily passed
down. It would seem that for a legacy system such as the EMU, it would be important to characterize the
EMU in its early development period when so much of the thinking, processes, and procedures were
developed.

Currently, there is no summary level of lessons learned. Each lesson learned from each mission or
increment is documented separately with no attempt to categorize or otherwise organize and synthesize
the information into a useable form. Trends in lessons learned are not tracked, so one cannot readily see
that a particular lesson learned has not been learned but merely experienced over and over again.
Currently, trainees are required to read each increments lessons learned (37 folders with multiple
presentations each with several pages). It was noted that many contained actions to be worked and there
were similar lessons across several increments.

Recommendation R-27: With the help of MOD, the EVA Office should review all existing EVA
knowledge databases and combine them into a set of databases that are complete, accurate, kept
up-to-date, and easily accessible to the entire EVA community
Recommendation R-28: The ISS Program should ensure that the EMU Requirements Evolution
Book is routinely updated to capture the maturing design and design rationale of the EMU and
include material originally intended for the placeholder sections in the 1994 version.
Recommendation R-29: The ISS Program should ensure that the EVA community uses the EMU
Requirements Evolution Book and the improved knowledge capture databases, once developed,
to improve ground team training requirements throughout the EVA community for better depth of
EMU system knowledge and attention to design and failure history.

O-16 Failures often are only tracked to proximate cause with little meaningful trending being
performed or root causes being pursued.
Supporting Evidence: Through MIB investigation into existing failure documentation, the MIB has a
significant concern about the relatively inconsistent and sometimes weak attention given to true root
cause determination, control and failure trending. The definition of Root Cause is: One of multiple
factors (events, conditions that are organizational factors) that contributed to or created the proximate
cause and subsequent undesired outcome and, if eliminated or modified, would have prevented the
undesired outcome. The board identified several FIARs in which true Root Cause was not identified:
BEMU100A012; BEMU121A004; BEMU122F004; BEMU100A037.

As a specific example from BEMU121A004, vent switch failure, the root cause was identified as
susceptibility of shielded bearing design to contamination introduced during manufacturing. An
appropriate root cause might have been something like the lack of a company-wide process to protect
contamination-sensitive items, including bearings, during manufacturing. Corrective Action taken was to
use one set of bearings during particle-generating operations, and substitute another set afterwards. To
address this as a root cause, the corrective action would have been expanded to a company-wide

126
 National Aeronautics and Space Administration EVA23Water Intrusion

application. By not identifying failures to their root cause a greater risk is taken for a recurrence of the
issue. A related issue was found in other documentation.

As a case in point, no FIAR/PRACA or IRIS entry exists for the vacuum cleaner oxygen incident, with
the rationale being that the hardware performed nominally. This waive off does not resolve that the failed
validation process allowed a potentially hazardous close call. A part of this gap may be because the
Agency has no guidance or training on expectations for root cause analysis for engineering investigations
(NPRs 7123 and 7120 are silent). Current labor and time constraints tend to focus upon hardware specific
fault trees to the detriment of logic flow based troubleshooting and seeking process based root causes. If
the tendency to focus on symptoms continues, more fundamental issues will be missed. Ideally, root
cause pursuit would be institutionalized in standard program and project practices and not left to rare
mishap investigations.

Recommendation R-30: The Agency, Centers, and Programs should improve requirements for
root cause determination and subsequent training and provide the training for Engineering and
Safety personnel to better ensure root cause determination of critical and reoccurring failures.
O-17 The Flight Crew and all ground-based MCC and MER personnel involved in the event were
properly certified for their positions by their respective organizations.
Supporting Evidence: Training and certification records including certification requirements for all
personnel were reviewed by the MIB. In this activity there was no determination as to the validity of the
certification. This was merely an audit function to verify that all personnel working in positions requiring
certification were properly certified according to the rules at that time.

O-18 Integrated sims are intended to run their entire scheduled length, which causes the Flight
Control Team to never experience the pressure of terminating an EVA early before a majority
of the objectives are accomplished and may be providing negative training that all problems
can be overcome in the course of an EVA and therefore delays the decision to terminate.
Supporting Evidence: An integrated EVA simulation is a complex undertaking. Integrated sims are
considered the most realistic training possible for the Flight Control Team and therefore are of the highest
fidelity. The instructors work very hard to provide challenges to the Flight Control Team to fulfill training
objectives for all the members of the Flight Control Team. For an EVA sim, three facilities are used: the
MCC, the ISS simulator, and the Neutral Buoyancy Laboratory. As a result, integrated sims are also very
expensive to conduct. In order to maximize the training benefit, it is important that the Flight Control
Team remain unaware of the objectives of the simulation so as not to anticipate where the sim scenario is
going. It is important that the scenarios are constructed so that there is a high probability that all of the
training objectives will be met no matter what course the Flight Control Team takes along the way. In
short, if the scenario is intended to cause the team to call for a terminate or an abort EVA, there is a risk
that this will cause the sim to end early and therefore training objectives for other flight controllers may
not be accomplished for that sim. Therefore it is rare, if ever, the case that a sim scenario will cause the
EVA team to call for a terminate significantly prior to the scheduled end of a sim. Thus, the EVA team
never gets the experience of making a call with significant consequence prior to flight i.e. they are
inadvertently trained not to terminate an EVA early in an EVA because they are never asked to do that in
training. They are much more comfortable late in an EVA when the consequences are lower both in terms
of the potential impact to a sim and the impact to the real EVA when most tasks are accomplished. This
can lead to the EVA team, when put into a real situation, of doubting when to end the EVA in real-time

127
 National Aeronautics and Space Administration EVA23Water Intrusion

since theyve never had to make a call of that severity of consequence that will end in a terminate EVA.
Additionally, the Flight Control Team is inadvertently being taught to believe that any problem they may
encounter can be overcome therefore allowing them to continue with the EVA. This may lead to them
taking longer to accept the fact that terminating is the proper response.

On a related note, simulations routinely end prior to the crew fully ingressing the airlock and re-
pressurizing. This does not allow the flight control team to fully exercise those parts of the EVA timeline
that may require tailored steps as a result of a terminate EVA condition that leaves the EMU in an off
nominal configuration.

From the MCC loops it is evident that the flight control team had to tailor the steps of the EVA
Ingress/Repress cue card during the ingress and repress of EVA 23 as a result of the failure. This took a
lot of discussion with multiple sources weighing in.

From discussions with relevant personnel it was discovered that EVA sims routinely end prior to the crew
ingressing the airlock and therefore this part of the EVA timeline is not often experienced by the flight
control teams. Routinely performing the complete ingress and repress at the end of a terminate EVA
scenario where the EMU is in an off-nominal configuration would help train the team including crew and
flight directors as to how to best manage the EMU during these periods in the case of an actual failure.

Recommendation R-31: MOD should provide integrated EVA sims with the possibility of ending
the sim early. These sims must be scheduled for the full duration, but allowed to end early if
required by the actions taken by the flight control team. Additionally, airlock ingress and repress
should be routinely included as part of simulations that involve terminating an EVA with an
EMU in an off nominal configuration.
O-19 The Air Flow Contamination procedure contained a step that states MCC with no
instructions and inadequate rationale about what to do if this point in the procedure was
reached.
Supporting Evidence: As the ISS Flight Rules contain no rules pertaining to water in the vent loop and
particularly in the helmet, there was no training or procedures covering the failure. However, there has
been a cuff checklist procedure Air Flow Contamination that has been in the cuff checklist since prior
to 1989 that discusses the steps to take in the event of LiOH dust or LiOH contaminated water in the
EMU helmet with the steps to open the helmet purge valve and terminate the EVA. One section of the
procedure that would have applied in this case, If excessive water in the helmet leads to a step that just
says MCC with no further instruction. From the EVA Procedures Rationale Handbook (1989) it is
clear that it was thought that water in the vent loop would cause a problem with the fan (i.e. cause it to
stop), it would not cause the same kind of medical problem to the crew that LiOH dust/LiOH water
would. The assumption was that it must be caused by a leaky drink bag, urine collection device, or LCVG
and MCC would call for a terminate if the water reached the hands and feet (the return path to the vent
loop). The concern was still that water travelling up the vent lines would enter the LiOH cartridge and
return caustic LiOH water to the helmet. No mention is made of water coming from the PLSS in any
quantity other than as flowing through the LiOH. This was inadequate to handle this failure. It is also
noted that on the procedure it does have a note that the PLSS holds approximately 1 gallon of water.

128
 National Aeronautics and Space Administration EVA23Water Intrusion

Recommendation R-32: MOD should review all procedures with a MCC step and verify that
rationale exists to explain the required actions to be taken by the flight control team if this step is
reached.
O-20 The knowledge exchange between the FCR, MER, the CORE, and UTAS, Windsor Locks
was inefficient and did not allow all information to flow to all team members effectively in real-
time.
Supporting Evidence: In the MCC the Flight Director communicates to his/her team on dedicated
internal voice loops to facilitate communication among the team as well as provide an archive of the
conversations occurring. Each FCR operator is on a headset and required to listen to the Flight Director
(FD) loop as well as the Space-to-Ground (S/G) loop to hear the crew on-orbit and be ready to respond to
questions and by tracking the actions of the crew and status of the vehicle and respond proactively.
Likewise each FCR operator has a back room or Multi-Purpose Support Room (MPSR) occupied by
support personnel who are also on headsets and are required to monitor the Flight Director loop, the
appropriate FCR support loop as well as the S/G loop to support the FCR operator in their position. Most
real-time operations are managed solely from the MCC.

For questions that require a deeper understanding of the ISS systems, Engineering personnel staff the
Mission Evaluation Room (MER) and monitor the appropriate communications loops including FD, S/G
and appropriate FCR and MPSR loops via headset. In the case of EVA, additional Engineering personnel
are supporting in the Central Operations Room for EVA (CORE) located in JSC bldg. 7 as well as the
EMU contractor plant, United Technologies Aerospace Systems (UTAS) in Windsor Locks, CT. In each
of these locations at most one person is on headset, others monitor over a speaker.

The MER communicates with the CORE via telephone and similarly the CORE communicates with
UTAS over a speaker phone. In fact, the speakerphone used by the CORE to communicate with UTAS is
not located in the same room as the CORE, but rather in an adjacent office.

This means that if an engineer at the plant in Windsor Locks has an idea that needs to be passed forward;
it is literally a game of telephone to get that information to the FCR. There are no comm loops available
for all EVA personnel to share. Due to the time-criticality of EVA and the time to effect of EVA hazards,
there should be no impediments between the various members of the EVA team to communicate in real-
time. In this case, it likely would not have mattered as it seems all parties interviewed had no relevant
information about the water in the EMU helmet to share that was not shared, but in the future it could
mean the difference between success and failure.

Recommendation R-33: The ISS Program should ensure appropriate connectivity between all
relevant parties who participate in EVA activities to support real-time operations including
talk/listen access to MCC Audio Loops.
O-21 The team made the correct call in this case not to perform an abort subjecting the crew to
extreme physiological duress brought on by a rapid change in pressure.

O-22 The team discussed the possibility of performing an Abort EVA, emergency/rapid re-
pressurization of the airlock without full knowledge of the potential physiological ramifications.
Supporting Evidence: During the termination of EVA 23 and subsequently in discussions with
cognizant personnel, the suggestion has been made that when it became apparent that EV-2 had a serious

129
 National Aeronautics and Space Administration EVA23Water Intrusion

issue, that the condition warranted an EVA abort, requiring a rapid re-pressurization of the airlock rather
than the nominal rate. To abort an EVA is a decision not to be taken lightly as an abort EVA requires the
immediate return to the airlock by the affected crewman with the potential of leaving the other crewman
stranded outside during the repress event as well as leaving the ISS in an unsustainable configuration.

The hatch between the ISS crewlock and equipment lock uses an equalization valve from the Space
Shuttle to accomplish repress. According to the table on page 1.13-3 of the EVA Console handbook,
Vol.3, the Shuttle airlock had an empty volume of 208 cu. ft. and the ISS crewlock has an empty volume
of 188 cu. ft. The table references only re-pressurization rates associated with the Shuttle airlock, so
actual inflight rates on ISS would be higher as the crewlock is smaller and the rates do not include the
volume reduction associated with the presence of two EMUs of an additional 70 cu. ft. Repressing to a
cabin pressure of 10.2 psi, the table states that normal repress accomplished using the valve in the NORM
position is +2.66 psi/min at a flow rate of 169 lbm/hr and the abort rate with the valve in the EMER
position is +0.24 psi/sec (14.4 psi/min) at a flow of 899 lbm/hr. To a 14.7 psi cabin the rates are: NORM-
+3.82 psi/min at a flow rate of 243 lbm/hr and EMER - +0.34 psi/sec (20.4 psi/min) at a flow rate of 1295
lbm/hr.

The physiological limits on pressurization rates are found in NASA STD 3001, Vol. 2 NASA Human
Space Flight Standard requirement 6.2.2.2 Rate of Pressure Change where the maximum limit of
increasing pressure change is +13.5 psi/min to prevent barotraumas in space flight conditions, where
microgravity may have affected head and sinus congestion. Clearly an abort EVA case will violate this
condition whether repressing to 10.2 or 14.7.

From interviews and other discussions, it is also clear that not everyone involved in the decision making
process is comfortable with their knowledge of the criteria for or the effects of a rapid re-pressurization of
the airlock.

Recommendation R-34: MOD should strengthen training to emphasize the physiological effects
of a rapid repress on the crew to aid in the decision making process in real-time.
O-23 The separate organizational structure between the Extravehicular Activity Office and the
ISS Program has led to communication deficiencies which decrease the effectiveness of EVA
planning and could lead to issues affecting the cost, safety, and operation of the ISS.
Supporting Evidence: To better investigate the organizational structure and decision making processes
used in managing EVAs, the MIB reviewed organizational charts from the ISS Program (OA) and the
Extravehicular Activity Office (XA), and also conducted numerous witness interviews and discussions
with multiple individuals from both organizations. The interviews ranged across a broad spectrum of all
levels of the organizations, including management, employees, and contractors. This data provided an
overall view of the organizational structure that identified several issues.

From the XA organizational website: The EVA Office is charged with the responsibility to serve as the
EVA Program management authority within NASA. As such, the EVA Office is chartered to provide
final review and approval for all areas of EVA, including safe execution of EVA, training, integration and
operations, development for suits, systems and support equipment, and all EVA-related advanced
technologies. The ISS Program Plan states that: The Program mission is to safely build, operate, and
utilize a continuously inhabited orbital research facility through an international partnership of
governments, industries, and academia. Since EVA capability is a critical component of building,

130
 National Aeronautics and Space Administration EVA23Water Intrusion

operating, and maintaining the ISS, it is clear from the official descriptions of the responsibilities of both
organizations that there is a significant overlap in the organizations responsibilities. In addition, from
interviews the MIB learned that a significant amount of XAs funding comes from the ISS Program.

During the timeframe of the close call, both preceding and following the close call, XA was undergoing a
significant reorganization and management change. Through interviews, the MIB learned that
immediately after the close call, the ISS Program Manager tasked XA with leading the failure
investigation. The organizational change in XA management did complicate the ISS investigation of the
close call since new managers needed to establish their organizations priorities and this perhaps slowed
the beginning of the investigation to some extent, but the MIB determined that this did not compromise
the investigation. A more important observation by the MIB is that the organization structure of the JSC
EVA office places it outside the direct line of responsibility of the ISS Program, and vice versa. Major
decisions regarding EVA hardware, logistics, and planning are made at the EVA Change Control Board
(CCB), however no ISS Program representative, or ISS integration lead, formally participates in that
forum. In addition, the ISS Program makes Program level decisions (such as ISS EVA schedules) which
affect XA areas of responsibility at the Space Station Program Control Board (SSPCB), which does
include a member of XA, but the representative is generally someone from the XA Increment team, not a
high level manager that controls strategic decisions made by XA. Given this separate organizational
structure, communication between the ISS Program and XA is challenging, which was pointed out during
several interviews. However, the ISS Program has a strong reliance on EVA capabilities, and thus XA, to
manage the safety, effectiveness, and continued operations of the ISS. Conversely, The ISS Program is
currently the major customer of the services provided by XA so ISS Program decisions directly affect XA
and their planning. This relationship thus requires very close integration between the organizations.

Recommendation R-35: The ISS Program and JSC EVA Office should improve technical and
management coordination between their two organizations and ensure that all strategic and
tactical decisions that are made by either organization are quickly and effectively understood, and
officially accepted by both.
O-24 The One EVA prime EVA contractor and their sub-contractors who provide EMU
expertise to both XA and the ISS Program are not placed in an environment that is conducive
to freely participate in and challenge technical decisions that are made by their governing
boards.
Supporting Evidence: During the early stages of the MIB investigation, meetings were held with the
prime EVA contractor UTAS at their facility in Windsor Locks, Connecticut. At this meeting, an
organizational chart was provided that described the contract, known as One EVA, which XA uses to
manage their EVA contractor activities. UTAS is the prime contractor overseeing several sub-contractors,
but UTAS also provides the technical expertise for the Primary Life Support System. The MIB learned
through observing the ISS Investigation Team and through interviews that the participation and authority
of the EVA contractors is generally less active in forums where decisions are being made. The MIB
determined that this leads to an environment where the contractor at times feels hesitant to voice concerns
and issues they may have and defers to the civil servant officials. This environment is not a structural
issue since One EVA is represented at the CCB; however, the culture of their participation is that they
generally limit their active participation in debates and defer to the decisions made by the government
officials at the boards. The MIB determined that this was not due to a fear of reprisal, but more of an

131
 National Aeronautics and Space Administration EVA23Water Intrusion

environment where the contractor was considered as having a support function rather than being a full
participating and voting member of the Board.

Recommendation R-36: The government officials and contract managers must put in place
expectations and create a board environment that allows the EVA contractors to freely challenge
technical decisions made by the governing boards when appropriate and encourage proactive
participation.
O-25 EVA risks have not been well integrated within the context of the ISSP risk management
process.
Supporting Evidence: EVA safety responsibilities are divided among 4 different organizations: the
EVA Safety and Reliability Group under the S&MA Integration Branch of the Quality and Flight
Equipment Division (NT), the ISS Safety & Mission Assurance/Program Risk Office (OE) in the ISS
Program, the ISS Chief Safety Officer in the JSC S&MA Directorate (NA), and the MOD Flight Safety
Office (DA8). A fifth organization that has a related role is the ISS Safety Division (NE) which provides
staffing for the Safety position in the ISS MER.

Each of these organizations has a role to play that they do very well, but the integration between them
needs to be robust. These organizations have the following high level responsibilities:

NT Is embedded within the EVA Office and has overall S&MA responsibilities for EVA including
providing hazard analysis, COFR, and quality assurance. The EVA Safety and Reliability Group further
provides support to the Safety Review Panel (SRP) for EVA related matters and supports ISS Safety in
the MER during EVA operations. The EVA Safety and Reliability Group is funded by the EVA Office
through Internal Task Agreements. Funding to the EVA Office is provided by the ISSP.

OE Is responsible for assuring all ISS hardware and software has successfully completed the S&MA
processes in an effort to minimize risk to the crew and vehicle by providing S&MA integration across ISS
program elements and chairing the Safety Review Panel. The ISS Safety & Mission Assurance/Program
Risk Office provides integrated hazard analysis for ISS as well as the integrated S&MA COFR for ISS.

NE Provides S&MA support for all other non-EVA ISS systems (with the exception of Government
furnished equipment (GFE)). The ISS Safety Division staffs the ISS Safety position in MER and provides
support to OE. NE is funded by the ISS Program through internal task agreements.

NA, Chief Safety Officer Serves as independent technical authority for HQ OSMA. The Chief Safety
Officers are funded by HQ OSMA.

DA8, MOD Flight Safety Office The MOD Operations Safety Office is primarily responsible for
establishing S&MA processes and procedures within the MOD, and for administering the MOD S&MA
program.

The ISS for the most part is contractor furnished via a prime contractor with NASA oversight provided by
various elements of the NASA community and integrated by the Program. The exception is for EVA
where the EVA hardware is government furnished via a non-prime contract to the ISS Program. EVA
integration is provided by the EVA Office more or less as a service to the ISS Program. EVA is then
integrated into the rest of the program through various ISS integration functions.

132
 National Aeronautics and Space Administration EVA23Water Intrusion

ISS safety is organized similar to the way the ISS Program is organized. The ISS Safety Division (NE)
provides the NASA safety oversight with integration being provided by the ISS Safety &Mission
Assurance/ Program Risk Office (OE). Once again the exception is EVA where the EVA Safety and
Reliability Group under the S&MA Integration Branch of the Quality and Flight Equipment Division
(NT) provides the S&MA oversight of EVA in support of the EVA Project Office. The ISS Safety
&Mission Assurance/ Program Risk Office (OE) also provides the safety integration function between the
ISSP and EVA.

The organizational deficiencies that have been identified by this Board with regard to EVA overall, (see
O-28) exist also in the realm of safety. The separateness of EVA from the rest of the ISS community
extends to the safety community as well. It is only natural that, although unintentional, the tendency is to
view EVA safety independent of the context of its use within the ISS Program. Risks, therefore, are not
always elevated in that context or if they are, are not communicated in such a manner as might be most
revelatory to the decision maker. As it has been explained to members of the Board, it is not clear that
when risks have been brought forward they have been put in terms meaningful to the ISS Program. They
may not have clearly identified that when accepting certain EVA risks, what the additional level of risk
being accepted by the Program is due to the termination or aborting an EVA at a time that would impact
the success of an EVA critical to ISS.

This is particularly important for EVA risks in that EVA is inherently more risky that other elements of
the ISS as the requirements under which the ISS has been developed are different from a failure tolerance
standpoint than the EMU. As has been pointed out elsewhere, the EMU is a legacy from the Space Shuttle
Program. It was designed originally to be brought into space, used several times, and brought home again
for refurbishment. It was designed as a single fault tolerant system with an abort capability within the
time to effect. The rest of the ISS is generally two fault tolerant. The EMU is also now operating in a
different environment than that for which it was originally designed. Its life has been extended and it is
getting older. Decision makers have to keep this all in mind when addressing EVA risks.

Generally, the MER Safety position is staffed by NE personnel who are very knowledgeable about ISS
systems and risks. For all of the reasons previously cited, they have less understanding of EVA systems
and the risks associated with EVA. That knowledge resides within NT. During EVA operations, NT
supports NE at the MER Safety console. However, without proper training, it is likely that the MER
Safety Officer will not know to act or pass along information or recommendations of the supporting EVA
personnel without extended discussion which may not be possible in real time.

It is also of concern to the Board that the situation may be further exacerbated by the funding sources of
the various safety entities. In these cases the risk is also high of personnel wanting to please the
customer. Concern has been raised that sometimes when issues have been brought forward, the
philosophy of the funding organization, whether it is perceived as from ISS or the EVA Office, is taken
by the supporting safety organization. This is a dangerous precedent and can lead to group think. Efforts
must be made to reinforce the independence of the safety community from the Program or Project.

One mechanism for maintaining and ensuring independence under the current funding model is the Chief
Safety Officer funded out of NASA Headquarters, independent of any program. Under the current
organizational structure, it is not clear that maximum use is being made of this office. Subconsciously, it
appears on both sides of the relationship there is a feeling of otherness. Within the EVA Office the role

133
 National Aeronautics and Space Administration EVA23Water Intrusion

of EVA CSO has been performed informally by the NT EVA Group Lead. NT should feel as though the
ISS CSO is their advocate within the Program structure and feel an obligation to help the ISS CSO have
the correct contextual understanding with respect to EVA risks. Steps should be taken to more fully
integrate the EVA safety function into the larger ISS safety community.

Recommendation R-37: To reinforce the independence of safety and recognize the unique
criticality of EVA in the safety community, consider altering the ISS CSOs office to more
closely mirror that of the ISS Chief Engineers Office by creating a deputy CSO for EVA position
to more closely work with the EVA safety community and help integrate them into the ISS
Program and aid the CSOs and Program Managers understanding of EVA risks in the context of
the ISS Program.
Recommendation R-38: JSC Safety and Mission Assurance should provide additional EVA
training and integration activities to the MER Safety Officer training syllabus.
O-26 JSC Safety and Mission Assurance personnel supporting EVA are relying largely on
experience gained from previous jobs to perform their current job in safety.
Supporting Evidence: By-and-large, the personnel in the EVA Safety and Reliability Group (NT) that
support EVA have done and continue to do an outstanding job. They are well respected in the EVA
community and work well with the engineering team and with the ISS and EVA Office personnel.
However, they are hampered by a lack of training in EVA hardware design, its use and its failure history.
The corporate knowledge contained within the EVA Safety and Reliability Group concerning EVA
hardware, especially the EMU, is very sparse, being contained in a couple of personnel who are soon to
be retirement eligible. By and large the training and experience the personnel within the group received
prior to coming into the group is what is being relied on at this point. There is currently no mechanism to
train the EVA Safety and Reliability Group personnel in the function or failure history of EVA hardware.
The contractor (SAIC) has a training and certification program for their personnel that are to work in
support of the MER, but there is no equivalent training for the civil servant personnel. The contractor
training also lacks sufficient knowledge capture of EVA failure history.

Recommendation R-39: JSC Safety and Mission Assurance should institute a training program
for all of its EVA personnel that includes a subset of MOD EVA task and EVA systems training
flows to gain the requisite training on EVA hardware and tasks it is being used on. This training
should be supplemented by observing EMU vacuum chamber runs, NBL runs, hardware reviews,
and ground testing both at SGT and UTAS Windsor Locks and studying the EMU Requirements
Evolution document should be mandatory.
Recommendation R-40: JSC Safety and Mission Assurance should routinely advocate for and
lead the periodic review of FMEA/CILs and Hazard reports and be intimately familiar with their
content.
O-27 Resources have been reduced to the point that EVA personnel can generally only cover
routine crew and personal training as well as multiple mission control shifts, but have little
margin for contingencies without burnout as well as pushing forward a back log of needed
work.
Supporting Evidence: From interviews and discussions with MOD management, EVA MOD personnel
have been reduced from a high of 54 during ISS assembly to 38 now due to the completion of ISS and the
retirement of the Shuttle. Those remaining are stretched thin to cover routine training and multiple
mission control shifts with little margin for contingency affairs without burnout. In the course of this

134
 National Aeronautics and Space Administration EVA23Water Intrusion

incident, MOD was strained to perform only tasks to support their MCC responsibilities and the Team
4/EVA recovery efforts and had to push everything else aside. Work that went undone includes not
participating in 12 integrated simulations delaying certification of new EVA Flight Controllers at all
levels 2-3 months, cancelling 18 Flight Controller Part Task Trainer sessions further impacting future
certifications, and delaying work on updates to the flight controller and instructor proficiency flows.

Prior to the incident, resources have not been available to perform necessary updates to the following
products: console handbook, work instructions, lesson plans, and the procedure rationale handbook.
Support to the advanced suit work has also been deferred due to resource limitations particularly because
that work is currently limited to civil service personnel only. This deferred work does not yet include the
additional work needed as a response to the Boards findings.

EVA community resources also do not seem adequate for long term support of the increased frequency of
EVAs needed for ISS planned maintenance as seen in Figure 3-69 combined with the expectation that
there may be a need for an increased number of CCEs per year through 2020.

Figure 3-69 Projected Future Maintenance EVAs

Recommendation R-41: ISS Program should augment, at least temporarily, MOD EVA personnel
to allow the existing backlog of work to be completed in a fairly short order by bringing on,
through rotational opportunities, personnel that can provide valuable technical assistance that will
not add to the training and certification burden already faced by the organization.
Recommendation R-42: ISS Program should provide additional long term resources to augment
current EVA community staffing to support the coming increased frequency of ISS maintenance
and contingency EVAs.
O-28 The roles and responsibilities of the MER and FCT along with their mutual expectations
with regard to anomaly resolution are not clearly understood.
Supporting Evidence: From interviews, discussions with relevant personnel and observation of the
various organizations and how they interacted during this investigation, it is clear to the MIB that there
has been an erosion in communication between and the defined roles and responsibilities of each of the
organizations that participate in real-time or near real-time operations. This can to a large part be

135
 National Aeronautics and Space Administration EVA23Water Intrusion

attributed to the history of the development of the two organizations during short term missions such as
Apollo and the Space Shuttle. The evolution of the relationship in the relatively new paradigm of long
term continuous ISS operations has not been completed. This can have a deleterious effect on the speed
and accuracy at which decisions can be made in real-time and transmitted to the crew onorbit. Although
not contributory to this HVCC, it may adversely impact a future situation. These issues can be broken
down into several areas that need to be worked to correct the situation.

The Flight Control Team is charged with the safe and efficient operation of the ISS. To fulfill this role,
they must rely on the support of the ISS Program and the Engineering communities. To that end the ISS
Program Manager, through his approval of the Station Program Implementation Plan (SPIP) (SSP-50200-
09) and ISS Flight Rules (SSP-50643) has delegated authority to the Flight Director (FD), and by
extension the Flight Control Team (FCT), to act on the behalf of the Program for all issues relating to
real-time operations. Specifically Volume 9 of the SPIP states: The Flight Director has overall
authority and responsibility for the safety of the ISS and crew, planning and plan execution, systems
operations, and anomaly troubleshooting. The Flight Director is the final authority on whether to
continue any ISS activity, including ISS payloads operations. The SPIP further states: The Flight
Director has the responsibility and authority to take any action required to ensure the safety of the crew
and ISS. When decisions are required outside of the ISS operating base, the Flight Director will
consult the MMT via the Mission Operation Directorate (MOD) when time permits. Additional detail as
to the authority of the FCT is contained in the ISS Flight Rules (SSP-50643).

The SPIP also defines the role of the Engineering community in real-time operations: ISS Program
Sustaining Engineering provides on-orbit engineering support with expert systems/hardware design
engineers. This support includes detailed systems performance analyses to resolve system anomalies and
identification of trends that could lead to degraded performance and/or system failure. Sustaining
Engineering will provide engineering support in the Mission Evaluation Room (MER) as required. In the
event of an anomaly, the flight control team will turn to Sustaining Engineering for resolution of the
anomaly.

It is clear that both sides are frustrated to a certain extent with the current state of affairs. Most of the
issues observed by the MIB the MIB feels can be overcome by improved communication at all levels
within both organizations. The roles and responsibilities of both organizations with regard to anomaly
resolution need to be more fully defined below that at the SPIP or Flight Rules level and expectations
discussed and agreed to by both parties.

A case in point: in reviewing the ISS In-Flight Anomaly Resolution Process Work Instruction (MGT-
OA-019), the process by which the MER operates in the identification, investigation, and ultimate
entrance into the PRACA system was found to be evidently well documented. It also holds clues as to
areas where conflicts between the FCT and the MER may arise. The MGT-OA-019 document details the
criteria for which Anomaly Resolution Teams (ARTs) and Failure Investigation Teams (FITs) may be
created at the discretion of the MER Manager apparently without consultation with the Flight Director.
This can be problematic as for an ART the FCT is a mandatory participant and for the FIT is highly
desired both of which can cause resource issues on the part of the FCT. This can also lead to confusion as
there seems to be significant overlap between an ART and an MOD-led and FCT initiated Team 4
activity. An ART is described as A formal Tiger Team established by the ISS MER (emphasis the

136
 National Aeronautics and Space Administration EVA23Water Intrusion

Boards) to investigate critical on-orbit anomalies, to identify immediate on-orbit corrective actions to
prevent/mitigate on-orbit impacts to the ISS and/or crew, and to prevent/minimize recurrence of the
anomaly. Similar language is used by the Flight Director Office when describing a Team 4 activity.
From the Flight Control Operations Handbook (FCOH) Standard Operating Procedure 6.10.2.3: The
Team 4 group will perform an evaluation of any major ISS systems anomaly requiring time-critical
resolution and, if required, develop a contingency operations plan which can be implemented by the real-
time mission controllers and the onboard crew. The conflict may be due to the fact that the work
instruction, although signed at the Program level and in the document is stated to be applicable to any
organization that is called upon to support the investigation and resolution of ISS on-orbit anomalies
including the Flight Control Team. Unfortunately, no one from MOD is a signatory to this document to
agree with it.

Furthermore, there seems to be an inconsistent understanding on the part of the MER personnel
concerning their understanding that the job of risk acceptance for execution planning and real-time
operations rests with the Flight Control Team when the need arises to provide support with best
engineering judgment.

Recommendation R-43: The ISS Program must define The Roles and Responsibilities of the
MER and the FCT to a level whereby each position (FCT and MER) on either side clearly
understands their role and the role of their counterparts and mutual expectations must be
established and agreed to. As part of this effort, the Program needs to reinforce the understanding
that it is the FCT that is authorized to accept risk on behalf of the Program in real-time operations
requiring best engineering judgment.
Recommendation R-44: The ISS Program must establish a protocol whereby whenever conflicts
arise between the MER and FCT concerning roles and responsibilities or one partys performance
during a particular event, the appropriate management from each side must meet to discuss the
conflict and revise the roles and responsibilities or expectations accordingly.
Supporting Evidence: In the course of the investigation from interviews with cognizant personnel, it
was determined that the certification requirements for each MER console position or MER Subsystem
Team (MST), as they are known, varied by subsystem and the various teams have differing requirements
for certification currency. From discussions it was concluded that at the MST level the certification
requirements are satisfactory. It should be noted that the MIB did not perform an audit of each position.
However, one area did stand out as a notable exception: the MER Managers who are charged with the
responsibility of managing the resources of the MER as well as providing integrated responses to the FCT
in a timely manner are certified for their positions once and have no recertification or proficiency
requirements. Furthermore, the MER managers, due to their workload, have little time for additional
training to deepen their understanding of ISS systems. MER Managers are funded for approximately 12
full time positions consisting of 6 Contractor and 6 civil servant personnel. Currently, due to attrition,
their numbers have been reduced to 10. Even at full staffing MER Managers are unable to find time to
augment their experience with training to broaden their experience. The MIB further found that the
system currently in place where MER Managers rotate around to different functional areas for periods of
time is to be commended and encouraged as a good way to provide broadening, however, it does not
replace good, solid training as is required and provided to their Flight Control counterparts.

137
 National Aeronautics and Space Administration EVA23Water Intrusion

Recommendation R-45: The ISS Program should develop proficiency requirements for MER
Managers by event they are certified to support, as well as on a time basis (e.g. annually) to
maintain currency.
Recommendation R-46: The ISS Program should provide training to the MER Managers to
deepen their systems and vehicle knowledge to ensure proper subsystem and situational
awareness during real-time operations.
Supporting Evidence: In discussions concerning the above observations, it was found that, under the
current contractual instruments, it may be difficult to implement these recommendations.

Recommendation R-47: The ISS Program should immediately modify the contractual clauses that
may prevent the recommendations contained in this report from being implemented within the
contractor community.
O-29 Human Factors issues played a significant role across the spectrum of findings discovered
by the MIB.
Supporting Evidence: In the course of this investigation, the MIB has looked inside NASA operations
and derived a number of contributing human factors, findings, and recommendations. One
recommendation that has not yet been addressed relates to how other organizations have tried to reduce
their accident (mishap) rate. It is one thing to be an outsider looking in on an operation. In such instances
one relies upon the statements and opinions of others, from which to draw their own conclusions.
However, a case can be made that it is the insider, working from within, who can best contribute to an
organizations operational performance and safety.

Military aircraft operations are inherently dangerous because teams are taught to train as they fight, and
fight as they train. Hence, the military sustains a higher rate of aircraft accidents than their commercial
counterparts, making it important for them to utilize solutions with a proven trend toward reducing
mishaps. Focusing on the human factors element, military organizations train their flight surgeons to
become subject matter experts in this field, and then embed them at the operational level. In addition to
providing medical care, flight surgeons, as human factors experts, work very closely with aircrew,
maintenance, and line personnel. For example, a significant portion of a flight surgeons time is spent in
the operational setting, outside of the clinic. In support of operations, they conduct such things as pre-
deployment briefs, lectures at safety seminars, and quarterly human factors training for aircrew. They
review aircraft flight recordings with a focus on assessing the pilots human factor performance, and they
function as board members in mishap investigations. It may be worth noting that a majority of flight
surgeons are not attached to the base clinic, but rather the operational squadron itself. It is from within the
organization that they are able to offer their greatest contribution toward performance, safety and the
prevention of mishaps.

Recommendation R-48: NASA real-time operations community should work with the JSC
Human Factors team to assess areas where human factors processes can be better trained and
implemented in operations and develop specific training to reduce the impact of human factors in
future mishaps.
O-30 The ISS Program is currently relying on a single fault tolerant system to provide critical
EVA capability.
Supporting Evidence: The ISS has stated a goal of operations through 2028. As this failure has
illustrated, relying on this legacy system to provide EVA capability far into the future carries risk to the

138
 National Aeronautics and Space Administration EVA23Water Intrusion

program. Most ISS systems are designed to be two fault tolerant and are maintainable on-orbit. The EMU
on the other hand was designed to be single fault tolerant with an abort capability. It was not designed to
be maintainable on orbit, nor was it designed to perform more than a handful of EVAs before returning to
the ground for refurbishment. The EMU has proven itself to be a workhorse enabling the assembly,
maintenance and operation of the ISS, but it is operating at the limits of its design margin and has shown
some significant vulnerability to the ISS environment. Most notably with issues of water compatibility as
discussed elsewhere in this report and illustrated by the failure of EMU 3015 and the previous disabling
failures with 3005, 3011, and 3013 post-Columbia.

To continue to 2028 it is the sense of the Board that it would be prudent to pursue a plan that addresses
these issues through solutions such as design changes that would decouple the vent loop from the cooling
loop to eliminate the risk of the failure currently under investigation, providing additional maintainability
features, improving in-situ suit reliability, performance trending, and failure diagnosis. In the process it
may be found that it may be most cost effective to consider implementing all or parts of a new EMU
which is under development and necessary for any future exploration missions.

Recommendation R-49: The ISS Program should commission an independent study team to
identify options to ensure an ISS EVA capability through 2028 that trades improvements to the
current single fault tolerant suit via options such as additional on-orbit diagnostics and
preventative/corrective maintainability, redesign to separate water and vent loops, and/or
implementation of an advanced suit.

139
 National Aeronautics and Space Administration EVA23Water Intrusion

4.0 Findings
This MIB found the following issues that contributed to this incident. Root Causes, Proximate
Causes, Intermediate Causes, Contributing Factors, and Observations are listed in this section. Since the
investigation of the Fan/Pump/Separator is still ongoing, not all root causes have been identified.

4.1 Proximate Causes

A Proximate Cause is the event(s) that occurred, including any condition(s) that existed
immediately before the undesired outcome, which directly resulted in the occurrence of the undesired
outcome and, if eliminated or modified, would have prevented the undesired outcome.
Based on this definition, the MIB noted three (3) proximate causes for this HVCC.
P-1. The ISS Program conducted EVA 23 without recognizing the EMU failure which occurred
on EVA 22
P-2. EMU 3011 Helmet had large quantity of water during EVA 23
P-3. Flight Control Team/Crew did not terminate EVA 23 as soon as water was reported in the
helmet

4.2 Intermediate Causes

An Intermediate Cause is an event or condition that created the proximate cause that, if eliminated
or modified, would have prevented the proximate cause from occurring.
Based on this definition, the MIB noted 19 intermediate causes.
I-1. Flight Control Team and Crew incorrectly attributed the water in the helmet during EVA
22 to the drink bag.
I-2. Flight Control Team did not perform investigation of the water source following EVA 22.
I-3. Flight Control Team accepted the explanation that the water during EVA 22 was from the
drink bag.
I-4. Ground Team allowed time pressures of impending EVA to influence actions.
I-5. Large quantity of water from vent loop leaked into helmet during EVA 23.
I-6. EVA 23 Crew did not immediately recognize the severity of the event.
I-7. Crew member Training did not include this failure mode.
I-8. Flight Control Team focused on the drink bag as the source of the EVA 23 water.
I-9. Critical Information was not communicated between Crew and Ground Team.
I-10. Airflow Contamination procedure did not address the failure mode.
I-11. Flight Control Team did not understand the failure mode.
I-12. Team had Channelized Attention on the drink bag as the primary source of water in the
helmet.
I-13. Team's set of responses led to a delay in identifying the vent loop as the source of the water
leak.

140
 National Aeronautics and Space Administration EVA23Water Intrusion

I-14. EV1 initially focused on drink bag as source in EVA 23.

I-15. Flight Rule to address this failure mode did not exist.
I-16. EMU Hazard Report did not identify the hazard.
I-17. Engineering Team did not understand the failure mode.
I-18. Safety Team did not understand the failure mode.
I-19. FMEA/CIL did not effectively describe the behavior of water entering the vent loop from
the PLSS.

4.3 Contributing Factors

A Contributing Factor is an event or condition that may have contributed to the occurrence of an
undesired outcome but, if eliminated or modified, would not by itself have prevented the occurrence.
Contributing Factors increase the probability that an event or condition will occur.
Based on this definition, the MIB noted 13 contributing factors.
CF-1. Flight Control Team sent EV2 back to Airlock unaccompanied during EVA termination.
CF-2. EV2 did not have the experience base to immediately recognize the severity of the situation.
CF-3. EV2 was unable to communicate the severity of his condition after the terminate call.
CF-4. Ground team did not identify PLSS as actual source of water.
CF-5. Cognitive Task Oversaturation contributed to the teams delayed identification of the actual
source of the water leak.
CF-6. EV2s comm system failed due to water.
CF-7. FMEA/CIL did not effectively quantify the amounts of water entering the vent loop from
the PLSS.
CF-8. Minimal Formal Training on EMU function existed for the Safety Team.
CF-9. FMEA/CIL did not undergo thorough review and update periodically.
CF-10. Requirement for specific EVA/EMU training of Safety personnel did not exist.
CF-11. ISS program FMEA/CIL requirement did not require complete FMEA/CIL periodic
update.
CF-12. Program cut funding.
CF-13. Community had lost sight of the value of the FMEA/CIL effort.

141
 National Aeronautics and Space Administration EVA23Water Intrusion

4.4 Root Causes

A Root Cause is one of multiple factors (events, conditions, that are organizational factors) that
contributed to or created the proximate cause and subsequent undesired outcome and, if eliminated or
modified, would have prevented the undesired outcome. Typically, multiple root causes contribute to an
undesired outcome.
Based on this definition, the MIB noted 5 root causes.
RC-1. Program emphasis was to maximize crew time on orbit for utilization.
RC-2. ISS Community perception was that drink bags leak.
RC-3. Flight Control Teams perception of the anomaly report process as being resource intensive
made them reluctant to invoke it.
RC-4. No one applied knowledge of the physics of water behavior in zero-g to water coming from
the PLSS vent loop.
RC-5. Minor amounts of water in the helmet was normalized.

4.5 Observations
An Observation is a factor, event, or circumstance identified during the investigation that did not
contribute to the mishap or close call, but, if left uncorrected, has the potential to cause a mishap or
increase the severity of a mishap; or a factor, event, or circumstance that is positive and should be noted.
Based on this definition, the MIB noted 30 observations. Descriptions of each Observation can be
found in Section 3.7.
O-1 Based on what they knew at the time, the Ground Team performed admirably.

O-2 Helmet Anti-fog material is problematic.

O-3 The EMU CO2 sensor has a history of failing during EVA due to excess moisture in the
event loop.

O-4 EVA POCC oversight of EVA flight control team may be detrimental.

O-5 Use of the Helmet Purge Valve (HPV) was questioned by some flight controllers after the
terminate call was made

O-6 The process by which on-orbit EMU non-conformances are initiated and ultimately closed
is inconsistent with best practices and seems to be implemented inconsistently.

O-7 Achievement of 6-year on-orbit certification of the EMU appears inadequate.

O-8 Differences noted between EMU plant and field procedures indicate a lack of two-way
feedback and procedure control.

O-9 The pace and potential hazards associated with EVAs on ISS are similar to other activities
that should receive similar scrutiny by the ISS Program.

142
 National Aeronautics and Space Administration EVA23Water Intrusion

O-10 Water quality and chemistry in multiple onboard ISS systems is constantly changing and
many critical ISS Systems are sensitive to these changes, yet a systematic method to
regularly monitor and maintain the systems that affect the EMU and other water systems
does not exist.

O-11 During EVAs, EMU suit telemetry that provides information about certain critical
operational parameters (fan speed, battery current, water temperatures, etc.) to the ground
teams is limited to one data point every 2 minutes.

O-12 The ISS program is developing the capability to launch and return only one full short EMU
at a time.

O-13 Flight control and Engineering teams have a general inability to access flight hardware to
perform training of personnel (including flight crew) as well as validating procedures or
performing other engineering tests.

O-14 There is no backup EVA capability in the event the need for a contingency EVA arises.

O-15 Lessons learned databases and corporate knowledge information exist, but are not always
easily accessible, often incomplete and are not being fully utilized.

O-16 Failures often are only tracked to proximate cause with little meaningful trending being
performed or root causes being pursued.

O-17 The Flight Crew and all ground-based MCC and MER personnel involved in the event were
properly certified for their positions by their respective organizations.

O-18 Integrated sims are intended to run their entire scheduled length, which causes the Flight
Control Team to never experience the pressure of terminating an EVA early before a
majority of the objectives are accomplished and may be providing negative training that all
problems can be overcome in the course of an EVA and therefore delays the decision to
terminate.

O-19 The Air Flow Contamination procedure contained a step that states MCC with no
instructions and inadequate rationale about what to do if this point in the procedure was
reached.

O-20 The knowledge exchange between the FCR, MER, the CORE, and UTAS, Windsor Locks
was inefficient and did not allow all information to flow to all team members effectively in
real-time.

O-21 The team made the correct call in this case not to perform an abort subjecting the crew to
extreme physiological duress brought on by a rapid change in pressure.

O-22 The team discussed the possibility of performing an Abort EVA, emergency/rapid re-
pressurization of the airlock without full knowledge of the potential physiological
ramifications.

O-23 The separate organizational structure between the Extravehicular Activity Office and the
ISS Program has led to communication deficiencies which decrease the effectiveness of EVA
planning and could lead to issues affecting the cost, safety, and operation of the ISS.

143
 National Aeronautics and Space Administration EVA23Water Intrusion

O-24 The One EVA prime EVA contractor and their sub-contractors who provide EMU
expertise to both XA and the ISS Program are not placed in an environment that is
conducive to freely participate in and challenge technical decisions that are made by their
governing boards.

O-25 EVA risks have not been well integrated within the context of the ISSP risk management
process.

O-26 JSC Safety and Mission Assurance personnel supporting EVA are relying largely on
experience gained from previous jobs to perform their current job in safety.

O-27 Resources have been reduced to the point that EVA personnel can generally only cover
routine crew and personal training as well as multiple mission control shifts, but have little
margin for contingencies without burnout as well as pushing forward a back log of needed
work.

O-28 The roles and responsibilities of the MER and FCT along with their mutual expectations
with regard to anomaly resolution are not clearly understood.

O-29 Human Factors issues played a significant role across the spectrum of findings discovered
by the MIB.

O-30 The ISS Program is currently relying on a single fault tolerant system to provide critical
EVA capability.

144
 National Aeronautics and Space Administration EVA23Water Intrusion

5.0 Recommendations
Proximate Cause
(PC),
Intermediate
Cause(I),
Recommendation Contributing
Factor (CF), Root
Cause(RC),
Observation (O)

Recommendation The ISS Program must reiterate to all team members that, if they RC-1
R-1: feel that crew time is needed to support their system, a request
and associated rationale must be elevated to the ISS Program for
an appropriate decision.
Recommendation ISS Program should ensure that the FMEA/CILs are updated I-7
R-2: and maintained and MOD should make them required
reading/study for all EVA Systems instructors and Flight
Controllers up to and including FCR operators as well as their
proficiency flows. EVA safety and Engineering MER support
personnel should also include this in their training flows
Recommendation MOD SSTF instructors should ensure that training includes use I-7
R-3: of the FMEA/CIL to develop failure scenarios for use in
integrated and stand-alone simulations.
Recommendation The ISS Program should ensure that updates are made to the I-16
R-4: EMU hazard reports to reflect the possibility of water in the
helmet resulting in a catastrophic event due to asphyxiation.
Recommendation The ISS Program should ensure that the FMEA/CIL is updated I-19
R-5: and reviewed thoroughly from end-to-end every two years to
ensure currency with participation by Engineering, MOD,
Safety, Medical, and appropriate contractor personnel.
Recommendation The ISS Program should ensure that all instances of free water RC-5
R-6: and contamination in the EMU are documented and
investigated, with corrective action taken, if appropriate.
Recommendation MOD must lead the development of appropriate flight rules and I-15
R-7: procedures to address the course of action to take in the event of
water in the helmet.
Recommendation The ISS Program should investigate alternate materials that O-2
R-8: effectively perform the helmet anti-fogging function without the
risk of eye irritation.
Recommendation The ISS Program should investigate alternate CO2 sensor O-3
R-9: designs that eliminate the sensitivity to moisture.
Recommendation MOD should evaluate how personnel who are located in the O-4
R-10: POCC facility and not part of the active flight control team
interact with the active flight control team and ensure that lines
of communication and the decision making chain is not
compromised.

145
 National Aeronautics and Space Administration EVA23Water Intrusion

Recommendation The ISS Program should perform testing and analysis to verify O-5
R-11: that use of the Helmet Purge Valve to remove free water from
the helmet is safe and effective. Results of this testing should be
made clear to the EVA community, including the flight control
team and documented in hazard reports, flight rules and
procedures.
Recommendation ISS Program and Safety and Mission Assurance should review O-6
R-12: and update the process as defined in JSC 28035 to resolve the
conflict of interest of the EVA Office in initiating FIARs.
Recommendation Safety and Mission Assurance with the assistance of the EVA O-6
R-13: Office should initiate a review of all non-conformances
contained in the PRACA database for the EMU and review the
assignment of FMEA associated with each one and update as
required.
Recommendation The ISS Program should commission an independent technical O-7
R-14: review of the EMU 6-year certification plan which should
identify all deficiencies or weaknesses in the certification and re-
establish the true life expectancy of the EMU, and then plan
appropriate use and logistic strategies commensurate with the
results of the review.
Recommendation The EVA Office should ensure that all EMU procedures are O-8
R-15: consistent between all teams that perform operations with the
EMU, and require that all contamination found during ground
processing be evaluated by the Engineering and Quality teams.
Recommendation MOD should ensure that simulations of specific, fast-paced O-9
R-16: failure scenarios (visiting vehicles, on-board emergency
response, software transition issues, and serious system
hardware failures) should include all phases of the teams
response to ensure that the response can be fully performed from
end-to-end in a quick, proficient manner.
Recommendation The ISS Program should ensure that FMEA/CILs related to fast- O-9
R-17: paced failure scenarios (visiting vehicles, on-board emergency
response, software transition issues, and serious system
hardware failures) are regularly updated, studied, and used in
training for flight controllers as well as engineering and safety
personnel.
Recommendation As the success of the ISS Program continues, the ISS Program O-9
R-18: must institute requirements and behaviors that combat the
tendency towards complacency by requiring regular training by
all teams in the safety critical aspects of failures related to fast-
paced scenarios (visiting vehicles, on-board emergency
response, software transition issues, and serious system
hardware failures).

146
 National Aeronautics and Space Administration EVA23Water Intrusion

Recommendation The ISS Program must ensure that full root cause determination O-9
R-19: of failures related to specific, fast-paced failure scenarios
(visiting vehicles, on-board emergency response, software
transition issues, and serious system hardware failures) must be
pursued and verified by ISS Program managers and the
Engineering and Safety Technical Authorities.
Recommendation The ISS Program should institute a systematic process of O-10
R-20: monitoring water quality and chemistry aboard ISS to track
changes that can affect critical ISS systems including the EMU,
crew health, and multiple ISS Systems that use water and are
sensitive to its chemical makeup (The Oxygen Generation
System, The Water Processor Assembly, the Common Cabin Air
Assembly, etc.). This process should include consideration of
onboard monitoring capability. It should also include return of
any removed hardware to the ground for evaluation.
Recommendation The ISS Program should develop a system that allows high rate O-11
R-21: data telemetry to be received by ground teams during an EVA to
allow near instantaneous monitoring of critical system
parameters.
Recommendation The ISS Program should develop a flexible system that allows O-12
R-22: multiple short EMUs, as well as EMU components such as the
PLSS, to be launched or returned on multiple vehicles.
Recommendation The ISS Program and the EVA Project Office should put O-13
R-23: schedules and processes in place to ensure access to flight
hardware to the broader EVA community including the
Astronaut Office, MOD EVA, and S&MA personnel.
Recommendation The ISS Program and the EVA Project Office should require O-13
R-24: close out photos be taken of all hardware with the participation
of operations personnel to document the precise configuration of
what is flying as well as accurate configuration records
maintained and made available to real-time support personnel to
facilitate effective communication between the ground and crew
in flight.
Recommendation The ISS Program and the EVA Project Office should ensure that O-13
R-25: all procedures are validated on flight hardware if the procedure
requires a functioning system versus a fit check.
Recommendation For critical external tasks, the ISS Program should provide at 0
R-26: least one viable and proven dissimilar backup EVA capability
(known candidates include dexterous robotics or Russian EVA)
Recommendation With the help of MOD, the EVA Office should review all O-15
R-27: existing EVA knowledge databases and combine them into a set
of databases that are complete, accurate, kept up-to-date, and
easily accessible to the entire EVA community

147
 National Aeronautics and Space Administration EVA23Water Intrusion

Recommendation The ISS Program should ensure that the EMU Requirements O-15
R-28: Evolution Book is routinely updated to capture the maturing
design and design rationale of the EMU and include material
originally intended for the placeholder sections in the 1994
version.
Recommendation The ISS Program should ensure that the EVA community uses O-15
R-29: the EMU Requirements Evolution Book and the improved
knowledge capture databases, once developed, to improve
ground team training requirements throughout the EVA
community for better depth of EMU system knowledge and
attention to design and failure history.
Recommendation The Agency, Centers, and Programs should improve O-16
R-30: requirements for root cause determination and subsequent
training and provide the training for Engineering and Safety
personnel to better ensure root cause determination of critical
and reoccurring failures.
Recommendation MOD should provide integrated EVA sims with the possibility O-18
R-31: of ending the sim early. These sims must be scheduled for the
full duration, but allowed to end early if required by the actions
taken by the flight control team. Additionally, airlock ingress
and repress should be routinely included as part of simulations
that involve terminating an EVA with an EMU in an off nominal
configuration.
Recommendation MOD should review all procedures with a MCC step and O-19
R-32: verify that rationale exists to explain the required actions to be
taken by the flight control team if this step is reached.
Recommendation The ISS Program should ensure appropriate connectivity O-20
R-33: between all relevant parties who participate in EVA activities to
support real-time operations including talk/listen access to MCC
Audio Loops.
Recommendation MOD should strengthen training to emphasize the physiological O-22
R-34: effects of a rapid repress on the crew to aid in the decision
making process in real-time.
Recommendation The ISS Program and JSC EVA Office should improve technical O-23
R-35: and management coordination between their two organizations
and ensure that all strategic and tactical decisions that are made
by either organization are quickly and effectively understood,
and officially accepted by both.
Recommendation The government officials and contract managers must put in O-24
R-36: place expectations and create a board environment that allows
the EVA contractors to freely challenge technical decisions
made by the governing boards when appropriate and encourage
proactive participation.

148
 National Aeronautics and Space Administration EVA23Water Intrusion

Recommendation To reinforce the independence of safety and recognize the O-25

R-37: unique criticality of EVA in the safety community, consider
altering the ISS CSOs office to more closely mirror that of the
ISS Chief Engineers Office by creating a deputy CSO for EVA
position to more closely work with the EVA safety community
and help integrate them into the ISS Program and aid the CSOs
and Program Managers understanding of EVA risks in the
context of the ISS Program.
Recommendation Provide additional EVA training and integration activities to the O-25
R-38: MER Safety Officer position training syllabus.
Recommendation JSC Safety and Mission Assurance should institute a training O-26
R-39: program for all of its EVA personnel that includes a subset of
MOD EVA task and EVA systems training flows to gain the
requisite training on EVA hardware and tasks it is being used
on. This training should be supplemented by observing EMU
vacuum chamber runs, NBL runs, hardware reviews, and ground
testing both at SGT and UTAS Windsor Locks and studying the
EMU Requirements Evolution document should be mandatory.
Recommendation JSC Safety and Mission Assurance should routinely advocate for O-26
R-40: and lead the periodic review of FMEA/CILs and Hazard reports
and be intimately familiar with their content.
Recommendation ISS Program should augment, at least temporarily, MOD EVA O-27
R-41: personnel to allow the existing backlog of work to be completed
in a fairly short order by bringing on, through rotational
opportunities, personnel that can provide valuable technical
assistance that will not add to the training and certification
burden already faced by the organization.
Recommendation ISS Program should provide additional long term resources to O-27
R-42: augment current EVA community staffing to support the coming
increased frequency of ISS maintenance and contingency EVAs.
Recommendation The ISS Program must define The Roles and Responsibilities of O-28
R-43: the MER and the FCT to a level whereby each position (FCT
and MER) on either side clearly understands their role and the
role of their counterparts and mutual expectations must be
established and agreed to. As part of this effort, the Program
needs to reinforce the understanding that it is the FCT that is
authorized to accept risk on behalf of the Program in real-time
operations requiring best engineering judgment.
Recommendation The ISS Program must establish a protocol whereby whenever O-28
R-44: conflicts arise between the MER and FCT concerning roles and
responsibilities or one partys performance during a particular
event, the appropriate management from each side must meet to
discuss the conflict and revise the roles and responsibilities or
expectations accordingly.

149
 National Aeronautics and Space Administration EVA23Water Intrusion

Recommendation The ISS Program should develop proficiency requirements for O-28
R-45: MER Managers by event they are certified to support, as well as
on a time basis (e.g. annually) to maintain currency.
Recommendation The ISS Program should provide training to the MER Managers O-28
R-46: to deepen their systems and vehicle knowledge to ensure proper
subsystem and situational awareness during real-time
operations.
Recommendation The ISS Program should immediately modify the contractual O-28
R-47: clauses that may prevent the recommendations contained in this
report from being implemented within the contractor
community.
Recommendation NASA real-time operations community should work with the O-29
R-48: JSC Human Factors team to assess areas where human factors
processes can be better trained and implemented in operations
and develop specific training to reduce the impact of human
factors in future mishaps.
Recommendation The ISS Program should commission an independent study team O-30
R-49: to identify options to ensure an ISS EVA capability through
2028 that trades improvements to the current single fault tolerant
suit via options such as additional on-orbit diagnostics and
preventative/corrective maintainability, redesign to separate
water and vent loops, and/or implementation of an advanced
suit.

150
 National Aeronautics and Space Administration EVA23Water Intrusion

Appendix A: Human Factors Analysis

The MIB Human Factors (HF) Analysis focused on significant events on the timeline. It should be
noted that this analysis was only applied to real time ops and not to all of the many other human factors
associated with the precursors and processes that led to this incident. Using transcripts and interview
notes, each event was evaluated to identify unsafe acts, errors, and organizational challenges. These
factors were evaluated to determine whether they were causal or contributory to this event. This data
assisted the Board in performing fault tree analysis, helping create the event and causal factor tree,
drawing conclusions, and generating recommendations to reduce human error and mitigate the negative
consequence of human actions.
The analysis began by sequentially and systematically following the timeline, according to
significant events. Each event is outlined in a brief narrative. Next the Department of Defense (DoD)
Human Factors Analysis and Classification System (HFACS) was used to identify and evaluate HF
conditions that may have contributed to or caused each event leading to the mishap. HFACS was
developed by the DoD specifically to identify hazards and risks in the context of human error and mishap
investigation. It is based on four tiers of failures/conditions including acts, preconditions, supervision and
organizational influences with related categories and subcategories for each tier. The applicable HFACS
codes related to the four tiers are identified and detailed below each narrative, followed by a presentation
of the evidence gathered by this Board with the applicable analysis and sources. Note that because some
of the HFACS codes are attributed to more than one event, the subsequent analysis may cite some events
outside of their chronological order.

A.1 EVA 22 Water in the helmet

EVA 23 High Visibility Close Call (HVCC) involving water in the helmet begins with the final
portion of EVA 22, which occurred 1 week prior to EVA 23. At the end of a nominal EVA 22, the crew
returned to the Airlock. Just prior to entering the Airlock, EV1 was facing EV2 and did not see any signs
of water in EV2s helmet. The crewmembers then closed the Airlock hatch, conducted the nominal
repress process, and afterwards began doffing their suits. When EV2s helmet was removed, EV1 and IV
reported to the ground team that they found an unusually large amount of water inside EV2s helmet.
The crewmembers began examining EV2s helmet, neck ring, and T2/vent loop areas of the suit to
determine the source of water. They noticed an accumulation of water in and around the vent loop, and
began to focus on the Disposable In-suit Drink bag (DIDB) and the Liquid Cooling Ventilation Garment
(LCVG) as possible sources. EV1 attempted to open the bite valve of EV2s DIDB while squeezing the
bag and the crewmembers determined that EV2s drink bag was completely empty. EV1 asked EV2
whether he drank water from the drink bag, and EV2 indicated that he drank some, but didnt remember
how much. He recalled that during repress inside the Airlock, he might have pushed into the bite valve
with his chin resting against the valve while unintentionally pressing the bag against the Hard Upper
Torso (HUT) portion of the space suit. The crewmembers LCVG was not water-saturated, which
suggested that it was likely not the source for the water leak. Because EV1 did not see any water in EV2s
helmet prior to repress, EV1 concluded that the water must have leaked from EV2s drink bag while they
were in the Airlock. It accumulated around the back of EV2s head and completely saturated his
Communications (Comm) Cap. Additionally, there was a thin film of water all around EV2s head,
estimated to be about half a liter to a liter of water (which equates to roughly 1 to 2 pounds of water).
The ground team determined that no further investigation was needed, and instructed the crew to
replace the drink bag that they believed to be faulty. At the end of EVA 22, the entire team (including

A-1
 National Aeronautics and Space Administration EVA23Water Intrusion

EV2 it seems) accepted the conclusion of a DIDB failure and then began preparing for EVA 23
operations.
After EVA 22, no one suspected that a similar event would happen during EVA 23. The perception
that the DIDB malfunctioned during EVA 22 created a series of human factors issues during EVA 23 that
hindered the ability of the crew and ground to recognize that the water in the helmet could have been
caused by other sources. For example, throughout the EVA 22 troubleshooting process, evidence
indicates that the crew and ground both expected the source to be the DIDB (due to training and prior
experience). This caused them to focus their attention on that particular solution, and to settle prematurely
on identifying the DIDB as the source of water.
Additional circumstance was that the EVA 22 ground team was in the process of handing over to
the next shift while the EVA 22 troubleshooting was taking place, provided the opportunity for
distractions, missed information, and missed opportunities to ask questions that later would turn out to be
critical (e.g. the temperature of the water and how much EV2 had consumed during the EVA). The
following section explains some of the most relevant human factors in more detail.

A.1.1 PC 214 Response Set

Response Set is a factor when the individual(s) has a cognitive or mental framework of expectations that
predispose them to a certain course of action regardless of other cues.
Response Set was investigated and found contributory to the HVCC.
Evidence: Post HVCC interview data from MOD EVA personnel, Recording of EVA 22 CAPCOM
Loop

At the end of EVA 22, it was determined by EV1 and EV2 that a large amount of water
(estimated at one half to one liter) had leaked into EV2s helmet during the time from airlock
ingress to complete re-pressurization. It was noted that the comm cap was completely soaked
with water. There was also a description of a lot of water in both the helmet and the vent loop,
which delivers oxygen to the astronaut, via the T2 port.
After a brief discussion about EV2s head positioning during repress and a possible drink bag
leak due to inadvertent pressure on the bite valve, it was concluded that the water in the helmet
had come from EV2s drink bag.
Based on operator console loops, it appears this theory was first suggested by EV1, shortly after
the conclusion of EVA 22 and after EV2 had indicated that he thought he saw some water escape
past the bite valve on his drink bag.
After the conclusion of EVA 22, the EVA team perceived that all the water in EV2s helmet
probably came from EV2s drink bag and that the solution was to provide a new drink bag prior
to the next EVA, per routine.
This conclusion was accepted by the ground team based on their perception that the crew were in
the best position to understand the source of water.
The ISS crew tested the suspect DIDB after EVA 23 and found it to be functioning correctly with
no leaks.
The following factors, thus support the finding that the crew was predisposed to focus their
attention on the drink bag: EV2 indicated that he saw some water escape from his drink bag
during repress and thought his body position at that time could have induced a drink bag
malfunction. The water was identified in the helmet area (not in the lower portion of the suit),
removing suspicion from the LCVG. Training did not prepare the crew to deal with water leaking

A-2
 National Aeronautics and Space Administration EVA23Water Intrusion

from the PLSS. Finally, the amount of water that leaked into the helmet toward the end of EVA
22 could have been consistent with the amount of water contained in EV2s drink bag. The
ground team was also predisposed to focusing on the drink bag for reasons similar to that of the
crew. That includes prior training, limited knowledge regarding the PLSS as the source of water,
and misperceiving that the crew had actually established the source of the water.
There was no evidence, based on the EVA comm loops and post EVA debriefs, that any other
theories were seriously considered, discussed, or communicated immediately following the
conclusion of EVA 22.
The drink bag failure conclusion precluded the EVA team from identifying the EMU 3011 failure
early on, and they followed the course of action to fix issues with the drink bag, thereby missing
the opportunity to prevent a future mishap.
Finding:

There is a widespread understanding within the EVA and MOD community that drink bags leak
on-orbit, despite the fact that history has actually shown that this is not true.
The ground and flight control team were predisposed to considering drink bag as the source of the
water, excluding other possibilities, primarily because they lacked sufficient training to identify
the PLSS as another source.

A.1.2 OC003 Perceptions of Equipment

Perceptions of Equipment is a factor when over or under confidence in an aircraft, vehicle, device, system
or any other equipment creates an unsafe situation.
Evidence: Audio Logs, Post HVCC interview data from MOD EVA, CORE, SA and CB/Astronaut
Office personnel
Perceptions of Equipment was investigated and found contributory to the HVCC.

After EVA 22 ended, the crew concluded that the water which collected in EV2s helmet had
inadvertently leaked from his drink bag. Based on post HVCC interviews, it was determined that
one of the reasons that ground team members did not challenge the crews conclusion was
because there was a perception that the drink bags leaked. More specifically, the way in which the
drink bag was supposed to have leaked made sense according to its design
Finding:

There is a perception within the EVA community that drink bags leak.

A-3
 National Aeronautics and Space Administration EVA23Water Intrusion

A.1.3 PP106 Communicating Critical Information

Communicating critical information is a factor when known critical information was not provided to
appropriate individuals in an accurate or timely manner.
Communicating critical information regarding the EVA 22 Water in the Helmet event was investigated
and found contributory to the HVCC.
Evidence: Post HVCC interview data from EV1, MOD EVA, MER EVA, and CORE personnel

The type of critical information that would have been useful was discussed in interviews, but was
not initially (or adequately) communicated by EV2 to EV1 and the ground. It includes the amount
of water that EV2 consumed during EVA 22, the temperature of the water in the helmet prior to
suit doffing, and the location of the water relative to the vent loop.
o There is no evidence why this information was not communicated, but it is possible, if
not probable, that no one on the team considered the information to be critical at the time.
Multiple team members indicated for example, had they known the water was cold they
would have eliminated the DIDB as a source. Water temperature and the amount of water
would have been vital clues that might have led the EVA team to consider other sources
of water, possibly determining that EMU 3011 had failed in the latter stages of EVA 22.
Had that been determined, the EVA 23 HVCC might have been averted.
Finding:

Critical information about an EVA 22 water leak was not communicated real time, or in the
debrief between EV2/EV1 and the ground.

A.1.4 SI003 Local Training Issues/Programs

Local Training Issues/Programs are a factor when one-time or recurrent training programs, upgrade
programs, transition programs or any other local training is inadequate or unavailable (etc.) and this
creates an unsafe situation.
Evidence: Post HVCC interview data from CB/Astronaut Office personnel
Local training issues/programs were investigated and found contributory to the HVCC.

Crew and ground team training did not include information relating to failure modes of the PLSS
that could result in water intrusion into the helmet.
Finding:

Training did not adequately characterize the vent loop water leak which contributed to this
HVCC.

A-4
 National Aeronautics and Space Administration EVA23Water Intrusion

A.1.5 AE201 Risk Assessment During Operation

Risk Assessment During Operation is a factor when the individual fails to adequately evaluate the risks
associated with a particular course of action and this faulty evaluation leads to inappropriate decision and
subsequent unsafe situation. This failure occurs in real-time when formal risk-assessment procedures are
not possible.
Evidence: Post HVCC interview data from MOD EVA personnel, Discussion with EVA MIB Chair
and MOD personnel.
Risk Assessment during operation was investigated.

The operative phrase in this definition refers to real-time operations. As the situation unfolded at
the tail end of EVA 22, the dynamics were such that lengthy meetings were not possible. In the
absence of a formal risk assessment, team members relied on their own individual techniques and
parameters for assessing risk.
During interviews, more than one team member indicated (in hindsight) that they wished they had
called a time-out. The feeling of being pressured for time outweighed their perception of need
for further risk assessment. Doing back to back EVAs one week apart left little time to
accomplish all the preparation work necessary for EVA 23.
There was also a perception that if the question concerning the source not being the drink bag was
raised, it would invoke a fairly resource intensive and potentially cumbersome process involving
Engineering and Safety for what most felt would likely turn out to be a non-issue. This would
have an impact on EVA 23 preparations. In hindsight, however, it is now apparent that EVA 23
should not have commenced until the EVA 22 issue had undergone a more adequate evaluation.
That is not to say that a lengthy formal risk assessment was required (that may, or may not be the
case), just that the EVA 22 water leak deserved a more refined assessment of risk. Had that been
done, the EVA 23 HVCC might not have occurred.
Finding:

The ground team did not assess the risks of alternative sources of water.
The ground team did not assess the risk (to the human) of having a large amount of water in the
helmet.

A.1.6 Organizational Influences

Organizational Influences are factors in a mishap if the communications, actions, omissions, or policies of
upper level management directly or indirectly affect supervisory practices, conditions, or actions of the
operator(s) and result in a system failure, human error, or an unsafe situation.
Evidence: Post HVCC interview data from MOD EVA personnel, Discussion with EVA MIB Chair
and MOD Manager.
Informational Resources/Support and Organizational Process of the EVA 22 event were investigated and
found to be contributory to the HVCC.

Some MOD team members perceived that the organizational process of running the EVA 22
water leak to ground would have slowed down operations by engaging in a cumbersome,
methodical, time-intensive process of engineering analysis.

A-5
 National Aeronautics and Space Administration EVA23Water Intrusion

Based on interviews, there is a general perception among teams that the ISS Program applies
pressure to maximize crew time, at the expense of other mission requirements. This has led lower
level teams to make risk decisions for the Program, assuming that they wont be allowed to use
crew time to accomplish other tasks that may be deemed necessary.
This is a high level organizational weakness and is not meant to reflect adversely on any one
organization.
Finding:

Organizational influences led to an acceptance of risk by workers below the Program

management level.

A.1.7 OP001 Ops Tempo/Workload

Ops Tempo/Workload is a factor when the pace of deployments, workload, additional duties, off-duty
education, PME, or other workload-inducing condition of an individual or unit creates an unsafe situation.
Evidence: Post HVCC interview data from MOD EVA personnel, Discussion with EVA MIB Chair
and MOD Manager.
Ops Tempo/Workload during EV 22 event was considered, and found contributory to the HVCC.

Discovery and troubleshooting of the EVA 22 water leak (in which approximately to 1 liter of
water leaked into EV2s helmet during repress) occurred around the time of a shift hand-over, a
time during which it is not uncommon to have distractions (note that this is an additional Human
Factor captured in PC106) that can skew ones attention away from certain details. For example,
if a hand-over conversation is going on between two controllers, the possibility exists to miss a
critical piece of information being discussed on the comm loops (another Human Factor captured
in PC102 Channelized Attention).
The team knew about the water in the helmet but they thought they had a reasonable explanation,
and there is interview evidence to indicate that a high Ops Tempo encouraged the team to accept
the established drink bag explanation.
During the interview process, there were expressions to the effect of having wished the team had
slowed down, to consider things in more detail, while on the other hand they thought that if they
had pursued other causes, that process of running the options to ground would have caused a
significant slowdown to the process of preparing for EVA 23, a challenge even under normal
circumstances.
Finding:

Ground team had started a shift hand-over when the event occurred.
Due to Ops Tempo, preparations for EVA 23 became the main focus after EVA 22 was
concluded.

A-6
 National Aeronautics and Space Administration EVA23Water Intrusion

A.1.8 PC405 Technical/Procedural Knowledge

Technical/Procedural Knowledge is a factor when an individual was adequately exposed to the
information needed to perform the mission element but did not absorb it. Lack of knowledge implies no
deficiency in the training program, but rather the failure of the individual to absorb or retain the
information.
Technical knowledge was investigated, but not found contributory to the HVCC.
Evidence: EVA 22 audio loops

Because personnel were facing a new failure mode that was not well described in previous
training, the steps leading up to an unsafe condition are adequately and better characterized by
SI003 and other HF codes listed above.
Finding:

The team possessed insufficient knowledge about possible water leaks originating in the vent
loop and areas upstream, however this lack of knowledge was due to insufficient training. It was
not a failure to absorb or retain knowledge and therefore technical knowledge was not
contributory, according to the preceding definition.

A-7
 National Aeronautics and Space Administration EVA23Water Intrusion

A.2 Response to EVA 23 Water Leak

The next significant event analyzed with HFACS involving water in the helmet continued with the
CO2 Sensor failure and subsequent assessment of water in EV2s helmet during EVA 23.
As the EVA progressed, EV2s CO2 sensor started to read high. The ground team initially
wondered whether EV2 might also be working hard and demonstrating an increased met rate. The ground
team, along with consultants at UTAS, also considered that EV2 might have inadvertently inserted a used
METOX canister into his EMU. Soon after, EV2s CO2 sensor quickly failed off scale high at EVA 23
Phased Elapsed Time (PET) of 38 minutes.
Even though the CO2 sensor is not a water condensation monitor its failure mode during past
EVAs has been tied to the presence of increased water condensation in the vent loop: this phenomenon is
associated with possible sublimator carryover, when a space suit thermostat is suddenly set from hot to
cold. However, these failures were not viewed as symptomatic of significant water leaks because the
process of moisture accumulation a) traditionally happened after several hours of EVA and b) never
resulted in an unsafe amount of water leaking into the vent loop. In the case of EVA 23, the ground team
surmised that the sensor failed due to excessive water condensation in the vent loop. While some ground
team members considered the early failure of this sensor as unusual, it is unclear whether any follow-on
discussions happened to ascertain whether the early CO2 sensor failure required further scrutiny beyond
the assumption that it was associated with nominal water condensation in the vent loop.
Six minutes after the CO2 sensor failed, EV2 called to the ground over the space to ground loop
saying that he felt a lot of water near the back of his head, specifically indicating that he didnt think it
was his drink bag. Because the DIDB had been erroneously identified as the source of water in the helmet
during the previous EVA, the ground team began to query EV2 in order to identify the source of the
water, evaluating the DIDB, LCVG, and Maximum Absorbency Garment (MAG). After ruling out the
LCVG and the MAG, EV2 seemed inclined to reconsider that the water could have come from the DIDB
after all. Based on the available audio logs and post incident interviews, it is clear that the ground team
did not adequately consider or diagnose all potential sources for water entering into the helmet.
After a few minutes of continuing with his scheduled EVA tasks, EV2 informed the ground team
that the amount of water behind his head was increasing, which was cause for concern with the MOD
EVA lead. Still unable to explain the cause of the leak, the MOD EVA lead expressed concern that EV2
could experience eye irritation due to water mixing with the anti-fog coating used to keep the visor clear.
The MOD EVA lead queried the team about terminating the EVA, but at that point, the MER team did
not see the immediate need to terminate solely because of the anti-fog concern. The team continued to
focus on further task guidance to EV1 and EV2.
Shortly afterwards, EV2 reiterated to the ground team that the leak was not from the DIDB. By this
time, EV2 had finished drinking the remainder of his DIDB water, possibly as a way either to stop the
leak from increasing or to eliminate suspicion of the DIDB as the source of the water leak. While the
ground team still seemed to be convinced that the DIDB was the source of the water, they were growing
more concerned about the increasing accumulation of water, and requested EV1 to take a close look into
EV2s helmet. After his visual inspection into EV2s helmet, EV1 called down saying that there was
indeed a large quantity of water (roughly about half a liter) around the back of EV2s head along with a
blob of water, about the third of the size of a baseball, floating near his right temple. The ground team
started to understand that EV2s condition was more serious than they thought, yet they still did not
understand that the water was leaking into the helmet from the vent loop and not the DIDB.
Ten minutes later, an hour and seven minutes into the EVA (23 minutes after the initial indication
of water in the helmet), the ground team terminated the EVA, not yet knowing the source of the
increasing water. Even though the team had not accurately identified the cause(s) of the water leak, they
terminated the EVA thus allowing enough time for EV2 to return to safety.

A-8
 National Aeronautics and Space Administration EVA23Water Intrusion

The background and circumstances of the ground teams response to the early CO2 sensor failure
created a series of human factors issues that hindered the ability of the crew and ground to recognize that
the water in the helmet could have been caused by a much more significant source than the DIDB.
Identifying these issues involves assessing available evidence to understand how and why the ground
team and the crew team struggled to diagnose systematically the source of the water leak and misjudged
the potential severity of the overall situation. The ground team did not realize that there might be a
problem with the suit itself, and therefore selected a course of action focused on mitigating leaks from the
DIDB at the expense of investigating potential issues with the suit. The team maintained a narrow focus
(channelized attention) on the problem without considering all the potential causes for an off-nominal
amount of water in the helmet. The crew and the ground teams were both unaware of all potential causes
of water leaks, especially water coming from the vent loop. The CO2 failure, water leaking into the
helmet, and the potential anti-fog eye irritation all happened around the same time, contributing to
cognitive oversaturation to particular individuals. Additionally, evidence shows that the ground team and
the crewmembers were accustomed to seeing CO2 sensors fail, and experiencing limited and manageable
amounts of water in the helmet. The following section addresses some of the most relevant human factors
of this event in greater detail.

A-9
 National Aeronautics and Space Administration EVA23Water Intrusion

A.2.1 PC214 Response Set

Response Set is a factor when the individual has a cognitive or mental framework of expectations that
predispose them to a certain course of action regardless of other cues.
Evidence: Audio Logs, Post HVCC interview data from MOD EVA, CORE, SA and CB/Astronaut
Office personnel
Response Set was investigated and found contributory to the HVCC.

The CO2 sensor sometimes fails during EVA ops, a fact that is well documented. The ground
teams mental framework of expectations related to failures with the CO2 sensor determined their
course of action to address known issues. When the sensor fails during real-time ops, the response
dictated by flight rule has the crew monitor their symptoms, the ground request an assessment
every 30 min., and the METOX canister de-rated to provide additional CO2 scrubbing margin.
Otherwise the EVA may proceed. This has happened often enough in the past that it is almost
considered nominal. This ops nominal characterization of an off-nominal event is sometimes
referred to as normalization of deviance.
In the case of EVA 23, the failure of the CO2 sensor was not nominal. It happened earlier than
usual, and because in hindsight, it likely provided an early indication of the actual failure mode,
which was related to water leaking into the helmet via the vent loop/T2 port. The reason why the
relationship between water in the vent loop and water in the helmet was not fully understood
early on in the EVA will be discussed in several subsequent sections, including
OC003/Perceptions of Equipment.
During previous EVAs, small amounts of water intrusion into the helmet had been attributed to
sublimator (slurper) carry-over. Over time, this led to a normalization of deviance, predisposing
the team to consider that water in the helmet was acceptable.
The team immediately began to question the source of the water and the one question that was
repeated by multiple team members was: is the water coming from his drink bag? This line of
questioning was reinforced by the perceptions of equipment and the mental framework of
expectations that arose from EVA 22, all of which constitute the teams response set.
Even though the team never fully understood the source of the water in EV2s helmet until after
the end of EVA 23, the teams combined efforts allowed them to recognize the general danger of
the situation, resulting in a decision to terminate the EVA 23 minutes after EV2 first called out
that he had water in his helmet, which was a correct response in that situation.
Finding:

The CO2 sensor failed early in the EVA.

The early failure of the CO2 sensor was an off nominal event that was normalized.
The team did not identify the vent loop as a possible cause of the water leak.

A-10
 National Aeronautics and Space Administration EVA23Water Intrusion

A.2.2 OC003 Perceptions of Equipment

Perceptions of Equipment is a factor when over or under confidence in an aircraft, vehicle, device, system
or any other equipment creates an unsafe situation.
Evidence: Audio Logs, Post HVCC interview data from MOD EVA, CORE, SA and CB/Astronaut
Office personnel
Perceptions of Equipment was investigated and found contributory to the HVCC.

At the end of EVA 22 the crew and ground team had concluded that the water which had leaked
into EV2s helmet had come from his drink bag. This, as well as the perception that drink bags
on-orbit have had a history of leaking primed the EVA 23 team with the perception that EV2s
drink bag had leaked during EVA 22, and might have been leaking again during EVA 23.
The faulty drink bag perception that existed prior to EVA 22, during EVA 22, and at the
beginning of EVA 23 persisted, to some degree, throughout most of EVA 23.
The severity of the leak was underestimated, because of the perception that the leak was caused
by the drink bag. The drink bag holds about a liter of water, so therefore any water accumulation
would be self-limiting.
The CO2 sensor was reported failed at EVA 23 Phased Elapsed Time (PET) of 38 minutes. The
CO2 sensor is sensitive to moisture in the vent loop and will fail off-scale high when too much
water enters its sensing orifice. Although not intended for this purpose it currently serves as a
detector of excess moisture in the vent loop. In the past, it characteristically fails late in an EVA.
At 38 min PET, it was noted by some as one of the earliest occurrences of this failure; however,
the perception that the CO2 sensor was unreliable precluded the ground team from carefully
considering the potential implications.
Some of the ground team members were confident that the failure of the CO2 sensor could be
explained within the normal operating envelope of the space suit. Based on interviews regarding
past experiences with the CO2 sensor, it was understood that moisture or water in the vent loop
could occur early on in an EVA, when the EMU temperature control valve is changed from very
warm to very cold.
In the case of sublimator carryover, the amount of water that can accumulate in the helmet is
limited, and not considered unsafe.
Based on interviews, it is known that the events of EVA 22 were on the mind of EV2, when he
surmised that he may have caused the water to leak in his helmet via his DIDB, and he wanted to
be sure that it didnt happen again.
Finding:

The CO2 sensor failed at PET 38 minutes, due to excess moisture or water in the vent loop.
The ground team had preconceived expectations regarding the unreliability of the CO2 sensor.
EV2 identified water in his helmet as not coming from his DIDB, at PET 44 minutes.
The crew and ground team had preconceived expectations regarding the performance of the drink
bag.

A-11
 National Aeronautics and Space Administration EVA23Water Intrusion

A.2.3 PC102 Channelized Attention

Channelized Attention is a factor when the individual is focusing all conscious attention on a limited
number of environmental cues to the exclusion of others of a subjectively equal or higher or more
immediate priority, leading to an unsafe situation. It may be described as a tight focus of attention that
leads to the exclusion of comprehensive situational information.
Evidence: RB interview, EVA 23 S/G and CAPCOM loops
Channelized Attention was investigated and found contributory to the HVCC.

The team primarily focused on EV2s DIDB as the possible source of water in his helmet. Other
suggestions included accumulation of sweat and leakage from the LCVG, but both were ruled out
fairly quickly.
Although the CO2 detector is not a moisture detector, its failure is commonly associated with
moisture in the vent loop. The attention given to the CO2 detector early on was because it initially
indicated rapidly increasing levels of CO2. When it was noted that EV2s metabolic rate was not
unusually high, part of the team wondered if a used METOX canister was accidentally inserted
into the EMU. This was only a brief supposition, because a few minutes later, the CO2 sensor
failed completely. It should be noted that the way the CO2 sensor failed (off-scale high voltage)
was indicative of excess moisture, which had never previously caused a serious hazard.
The sudden sensor failure led some of the team to believe that it failed due to a nominal
accumulation of water, or moisture in the vent loop. Since nominal water carryover only results in
a limited/manageable amount of water in the helmet, the significance of the CO2 sensor failure
was quickly disregarded, despite the fact that this type of failure almost always occurred near the
end of a long EVA.
Channelization may have prevented the team from continuing to ask questions to come up with a
different answer or ask new and more specific questions that would have pointed to something
other than the drink bag.
Finding:

No one on the team recognized the relationship between the early failure of the CO2 sensor and
an abnormally large amount of water in the vent loop because they channelized on the drink bag
as the source and missed the potential cues from the early CO2 sensor failure.

A-12
 National Aeronautics and Space Administration EVA23Water Intrusion

A.2.4 PP106 Communicating Critical Information

Communicating critical information is a factor when known critical information was not provided to
appropriate individuals in an accurate or timely manner.
Evidence: Audio Logs, Post HVCC interview data from MOD EVA and CORE personnel
Communicating Critical Information was investigated and found contributory to the HVCC.

Good communication is dependent on at least three factors: transmission, reception, and

comprehension.
During EVA 23 there was a failure to receive critical information that had been transmitted. The
first example of this was when EV2 called down over the S/G loop (at PET 44 minutes) to say
that he had a lot of water at the back of his head and he didnt think it was from his drink bag.
The ground team did not acknowledge that EV2 believed that the water was not coming from his
drink bag. EV2s call at PET 44 minutes was clearly transmitted. It was also heard, but it seems
that only the first half of the transmission was understood. The team did not comprehend the
critical information that the water had not come from his DIDB.
The amount of activity and communication traffic occurring around the time of the call reporting
the water likely contributed to a lack of comprehension of the call.
In reaction to EV2s first call, the team questioned EV2 as to whether or not the water had come
from his DIDB. This might have caused EV2 to second-guess his first assumption, so he made a
second call soon after his first, at PET 49 minutes, to state that the DIDB may be suspect. Five
minutes after the second call, EV2 made a third call, at PET 54 minutes, in which he emphatically
states that the leak was not from the water bag. At this time, most of the team was heavily
absorbed by the water in EV2s helmet, but they did not clearly understand how much water was
there, if it was increasing in quantity, or what temperature it was. It is worth noting that water
temperature was probably not discussed by the team, because they were not focusing on the PLSS
as the source of water. This was a failure mode that had not been adequately covered in previous
training.
The comm loops were very busy with team members trying to resolve the problem, and there
were many conversations occurring off the loops as well. This may explain why some of this
critical information remained unclear to most of the team members and certainly to Flight, who,
at PET 57 minutes, states sounds like the water source was DIDB. This was in contradiction to
what EV2 stated three minutes earlier, in which he clearly transmitted that the water was not from
his DIDB.
There was a breakdown in communicating critical information, despite no failure in the ability to
make clear voice transmissions over the communication loops at that time.
Finding:

There were multiple instances during which critical information was not adequately
communicated.
There were multiple causes for the inadequate communication of critical information.
Several times the ground team had to ask the crew to repeat themselves, because critical
communications were missed, or stepped on.

A-13
 National Aeronautics and Space Administration EVA23Water Intrusion

A.2.5 PC103 Cognitive Task Oversaturation

Cognitive Task Oversaturation is a factor when the quantity of information an individual must process
exceeds their cognitive or mental resources in the amount of time available to process the information.
Evidence: Audio Logs, Post HVCC interview data from MOD EVA personnel
Cognitive Task Oversaturation was investigated and found contributory to the HVCC.

Emergencies which are not covered by training have the potential to open up a Pandoras Box of
technical resources and information that must be deciphered in a very short amount of time. This
plethora of information that the individual must process may exceed their cognitive resources in a
limited amount of time. In reviewing multiple transcripts and interviews from several team
members, it was apparent that individuals were cognitively task oversaturated by the events of
EVA 23.
In addition to processing technical information, the team members also had to process verbal
information, which in itself can lead to cognitive oversaturation if an individual is simultaneously
listening to multiple conversations. From interviews, the MIB learned that due to the multiple
communications going on at the same time, some team members didnt have sufficient time to
work through the suspicion that the DIDB was not the source of water.
There are many tasks required of controllers at each position within mission control. They are
mentally engaged in/directing choreographed EVA activities, dealing with off-nominal events,
communicating on the comm loops, accessing flight rules, referencing flight notes, and, at times,
navigating a host of other informational databases. Multi-tasking of this nature can lead to
cognitive task over-saturation.
Finding:

Various human factors, combined with the inability to rapidly determine the cause of water
leaking into the helmet allowed members of the team to become cognitively task oversaturated.
Some ground team members felt that they were more cognitively oversaturated during the EVA
than they had experienced during simulation training.

A-14
 National Aeronautics and Space Administration EVA23Water Intrusion

A.2.6 SI003 Local Training Issues/Programs

Local Training Issues/Programs are a factor when one-time or recurrent training programs, upgrade
programs, transition programs or any other local training is inadequate or unavailable (etc.) and this
creates an unsafe situation.
Evidence: Audio Logs, Post HVCC interview data from MOD EVA personnel
Training was investigated and found contributory to the HVCC.

Astronauts, pilots, controllers and others who deal with real-time emergencies undergo years of
rigorous training. The information they receive is gathered from records and data that have been
compiled, processed and eventually put into training manuals. The original products (more
voluminous/comprehensive) are then shelved, while the final cut and sort is put in electronic
format, for all to learn and reference when needed. The quality of the final training product is
highly dependent upon the thoroughness of the process by which the source information was
processed. FMEA/CILs ineffectively addressed issues with water in the helmet; therefore that
information was not incorporated into ISS/EVA training.
There is little doubt that the amount of critical information (information vital for safe operations)
that team members must learn has the potential to be overwhelming. That is why astronauts and
controllers alike practice one emergency drill after another. Procedures to handle emergencies are
developed years in advance. They are ingrained, until individual responses become automatic,
which facilitates the management of overwhelming tasks. The absence of sufficient training, or
experience in a particular failure mode, raises the possibility of cognitive task oversaturation.
Finding:

Crew and ground team training did not include this failure mode of water intrusion into the
helmet.

A-15
 National Aeronautics and Space Administration EVA23Water Intrusion

A.3 EVA 23 Translation to the Airlock Event

EV2 first called the ground to report water accumulating in his helmet PET 44 minutes into EVA
23. In the following 10 minutes, the ground team tried to get additional information from EV1 and EV2 in
order to try to ascertain the source of the water leak. On the Space to Ground loop, EV1 reported seeing
too much water in EV2s helmet, and EV2 reported that the water was increasing. After 5 to 10 minutes
of discussion, the ground team decided to terminate, making the call 23 minutes after EV1 first reported
the water. They instructed EV2 to return to the Airlock and for EV1 to secure the hardware outside and
return to the Airlock.
It took about 5 minutes for EV2 to translate back to the Airlock. During that time, EV2 expressed
disappointment with having to end the EVA too soon. From the interviews, it is clear that EV2 tried to
communicate during his translation to the Airlock; the ground team inquired about EV2s status while
translating, but received no response until EV2 reached the Airlock. During post-mission interviews, it
was discovered that EV2 had experienced loss of comm issues during this timeframe. EV1 arrived at the
airlock 2 minutes later, closed the hatch, and began repress. As repress commenced, it became clear that
EV2 was unable to hear or speak well, so at one point the crewmembers communicated by squeezing each
others hands. The ground team reviewed the repress procedures and opted to expedite the process, based
on their concerns for EV2s safety with the increasing water. After the repress was completed, IV helped
EV2 and then EV1 doff their suits. When they removed EV2s helmet, they noticed that his head and face
were lined with a thin film of water: the water was over his eyes and near his nose and mouth. His comm
cap was completely saturated with water, and it had filled his ears. There was also a large of amount of
water pooling near the back of his helmet, around the T2 vent port.
During the post EVA 23 debrief a few hours later, EV2 reported losing visibility as soon as he
started translating to the Airlock following the call to terminate. He recalled using the safety tether as a
guide to get back to the Airlock. EV2 also confirmed that he had comm issues that began early on during
his translation to the Airlock; initially he had been able to hear the ground and EV1, but wasnt receiving
responses. Once he was inside the Airlock, he wasnt able to hear even those communications. This
information surprised the ground team. During interviews, team members expressed surprise about the
amount of water in EV2s helmet and how it impacted his ability to see, communicate, and breathe. EV1
indicated in a post-flight interview with the MIB that he had a strong sense that EV2 was having
difficulty navigating to the Airlock and he wished he had been there to help him. While the team was
concerned about crew safety, specific concerns regarding EV2s ability to get back to the Airlock on his
own were not discussed, most likely because neither EV1 nor the ground team was aware of the severity
of the situation. This information was not communicated real-time, possibly because EV2s comm cap
was not working properly.
The following human factors issues were considered during EV2s translation to the Airlock, and
will be discussed in detail in the following section: conditions that limited EV2s visibility;
communication equipment issues that prevented effective communication between EV2, the ground team,
and EV1; personality styles that may have impeded communications; inadequate risk assessment; and
training issues that may have contributed to the lack of knowledge that made it difficult for both teams to
share and request information that would have helped them understand the severity of the situation or
evaluate the risks that EV2 could become incapacitated because of the situation.
Its important to emphasize the fact that the ground team ultimately made the right decision to
terminate the EVA when they did. EV2 was able to return to the Airlock with enough time to complete
the expedited repress. If the ground team had taken additional time during EVA 23 to diagnose the source
of EV2s water, the outcome might have been catastrophic. While the decision to not spend more time
diagnosing the source of the water leak during EVA 23 prevented a catastrophic situation, in retrospect it
becomes apparent (as explained in Appendix A.1) if a more thorough diagnosis of the leak that occurred
during EVA 22 might have prevented this life-threatening situation. The following human factors analysis

A-16
 National Aeronautics and Space Administration EVA23Water Intrusion

section details the human factors issues that contributed to the specific events that created a life-
threatening situation as EV2 returned to the Airlock.

A.3.1 PE203 Visibility Restrictions

Visibility Restrictions are a factor when the lighting system, windshield / windscreen / canopy design, or
other obstructions prevent necessary visibility and create an unsafe situation. This includes glare or
reflections on the canopy / windscreen / windshield. Visibility restrictions due to weather or
environmental conditions are captured under PE101 or PE102.
Visibility Restrictions was investigated and found contributory to the HVCC.
Evidence: Transcripts from Audio Logs, Post EVA 23 Crew Debrief, Expedition 35/36 Post Flight
Debrief, Post HVCC interview data from MOD EVA, CORE, SA and CB/Astronaut
Office personnel

Shortly before the call to terminate, EV1 looked through the face shield of EV2s helmet and
noticed a large amount of water pooling around his eyes, ears, and nose, as well as blobs of water
on the face shield.
During portions of the EVA, EV2s vision was degraded due to both face shield obscurations and
water in his eyes. EV2 was disoriented during his translation to the airlock and his visibility
conditions due to water in the helmet created an unsafe situation.
While visibility restrictions alone did not lead to the HVCC, they impacted safety, making them a
contributing factor.
Finding:

EV2s vision was impaired by water in his helmet.

EV2 was unable to remove the water from his eyes and his face shield.

A-17
 National Aeronautics and Space Administration EVA23Water Intrusion

A.3.2 PE102 Vision Restricted by Meteorological Conditions

Vision Restricted by Meteorological Conditions is a factor when weather, haze, or darkness restricted the
vision of the individual to a point where normal duties were affected.
Vision Restricted by Meteorological Conditions was investigated and found contributory to the HVCC.
Evidence: Transcripts from Audio Logs, Post EVA 23 Crew Debrief, Post HVCC interview data
from MOD EVA, CORE, SA and CB/Astronaut Office personnel

After the call to terminate, sunset occurred while EV2 began his translation back to the airlock.
During times of darkness, greater caution is required when translating around the space station.
There are some objects/obstacles which impede movement and others which can cause damage to
the space suit, such as cuts to gloves or other material.
EV2 was disoriented during his translation to the airlock and his visibility conditions due to water
in the helmet created an unsafe situation. Poor meteorological conditions (e.g. darkness)
contributed to an already unsafe situation.
While visibility restrictions alone did not lead to the HVCC, they could have led to undesired
outcomes, making them a contributing factor.
Finding:

EV2 had more difficulty than usual translating to the airlock at night.

A-18
 National Aeronautics and Space Administration EVA23Water Intrusion

A.3.3 PE208 Communications Equipment

Communications - Equipment is a factor when comm. equipment is inadequate or unavailable to support
mission demands (i.e. aircraft/vehicle with no intercom). This includes electronically or physically
blocked transmissions. Communications can be voice, data or multi-sensory.
Communications Equipment was investigated and was found contributory to the HVCC.
Evidence: Transcripts from Audio Logs, Post EVA 23 Crew Debrief, Expedition 35/36 Post Flight
Debrief, Post HVCC interview data from MOD EVA, CORE, SA and CB/Astronaut
Office personnel and MIB Interviews

During the post EVA debrief, EV2 indicated he was unable to communicate because his comm
cap was saturated with water.
EV2s restricted ability to communicate contributed to the grounds limited awareness of the
severity of the situation.
During translation to the airlock, EV2 indicated that he was making calls in the blind that were
not heard by the ground team.
While in the airlock, EV1 admitted that he considered going to abort procedures because he was
extremely concerned for the condition of EV2 when he was unable to get a verbal response from
him. It was only when they communicated through hand squeezing that EV2 decided to continue
down the path of an expedited termination, which fortunately had a safe outcome.
Finding:

EV2 was unable to communicate due to water saturating the comm cap.

A.3.4 PP106 Communicating Critical Information

Communicating critical information is a factor when known critical information was not provided to
appropriate individuals in an accurate or timely manner.
Communicating Critical Information was investigated and found contributory to the HVCC.
Evidence: Transcripts from Audio Logs, Post EVA 23 Crew Debrief, Expedition 35/36 Post Flight
Debrief, Post HVCC interview data from MOD EVA personnel

The ground team realized that there was a significant disparity between their real-time
understanding of the severity of the situation surrounding the water in EV2s helmet and the
details of the event conveyed by EV2, during his post EVA 23 debrief. The sentiment among
ground team members was that had they known in real-time what was conveyed during the post
EVA debrief, they would have been more concerned for EV2s safety.
During the post EVA 23 debrief, EV2 indicated that translating to the Airlock was not a simple
task due to the challenges presented by darkness and the water in his helmet. He was dealing with
the challenge of getting himself to the Airlock both quickly and safely while unable to
communicate due to equipment failure.
Finding:

The ground team did not understand the amount of water in EV2s helmet.

A-19
 National Aeronautics and Space Administration EVA23Water Intrusion

A.3.5 AE201 Risk Assessment During Operation

Risk Assessment During Operation is a factor when the individual fails to adequately evaluate the risks
associated with a particular course of action and this faulty evaluation leads to inappropriate decision and
subsequent unsafe situation. This failure occurs in real-time when formal risk-assessment procedures are
not possible.
Risk Assessment was investigated and was found contributory to the HVCC.
Evidence: Transcripts from Audio Logs, Post EVA 23 Crew Debrief, Expedition 35/36 Post Flight
Debrief, Post HVCC interview data from MOD EVA and CORE personnel

EVA 23 was terminated 1 hour and 7 minutes after it began. A call to terminate has a specific
meaning, in which crewmembers may have the option to cleanup/safe their work sites before
returning to the Airlock. An abort is more serious and involves immediately returning to the
airlock.
The team elected to terminate with EV2 returning to the airlock alone based on their overall
assessment of the situation; however, from the post flight debrief transcript, it was apparent that
(at the time of the termination) the team did not fully understand the severity of the water leak.
This could have led the team to underestimate the risk of sending EV2 back to the Airlock alone.
Additional risk exposure that the team could have considered was the aspiration of water, failure
of comm equipment, and impaired visibility; visibility was impaired because it was night during
EV2s translation to the airlock and he had water in his eyes.
While enhanced risk assessment would not by itself have prevented the HVCC (because most
information about EV2s condition during translation was learned after the fact), incomplete risk
assessment in future scenarios could lead to undesired outcomes, thus making risk assessment a
contributing factor.
Finding:

It was not possible to completely assess the risks of EV2 translating alone from the work site to
the airlock.

A-20
 National Aeronautics and Space Administration EVA23Water Intrusion

A.3.6 SI003 Local Training Issues/Programs

Local Training Issues/Programs are a factor when one-time or recurrent training programs, upgrade
programs, transition programs or any other local training is inadequate or unavailable and this creates an
unsafe situation.
Local Training was investigated and found contributory to the HVCC.
Evidence: Transcripts from Audio Logs, Post EVA 23 Crew Debrief, Expedition 35/36 Post Flight
Debrief, Post HVCC interview data from MOD EVA personnel

The approximate spare volume inside a space helmet is 12 liters empty, 8 liters when filled with a
head and com cap. With so much empty space, it may not seem intuitive that by adding as little as
one liter of water, an astronaut could aspirate water and drown.
Originally, engineers thought that water leaking into the helmet in zero gravity would stream
down the front of the visor, like rain drops on a windshield. From there, the drops would follow
the flow of air, eventually working their way into the body of the suit.
What was learned by this accident was that water from the vent loop tends to collect around the
T2 port, which is near the back of the astronauts head.
Another lesson learned was that surface tension causes that water to remain in that area, sticking
to the astronauts head. As more water accumulates, it gradually works its way forward, covering
the ears, then the nose, and eventually the mouth, until the airway risks being compromised.
There is a purge valve on the side of the helmet, but water does not readily migrate to the valve
when open. Water must be moved there by the Astronaut for it to be evacuated from the helmet.
Finding:

The manner in which water in the helmet would behave in zero gravity was neither intuitive, nor
understood by the team members.

A-21
 National Aeronautics and Space Administration EVA23Water Intrusion

A.3.7 PC205 Personality Style

Personality style is a factor when the individuals personal interaction with others creates an unsafe
situation. Examples are authoritarian, over-conservative, impulsive, invulnerable, submissive or other
personality traits that result in degraded crew performance.
Personality style of the crew members was investigated and not found contributory to the HVCC.
Evidence: Transcripts from Audio Logs, Post EVA 23 Crew Debrief, Expedition 35/36 Post Flight
Debrief, Post HVCC interview data from MOD EVA personnel

There is no evidence that personality style degraded crew performance, or had any negative
bearing on the events of EVA 23.
However, to say that personality style was non-contributory could be misleading to some. For
example, it appears that the personalities of the ISS crew and all the ground team members had an
overwhelmingly positive effect on the outcome of events.
Maintaining positive attitudes, being a team player and remaining calm under stress are all
essential traits for those who deal with the complexities of space flight.
EV2 managed to remain calm and focused throughout the EVA despite being at risk. He
interacted with the team such that they arrived at the unanimous decision to terminate, allowing
enough time to complete a successful recovery.
Finding:

The crew of EVA 23 was successfully recovered, without injuries.

A-22
 National Aeronautics and Space Administration EVA23Water Intrusion

A.4 Secondary Undesired Outcome Event

After the fan in the water separator pump was flooded, the ground team submitted a procedure for drying
out EMU 3011s Vent Loop. The transmitted procedure was not fully vetted on the ground, and once
implemented aboard ISS, caused secondary undesired outcome Inadvertent activation of EMU 3011
SOP during Vent Loop Wet/Dry Vacuum and Dryout activity (SUO) (see Section 2.5).

A.4.1 AE206 Decision-Making During Operation

Decision-Making During Operation is a factor when the individual through faulty logic selects the wrong
course of action in a time-constrained environment.
Decision-Making During Operation was investigated and found contributory toward inadvertently
activating the SOP.
Evidence: Post HVCC interview data from MOD EVA & UTAS personnel

After EVA 23 was terminated, the EVA ground team began troubleshooting the water leak
problem with EMU 3011. In the process of narrowing down the source of water, the ground team,
working with the ISS crew, reintroduced water into 3011s vent loop. Unfortunately, during the
procedure 3011s fan was flooded with water and stalled. The ground team wanted to dry the vent
loop as soon as possible, because they were worried about corrosion on the fan assembly.
Lacking the mechanism to push air through the vent loop, as had been done in the past on the
ground, the EVA team developed a procedure that they thought would work with the available
hardware aboard ISS. The procedure involved drying the fan by sucking air through the vent loop
with a vacuum cleaner.
There were multiple time constraints on the day the procedure was to be performed. First, the
team had been concerned with quickly drying the vent loop to avoid corrosion. Second, the team
perceived that crew time was limited and finally, when crew time was available, the procedure
was running up against a loss of signal (LOS) period.
Once the procedure began, it appeared to be going according to plan until the ground team lost
comm and telemetry with the ISS due to an LOS. By the time the ground re-established comm,
the telemetry indicated that EMU 3011s SOP had been activated and was reading 500psi lower
than before. They quickly realized that their procedure had resulted in the EMU releasing 100%
oxygen from the SOP into the vent loop, which was then sucked into the vacuum cleaner. This
was a potentially dangerous situation involving unintended consequences. During interviews,
system experts indicated that they should have been able to anticipate SOP activation due to the
reduced pressure created by the vacuum cleaner. The procedure was immediately stopped. No fire
occurred and the crew was not harmed.
Finding:

Oxygen enriched air was ingested by the vacuum cleaner creating the potential for a hazardous
situation.
This potentially dangerous situation occurred during an LOS.
During the LOS, the crew was not tasked to monitor for SOP activation.

A-23
 National Aeronautics and Space Administration EVA23Water Intrusion

Appendix B: List of Documents

Date of
File Name Description and Label
Document
Witness Interviews with 29 different personnel (or
Witness Interviews (32 Files) 8/9/13 - 9/26/13
groups)
Root Cause Analysis tool file. Contains Timeline, Fault
Tree, and Event & Causal Fault Tree information
EMU_MIB_(DATE) Rolling
being assembled by the MIB team. Folder contains
previous versions.
9-10-13 EVA 23 Investigation Plan 9/10/2013 MIB Hardware Team plan and data requirements
EMU Hardware Team Plan 9-11-13 9/11/2013 MIB Hardware Team plan presentation
Created and maintained by MIB Hardware Team.
EVA 23 Investigation Plan 9/4/2013
Visio flowchart of hardware team's plan
EVA 23 Investigation Plan 9/4/2013 PDF Version of .vsd file with same name. 9/4/13.
30 day status final released version. Adobe acrobat
30 day status final released version 9/9/2013
format, signed version
9/3/13: major updates after team meeting
EMU_MIB_30-Day_Outbrief_DRAFT[3] 9/3/2013
Updated on 9/2/13
9/3/13: minor updates
EVA_Water_Intrusion_MIB_30DayStatus 9/3/2013 8/31/13: updated photo and on-orbit test results
8/30/13: Tech Edited version
Richard Karl Fullerton's MIB Observations and
RKF Mishap Findings Rolling
Recommendations
Findings, Observations, Recommendations by MIB
MIB FOR Draft - 9-30-13 HJM & JP Rolling
Hardware team
John Steele (UTAS) recommendations for SEMU 3011
UTAS_John Steele_MIB_Input 9/4/2013
mishap.
Memo for EVA Suit Water Intrusion MIB from Dr.
MIB Surgeon Assessment of Mishap
8/13/2013 Laurence Ulissey assessing the health of EV1 & EV2
Astronauts
leading up to EVA 23
Root Cause Analysis Tool, Version 2.7, September
Root Cause Analysis Install Disc 9/1/2013
2010
Zip file with the draft CILs and HR 013. To unzip, you
CILs&HR for review on 8-8-13 8/8/2013
need a password, which is ESOCTeam
2013-07-30_Troubleshooting-Results-and-
8/12/2013 First troubleshooting presentation created by team
Theory
These are the notes taken by MIB Flight Surgeon
Notes from GJOP 08152013 8/15/2013 when he attended the GJOP meeting (Generic Joint
Ops Panel).
Troubleshooting Plans_RevL 8/16/2013 EMU 3011 Troubleshooting plans
Presentation from 1st visit to Windsor Locks in
UTCs EMU MIB Presentation 082213 9/9/2013
August.
Presentation at Windsor Locks to ISS Program
UTAS_Presentation_to Suffredini_09-23-13 9/9/2013
Manager on 9-23-13
EMU101 fault tree for MIB 8/7/2013 Fault tree provided by EC ISS Investigation Team
EMU3011_07242013 8/13/2013 Fault Tree from EC ISS Investigation Team
EMU3011_FaultTree_EC 8/16/2013 EC ISS Investigation Team engineering fault tree
EMU3011_FaultTree_EC_onepage 8/16/2013 EC ISS Investigation Team engineering fault tree
UTAS_2013-10-02 SEMU 3011 Item 134 Item UTAS Presentation of Item 134 and 141 inspection
10/2/2013
141 Investigation Interim Summary and investigation.
2013-08-22 UTAS Presentation to MIB UTAS presentation to MIB at the first visit to Windsor
8/22/2013
Windsor Locks Locks, 8-22-13
EMU_FT_Action Plan UTAS 8-9-2013 - closure UTAS Fault Tree Action Plan as of 8/9/2013. Contains
9/17/2013
rationale closure rationale for faults.
MIB Support Requests to UTAS 9-17-13 9/17/2013 UTAS's list of information requested by MIB as of 9-

B-1
 National Aeronautics and Space Administration EVA23Water Intrusion

Date of
File Name Description and Label
Document
17-13
WL MIB Action Item #12 Resp - EMU 6 Yr. Cert One EVA is to provide a summary of the certification
9/17/2013
Life data for the 6 year maintenance interval
One EVA to provide a list of project
recommendations that were not pursued/initiated by
WL MIB Action Item #13 Resp - Project
9/17/2013 NASA that would be improvements to the EMU
Recommendations
System, would help reduce risk, or help assess the
health of the EMU.
Determine the amount of water SEMU 3010
consumed During EVA# 23. Response: SEMU 3010
WL MIB Action Item #3 Resp - 3010 Water
9/17/2013 was due for an annual drain and fill of the feedwater
Consumed EVA 23
tank to re-iodinate the feedwater. After EVA#23 no
refill water quantity was measured.
One EVA to provide a chart that describes the life of
WL MIB Action Item #9 Resp - MEGA Process 9/17/2013
an EMU through the MEGA process.
This Standard Procedure defines the process to
WL MIB Action Item #9 Resp - MEGA Process - manage MEGA SEMU processing within One EVA.
1EVA-SP-0019 Rev. G SP for MEGA SEMU 9/17/2013 In addition, this defines the process for implementing
Management Process initial hardware requirements that have been set in
FEMU-R-001. Effective Apr 2013.
EMU Processing Requirements and Constraints. This
document provides requirements for the normal
WL MIB Action Item #9 Resp - MEGA Process -
9/17/2013 flight, manned chamber, fit check and NBL
FEMU-R-001 Rev. CR (DRD 30)
processing of the EMU and its CEI's at JSC (FCE/EVA)
and KSC. Aug 2013.
Assured Shuttle Availability EMU Enhancement
Assured Shuttle Availability EMU
9/26/2013 Study. Conducted by Andrew Hoffman, East Windsor
Enhancement Study 1991
Associates, September 1991.
The latest iteration of the US EVA 23 timeline. Credit
EVA 23 Anomaly Timeline 9-19-13 (S&MA) 9/19/2013 goes to Stacie Greene (EVA S&MA) for coordinating
the timeline inputs.
MANAGEMENT OF THE POST-SHUTTLE
ICES 2012 EMU Water Circuit 9/25/2013 EXTRAVEHICULAR MOBILITY UNIT (EMU)
WATER CIRCUITS
EFFORTS TO REDUCE INTERNATIONAL SPACE
STATION CREW MAINTENANCE FOR THE
ICES 2013 EMU Water Management 9/25/2013
MANAGEMENT OF THE EXTRAVEHICULAR MOBILITY
UNIT TRANSPORT LOOP WATER QUALITY
NRC's Advanced Technology for Human NRC's Advanced Technology for Human Support in
1997
Support in Space (1997) Space (1997)
Advanced EVA Capabilities for RASC (Revolutionary
NASA/TP-2004-212068 2004
Aerospace Systems Concepts), 2004 Report
Audio of several loops (S/G, FD, EVA, etc.)
EVA 23 Prep and EVA Transcript 8/12/2013
transcribed
CAPCOM L Audio Console Loop for EVA 23, O1: Kate
CAPCOM L - EVA 23 7/30/2013 Rubins, O2: Megan Behnken, O2 Ground IV: Shane
Kimbrough 197:06:40 - 14:00, Keyset 4278.
EVA PROC ISS 1 Audio Console Loop for EVA 23, EVA
EVA PROC ISS 1 - EVA 23 7/30/2013
to EMU & Airlock, 197:06:40 - 14:00
EVA ISS 1 Audio Console Loop for EVA 23, EVA Coord
EVA ISS 1 - EVA 23 7/30/2013 Task MPSR to MER FCR Loop, 197:06:40 - 14:00. (Side
comm channel between consoles, mostly silent)
ETHOS L Audio Console Loop for EVA 23 197:06:40 -
ETHOS L - EVA 23 7/30/2013
197:14:00, Keyset 4209
OSO R Audio Console Loop for EVA 23, O2: Brian
OSO R - EVA 23 7/30/2013
Berry, 197:06:40 - 14:00, KeySet 4208

B-2
 National Aeronautics and Space Administration EVA23Water Intrusion

Date of
File Name Description and Label
Document
ETHOS MPSR Audio Console Loop for EVA 23, O1:
ETHOS MPSR - EVA 23 7/30/2013 Michael Salopek, O2: Brandon Lloyd, 197:06:40 -
14:00 Keyset 4441
ISS Space-To-Ground 2 Audio Console Loop for EVA
ISS S/G 2 - EVA 23 7/30/2013
23, 197:06:40 - 197:14:00 (Mostly silent)
EVA Right Audio Console Loop for EVA 23, O1: Paul
Dum, O2: Karina Eversley, O3: Allison Bollinger,
EVA R - EVA 23 7/30/2013
197:06:40 - 14:00, KeySet 4290 (Comm btwn Front &
Back Room plus other loops in background)
ISS Flight Director 1 Audio Console Loop for EVA 23
ISS FD 1 - EVA 23 7/30/2013
197:06:40 - 197:14:00 (Very similar to FD R Loop)
ISS Space-To-Ground 1 Audio Console Loop for EVA
23, contains FD/EVA
ISS S/G 1 - EVA 23 7/30/2013
Officer/Backroom/CapCom/EV1/EV2, MDRF 217-02,
197:06:40 - 197:14:00
FD R (Flight Director Right) Audio Console Loop for
FD R - EVA 23 7/30/2013 EVA 23, O2: Dave Korth, 197:06:40 - 197:14:00,
Keyset 4273 (Very similar to ISS FD 1 loop)
EVA Multi-Purpose Support Room ISS 1 Audio
EVA MPSR ISS 1 - EVA 23 7/30/2013 Console Loop for EVA 23, 197:06:40 - 14:00. (No use
of this loop after 6 seconds)
EVA Multi-Purpose Support Room ISS 1 Audio
Console Loop for EVA 23 (thru GMT 198) 197-:04:04 -
EVA MPSR ISS 1 Loop - EVA 23 7/30/2013
197:20:00, 197:10:30 - 197:20:00, 197:20:00 -
198:05:00, MDRF 225-06
ISS Mission Evaluation Room Manager 1 Audio
Console Loop for EVA 23, 197:04:04 - 197:20:00,
ISS MER MGR 1 - EVA 23 7/30/2013
197:10:30 - 197:20:00, 197:20:00 - 198:05:00,
KeySet 4139
MER Logs 7/16 - 7/18 7/18/2013 MER Logs from EVA 23 to 2 days later, 7/16 - 7/18
EMU 3010 and 3011 Battery data from EVA 21, 22,
Three_EVA_Comparison 8/28/2013
23
3011_USEVA 23_EVA_Valid-Data-Points-Only 8/7/2013 EMU 3011 EVA 23 EMU Telemetry data
3011_USEVA 23_PreEVA 8/7/2013 EMU 3011s data from EVA 23 Pre-Breathe
Action list for response to SEMU 3011 water
8/2/2013 Action List received from XA
in vent loop anomaly r7
EMU telemetry for EMUs 3010 and 3011 during EVAs
Combined Data Set from EVAs 21 22 23 for
8/7/2013 21, 22, and 23. These files have been cleaned-up by
MIB
removing checksum errors/bad data passes
Plots of Last Three EVAs 9-10-13 9/17/2013 Data plots from UTAS of EVA 21, 22, 23
EVA 23 Private Crew Debrief 7/17/2013 Private debrief with EV1 and EV2 after EVA 23
U.S. EVA 22 Crew Debrief 9/26/2013 Private debrief with EV1 and EV2 after EVA 22
EVA 23 Prebrief - CD1 9/26/2013 EVA 23 crew Prebrief w/ EV1 & EV2 - Part 1
EVA 23 Prebrief - CD2 9/26/2013 EVA 23 crew Prebrief w/ EV1 & EV2 - Part 2
E35_36_EVA_Transcript 9/27/2013 Post-flight EVA debrief with Chris EV1
EVA Task Group's presentation prior to EVA 22 & 23
EVA Task Priorities 3/6/2013 lining out rules, procedures, schedules, console
schedules, etc.
EV2's blog on ESA's website about his experience
EV2s Blog of EVA 23 8/21/2013
during EVA 23.
Dog and Pony Show .PPTs explaining design of
DP _________ (15 Files) 8/13/2013
different EMU systems
brains book SOP 9-10-08 8/14/2013 Animation of SOP workings
EMU-Schematic_MOD 7/30/2013 MOD's large schematic of the EMU

B-3
 National Aeronautics and Space Administration EVA23Water Intrusion

Date of
File Name Description and Label
Document
"Engineering" version of the large EMU schematic.
EMU-Schematic-HS_rev M 2007 7/30/2013
Originated from Hamilton possibly.
Development and certification of new EMU
SSPCB_EMU_DIDB 8/21/2013
Disposable In-suit Drink Bag (DIDB)
U.S. Spacesuits by K.S. Thomas & J.H.
10/28/2011 "US Spacesuits"
McMann
EMU Requirements Evolution 1994 Rev A 9/17/2013 EMU Requirements Evolution 1994 Rev A
Rev B of book originally written in 1994. Some parts
One EVA EMU Requirements Evolution 2005
9/17/2013 of 1994 version were removed from 2005 One EVA
Rev. B
version.
Orlan Fluids Schematic 9/18/2013 For comparison to EMU
Orlan Integ Schematic 9/18/2013 For comparison to EMU
Anomaly_GMT 198-EMU 3011 SOP Pressure Anomaly report of EMU 3011 SOP Pressure drop
Drop During WetDry Vacuum and Dryout 8/21/2013 during Wet/Dry vacuum and dryout activity on GMT
Activity 198
Challenger Actions Report 1986 8/15/2013 PDF Report
Challenger-Rogers_Commission_Report 8/15/2013 PDF Report
Columbia CAIB Report Vol 1 2003 8/15/2013 PDF Report
Links to Lessons Learned databases for DX and XA
DX and XA Lessons Learned 8/16/2013
(may require granted access)
EMU Coolant Loop Recovery 2005 8/15/2013 PDF Report
Huge data file with all EMU failures recorded up to
EMU Failure History 6-12-2012 asf 8/15/2013
2012
EMU Metox Mishap 2007 8/21/2013 PDF Report
Trimmed version of the larger file. Contains all
EMU Water Failure History - RKF 9/10/2013 water/contamination related EMU failures and other
failures of interest to MIB
MCC Anomaly Log for EMU 3011 SOP GMT 198 - EMU 3011 SOP Pressure Drop During
8/15/2013
Pressure Drop During Vacuum Dryout Wet/Dry Vacuum & Dryout Activity
Recovery Plan for Extravehicular Mobility Unit (EMU)
RP-05-121_05-027- and International Space Station (ISS) Airlock Coolant
8/14/2013
E_Recovery_Plan_for_EMUISS_Final_Report Loop Review for Return-to-Flight Technical
Assessment Report (Oct 2005)
Significant Incidents and Close Calls in Human
Significant Incidents and Close Calls in Human
8/6/2013 Spaceflight: EVA Operations. EVA Incident History
Spaceflight EVA_8.2.2011
Chart
EMU/Airlock Coolant Loop Status and Go Forward
SSPCB_SPCU_032205 8/21/2013
Plan (March 2005)
EMU/Airlock Coolant Problem Test results with
SSPCB_SPCUHXon17P_120704 8/21/2013
respect to SPCU Heat Exchanger (Dec 2004)
Spare SPCU Heat Exchanger Screening Test
VCB_SPCUHXFix_010405 8/21/2013
Recommendation (Dec 2004)
Suit/PLSS specific failure histories from PRACA. They
EMU 3005 FPRs Failure History 9/10/2013
appear to cover incidents from 2000 thru today.
Suit/PLSS specific failure histories from PRACA. They
EMU 3010 FPRs Failure History 9/10/2013
appear to cover incidents from 2000 thru today.
Suit/PLSS specific failure histories from PRACA. They
EMU 3011 FPRs Failure History 9/10/2013
appear to cover incidents from 2000 thru today.
Suit/PLSS specific failure histories from PRACA. They
EMU 3015 FPRs Failure History 9/10/2013
appear to cover incidents from 2000 thru today.
EMU CO2 Sensor Item 122 PRACA Failure Item 122 (C02 sensor) histories. These go back to the
9/10/2013
History 2 late 1970s.
EMU CO2 Sensor Item 122 PRACA Failure Item 122 (C02 sensor) histories. These go back to the
9/10/2013
History FPRs late 1970s.
EMU ITEM 123 SV787994 FPRs Failure History 9/10/2013 EMU Component specific failure histories from

B-4
 National Aeronautics and Space Administration EVA23Water Intrusion

Date of
File Name Description and Label
Document
PRACA.
EMU Component specific failure histories from
EMU ITEM 125 SV769480 FPRs Failure History 9/10/2013
PRACA.
EMU Component specific failure histories from
EMU ITEM 128 SV767699 FPRs Failure History 9/10/2013
PRACA.
EMU Component specific failure histories from
EMU ITEM 134 SV769403 FPRs Failure History 9/10/2013
PRACA.
EMU ITEM 140 SV7292500 FPRs Failure EMU Component specific failure histories from
9/10/2013
History PRACA.
EMU Component specific failure histories from
EMU ITEM 140 SV805280 FPRs Failure History 9/10/2013
PRACA.
EMU Component specific failure histories from
EMU ITEM 141 SV805257 FPRs Failure History 9/10/2013
PRACA.
EVA Lessons Learned Volume1 Rev A Oct 1994.
EVA Lessons Learned Volume1 RevA Oct 1994 9/18/2013
Document number: JSC-26055:I
EVA Lessons Learned Volume2 Rev A Oct 1994.
EVA Lessons Learned Volume2 RevA Oct 1994 9/18/2013
Document number: JSC-26055:II
NASA FESSRP Approved Hazard Report covering EMU
Hazard Report EMU-013 Contamination
9/17/2013 Contamination Corrosion, Loss of Visibility. Fogging
Corrosion-Loss of Visibility
of the helmet.
PRACA EMU History Report 8-19-2013 9/17/2013 PRACA listing of all EMU related failures since 1980
NASA EMU LSS/SSA Data Book. Rev P, September
DRD 13 EMU LSS-SSA Data Book, Rev. P, Sept
8/20/2013 2010. MIB had requested a newer version of this
2010
data book in digital format.
Redbooks for EMUs spanning from STS-114 to 135.
Flight Data Book for STS-114 to 135 (14 files) 8/20/2013 Redbooks contain all the specific information for
EMU components, calibration data, etc.
ETHOS_Log_EVA 22 8/27/2013 ETHOS console logs from EVA 22
EVA_Log_EVA 22 8/27/2013 EVA console logs from EVA 22
BME Log EVA 23 8/17/2013 BME console logs from EVA 23
CAPCOM_E36_LOG_EVA 23 8/5/2013 CAPCOM_E36_LOG_EVA 23
CB EVA discussion_EVA 23 8/5/2013 CB Pandion discussion curing EVA 23
Combined timeline of events - INC 36 EVA 23 8/7/2013 Transcription of EVA 23 loop traffic by EC Co-op
EMU CORE Electronic Log EVA-23 8/28/2013 EMU CORE Electronic Log EVA-23
EMU MER Data Log - US EVA #23 8/28/2013 EMU MER Data Log - US EVA #23
ETHOS_Logs_EVA 23 8/5/2013 ETHOS console logs from EVA 23
EVA POCC Pandion Log EVA 23 8/12/2013 Pandion discussion within EVA POCC during EVA 23
EVA_Logs_EVA 23 8/5/2013 EVA console logs from EVA 23
Hand_13_FD197_O1_Exp_36_-
_O1_(Ceccacci_Intrepid)_to_O2_(Korth_Odys 8/7/2013 Orbit 1 to 2 FD handover notes from EVA 23
sey)
Hand_13_FD197_O2_Exp_36_-
_O2_(Korth_Odyssey)_to_O3_(Nelson_Perido 8/7/2013 Orbit 2 to 3 FD handover notes from EVA 23
t)
Hand_13_FD197_O3_Exp_36_-
_O3_(Nelson_Peridot)_to_O1_(Contella_Steel 8/7/2013 Orbit 3 to 1 FD handover notes from EVA 23
)
Joe's Pellicciotti's integrated log from multiple
Log Summary 8/15/2013
sources
Surgeon Log Notes EVA 23 8/7/2013 Surgeon console log notes from EVA 23
TOPO_Logs_EVA_23 8/7/2013 TOPO Console logs from EVA 23
MER Console Logs (50 files) 8/12/2013 Logs from various MER consoles GMT 197-200
Hand-Written console log notes of FD (Dave Korth)
Dave Korth (FD) EVA 23 Notes 7/17/2013
during EVA 23

B-5
 National Aeronautics and Space Administration EVA23Water Intrusion

Date of
File Name Description and Label
Document
Hand-Written console log notes of EVA Task (Sandy
Sandy Moore (EVA Task) EVA 23 Notes 7/17/2013
Moore) during EVA 23
Hand-Written console log notes of EVA (Karina
Karina Eversley (EVA) EVA 23 Notes 7/17/2013
Eversley) during EVA 23
Hand-Written console log notes and procedures of
ETHOS EVA 23 Notes 7/17/2013
ETHOS console person during EVA 23
Airlock EVA Sys Console Handbook 8/21/2013 Airlock EVA Systems Console Handbook
EMU Console Handbook 2011 8/15/2013 EMU Console Handbook 2011
EMU Flight Checklist 8/15/2013 EMU Flight Checklist
Supplied by MIB member from dusty personal MOD
EVA Checklist Rationale 1989 8/19/2013
files
FEMU-R3 shows the on-orbit maintenance and
FEMU-R-003_Rev_AP 8/6/2013
checkout requirements
air flow contam proc 8/28/2013 Air flow contamination flight procedure
co2 sensor bad proc 8/28/2013 CO2 Sensor Bad Procedure
coolant loop flush proc 8/28/2013 Coolant Loop Flush Procedure
coolant loop maint proc 8/28/2013 Coolant Loop Maintenance Procedure
depress repress cue card 8/28/2013 Depress-Repress Cue Card
emu checkout proc 8/28/2013 EMU Checkout Procedure
emu midterm checkout proc 8/28/2013 EMU Midterm Checkout Procedure
loss of cooling proc 8/28/2013 Loss of Cooling Procedure
terminate eva proc 8/28/2013 Terminate EVA Procedure
The ISS EVA Flight Procedures Rationale Handbook
was developed by the EVA Systems Group for
EVA Procedure Rationale Handbook (28 files) 8/16/2013 training purposes. It contains the rationale behind
most steps found in the ISS EVA Checklist (JSC
48538)
List of EVA Flight Rules (list, not the actual rules
EVA FLT RULES 21007 (LISTING) 11_30_12 8/4/2013
themselves).
ISS EVA Flt Rules VolB_Sec15 8/6/2012 ISS EVA Flight Rules Volume B, Section 15
EVA portions of ISS GGR&C (Generic Groundrules,
Requirements, and Constraints). The applicable
ISS GGR&C - EVA Portions (SSP 50261-01 & - section are as follows:
2011, 2012
02) In Part 1 (SSP 50261-01): Paragraphs 3.3.5.3, 3.10,
and 4.3.2
In Part 2 (SSP 50261-02: Paragraphs 3.10, and 4.5
FMEA/CIL for I-123 123FM04 9/16/2013 FMEA/CIL for I-123 as of 3/13/2007
FMEA/CIL for I-123 123FM07 9/16/2013 FMEA/CIL for I-123 as of 3/13/2007
FMEA/CIL for I-125 123FM02 9/16/2013 FMEA/CIL for I-125 as of 5/10/2007
FMEA/CIL for I-134 123FM02 9/16/2013 FMEA/CIL for I-134 as of 11/16/2005
FMEA/CIL for I-140 123FM04 9/16/2013 FMEA/CIL for I-140 as of 7/21/2008
FMEA/CIL for I-140 123FM05A 9/16/2013 FMEA/CIL for I-140 as of 7/21/2008
FMEA/CIL for I-141 123FM05 9/16/2013 FMEA/CIL for I-141 as of 3/13/2007
MIB Interview Listing Rolling List of people interviewed, job, date of interview, etc.
Categorization of witness interviews by topic, with
EVA Mishap Transcript Data Rolling content all into one excel file for easy lookup and
comparison.
NOTES FROM TELECON WITH VICKIE Telecon with MIB Advisor prior to in-person
8/15/2013
MARGIOTT OF HAMILTON interview at Windsor Locks
IRIS record s-2013-199-00005 8/9/2013 IRIS record of EVA 23 Mishap as of 8-9-2013
IRIS record revised 8/12/2013 Revised IRIS record of EVA 23 Mishap as of 8-12-2013
IRIS Files 7/17/2013 GMT 198 - EMU 3011 SOP Pressure Drop During

B-6
 National Aeronautics and Space Administration EVA23Water Intrusion

Date of
File Name Description and Label
Document
Wet/Dry Vacuum & Dryout Activity (NOT actual IRIS
report)
ISS Independent Safety Task Force Report
8/15/2013 ISS Independent Safety Task Force Report 2007
2007
ISS Response to Columbia Accident Investigation
ISS Response to CAIB 2005 8/15/2013
Board 2005
Overview of ISS Flight Controller Ratings on
ISS_MOD_Survey_2004 8/21/2013
Organizational Risk and Tool Development Survey
Certification Status EVA 23 8/6/2013 EVA Certification Status of console folks
List of support personnel here at JSC and at our
Copy of EVA 23 MSP INC 36 REV1 for MIB 8/7/2013 contractors plants at Windsor Locks, CT and Dover,
DE
EVA Trng Rqmts 1987 and before 8/27/2013 EVA Training Requirements from 1987 and before
EVA Trng Rqmts 1991 8/27/2013 EVA Training Requirements 1991
EVA_Flight_Assignments web version 6-26-13 8/7/2013 EVA MOD flight assignments spreadsheet
List of people in the POCC 8/9/2013 List of people in the POCC during EVA 23
JSC-19450 1999 EMU Systems training workbook
JSC-36344 2001 ISS Joint Airlock Systems training manual
MOD Personnel Cert. Records 8/20/2013 MOD Personnel Cert. Records
CPR Certification DX3 - detailed 9/17/2013 Listing of completed training courses by personnel
EVA Systems Training Flow RevH_7_8_13 9/17/2013 EVA Systems training flow chart
EVA Training Flow Overview 9/17/2013 EVA Training Flow Overview flowchart
MOD EVA Cert Guide 2010 jsc-64082 9/17/2013 EVA Certification Guide from March 2010. From DX3.
Personnel Certification Statuses EVA 23 9/17/2013 Personnel Certification Statuses EVA 23.
Standard Procedure for Training and Certification
Training Cert Req for CORE and MER EVA
9/30/2013 Guidelines for CORE/MER EVA Operations and
(MST061 1EVA-SP-0023_RevA SP)
Events- EMU LSS & SPCE. 11-28-2012.
TrainingRecord_MER1Level1_Jeff Outlaw 9/30/2013 MER training record
TrainingRecord_MER2Level1_Alicia Ruiz 9/30/2013 MER training record
3-Ring Binder of EVA S&MA Cert Guide, Cert Records,
EVA Safety Book 9/24/2013
Console Hours Log, and TO-8 Funding Trend Data
DX Briefing package covering the R&R for I-123 FPS
EMU I-123 FPS On-Orbit R&R Briefing Package 10/3/2013
On-Orbit
EMU I-123 FPS On-Orbit R&R - PLSS Impact DX Procedure covering the R&R for I-123 FPS On-
10/3/2013
Shield Removal and Installation Orbit
EMU I-123 FPS On-Orbit Remove and DX Procedure covering the R&R for I-123 FPS On-
10/3/2013
Replace_Rev6 Orbit
Supplement to DX Procedure covering the R&R for I-
EMU Tilt on EDDA 10/3/2013
123 FPS On-Orbit
Fan Pump Separator Cue Card for DX Procedure
F_P_S Cue Card Rev 6 10/3/2013
covering the R&R for I-123 FPS On-Orbit
Presentation showing results and PHOTOS of the I-
EMU 3011 Troubleshooting on-orbit Aug 31
9/9/2013 134 valve, filter, gas trap removal performed on orbit
2013
on August 31, 2013
RPM Data during 3011 Screen test 8-27-13 9/4/2013 RPM Data during 3011 Screen test 8-27-13
Plots from on-orbit screen test of EMU 3005. No
2013-08-14_Screen-Test_3005 9/18/2013
high-rate data.
Plots from on-orbit screen test of EMU 3010. No
2013-08-14_Screen-Test_3010 9/18/2013
high-rate data.
Plots from on-orbit screen test of EMU 3005 & 3010
2013-08-14_Screen-Test_Both-3005-3010 9/18/2013
combined. No high-rate data.
High-Rate data plots of on-orbit EMU 3011 screen
3011-screen_2013-08-27-RevA 9/13/2013
test on 8-27-13. Pre I-134 & I-141 R&R. Revision A.

B-7
 National Aeronautics and Space Administration EVA23Water Intrusion

Date of
File Name Description and Label
Document
High-Rate data plots of on-orbit EMU 3011 screen
3011-screen_2013-08-31-Part1 9/13/2013
test on 8-31-13. Post I-134 & I-141 R&R. Part 1.
High-Rate data plots of on-orbit EMU 3011 screen
3011-screen_2013-08-31-Part2 9/13/2013
test on 8-31-13. Post I-134 & I-141 R&R. Part 2.
Drawing to show how to remove shim during I-
02_Shim-Removal-Technique 9/17/2013
134/141 R&R.
Summary of 3011 screen test data from before and
3011 Screen Test 8-31-13_Summary_RevB 9/17/2013
after the I-134/141 R&R (8-27-13 and 8-31-13).
36-0836 EMU 3011 WATER SEPARATOR Documentation used for the on-orbit R&R of I-
9/17/2013
SCREEN CRIBSHEET 134/141 on 8-31-13
36-0861 Procedure EMU 3011 CONDENSATE
Documentation used for the on-orbit R&R of I-
WATER RELIEF VALVE (I-134) AND GAS TRAP 9/17/2013
134/141 on 8-31-14
(I-141) REMOVE AND REPLACE
Documentation used for the on-orbit R&R of I-
36-0862 EMU 3011 R&R CHECK PROCEDURE 9/17/2013
134/141 on 8-31-15
36-0866 EMU 3011 R&R CHECK PROCEDURE Documentation used for the on-orbit R&R of I-
9/17/2013
WATER CLEANUP 134/141 on 8-31-16
EMU 3011 Component R&R Procedure Documentation used for the on-orbit R&R of I-
9/17/2013
Flowchart 134/141 on 8-31-17
Links to Imagery Online photos of the removal of I-
I-134 and 141 R&R Photo Links from 8-31-13 9/17/2013
134 and 141
Documentation used for the on-orbit R&R of I-
I-134 R&R Comm-Loop Schematic_RevCAB 9/17/2013
134/141 on 8-31-17
MCC Internal Note EMU 3011 R&R, Check, & Documentation used for the on-orbit R&R of I-
9/17/2013
Cleanup deltas 134/141 on 8-31-18
Calendar for planning the removal of I-123 Fan Pump
Schedule for I-123_R&R_revF 9/17/2013
Separator, populated by EC
EVA_MIB_Phone_List Rolling Board Contact Info
MIB Schedule Rolling Master Schedule for MIB reports, due dates, etc.
KN CoP Instructions (5 files) 8/8/2013 Various .PPTs to help manage NSCKN
HEO MIB Appointment Letter - 22 July 2013 7/22/2013 Official MIB Appointment letter
Various examples of reports, presentations, NDA,
Examples of MIB Products (6 files) 8/12/2013
etc. for use of the MIB in proceedings
Training presentations and documents to bring MIB
MIB Training (9 files) 8/9/2013
members up to speed on process and rules
MIB_Action_Item_Status_(DATE) Rolling List of current board action items
MIB Org Charts (5 files) 8/22/2013 Org charts for DX, EC, EC5, NT, XA
ISS EVA Suit Water Intrusion_MIB_RFIA_Form 8/12/2013 Request for Information Template
Listing of all data impounded and requested (file you
MIB Evidence Log 9/11/2013
are currently using)
Knowledge Now User Access Controls 8/9/2013 Knowledge Now User Access Controls
ISS EMU MIB Knowledge Now (KN) Community of
NSCKN Verify CoP Member Security Access 8/9/2013
Practice (CoP) - Verify CoP Member Security Access
ISS EMU MIB Knowledge Now (KN) Community of
NSCKN Verify Folder Security Access 8/9/2013
Practice (CoP) - Verify Folder Security Access
Sheet that defines different terms used in creating
MIB Process Definitions 9/13/2013 and completing the MIB report - Observation, Cause,
etc.
One EVA Org Chart 9-12-2013 9/12/2013 One EVA Org Chart 9-12-2013
On-Orbit photos of EMU 3011 post EVA 23 showing
3011 Helmet/HUT Post-EVA (24 files) 8/15/2013
water in/around T2 Vent and helmet area
On-Orbit photos of EV2 taken by EV1 during EVA 23,
EVA 23 Photos of Parmitano (20 files) 8/15/2013
showing water inside EV2's helmet
EVA 23 Photos 7/17/2013 EVA Suit MIB Video, Digital Imagery, GMT 197-0734-

B-8
 National Aeronautics and Space Administration EVA23Water Intrusion

Date of
File Name Description and Label
Document
0737
Water in Helmet Report Cover Photo 9/17/2013 From Imagery Online
The maintenance history of the specific hardware on
OOT Access 8/6/2013
ISS can be seen in OOT (On-Orbit Tracking) Database.
The EMU rotation history since STS-98. From XA/EVA
EMU_Rotation 8/16/2013
Logistics Lead
US and Russian EVA summaries, stats on # of EVAs,
Integ EVA History 8/15/2013
hours, etc.
A count of EVAs per SEMU per year. The data used
was based off Shuttle launch date not actual EVA
date so the count per year might be slightly off but
the EVA count per SEMU is accurate. Something to
SEMU Uses 8/19/2013
consider when reviewing the EVA count data, it does
not truly illustrate the configuration of the SEMU
(PLSS or SSA) during the EVA, as configuration and
components might vary from rotation to rotation
PLSS Logistics One EVA 9/17/2013 PLSS Logistics One EVA. Status of all PLSS/SEMUs
EVA 22 Part 1 & 2 (2 files) 8/1/2013 Video feed from EVA 22 (GMT 190)
EVA 22 Prep Part 6 + Post Ops + H2O
8/1/2013 Video feed from EVA 22 pre and post-EVA (GMT 190)
Recharge
Video of EV1 squeezing EV2's DIDB from EVA 23,
EVA 23 Luca's DIDB 8/1/2013
showing no signs of leakage.
Wireless Video System (helmet cam) of EV1 during
EV1 WVS 8/1/2013
EVA 23
EVA 23 EV2 (2 copies) 8/1/2013 DVD
EVA 23 EV1+Post Repress+Search For Leak (2
8/1/2013 DVD
copies)
EVA 23 Post EVA 8/1/2013 DVD
EVA 23 Debrief Part 1 + 2 8/1/2013 DVD
EVA 23 + Post Debrief Part 1 + 2 8/1/2013 DVD
EVA 23 Prep & EVA 8/1/2013 DVD
Video of Post EVA 23 Restow, PAO, Drink Bag GMT
GMT 199-200, Restow-PAO-Drink Bag 8/1/2013
199-200
EMU Trouble Shooting (3011) GMT 198, On-orbit
GMT 198 EVA 23 EMU T/S Parts 1 8/1/2013 troubleshooting of EMU 3011 the day after EVA 23,
Part 1
EMU Trouble Shooting (3011) GMT 198, On-orbit
GMT 198 EVA 23 EMU T/S Parts 2 8/1/2013 troubleshooting of EMU 3011 the day after EVA 23,
Part 2
EMU Trouble Shooting (3011) GMT 198, On-orbit
GMT 198 EVA 23 EMU T/S Parts 3&4 8/1/2013 troubleshooting of EMU 3011 the day after EVA 23,
Part 3 & 4
EMU 3011 Troubleshooting GMT 207, On-orbit
GMT 207 (7/26/13) EMU 3011 T/S 1&2 8/1/2013 troubleshooting (Round 2) of EMU 3011, 10 days
post-EVA 23. Parts 1 & 2.
EMU 3011 Troubleshooting GMT 207, On-orbit
GMT 207 (7/26/13) EMU 3011 T/S Parts 3-4-5 8/1/2013 troubleshooting (Round 2) of EMU 3011, 10 days
post-EVA 23. Parts 3-4-5.
EVA Suit MIB Video, GVS Downlink 1, GMT 197-1238-
EVA Suite MIB Video 8/11/2013
1240, GMT 197-1303-1311
EVA Suit MIB Video, GVS Downlink 2, GMT 197-1532-
EVA Suite MIB Video 8/11/2013
1549, GMT 197-1551-1600
EVA Suit MIB Video, GVS Downlink 3, GMT 197-1238-
EVA Suite MIB Video 8/11/2013
1240, GMT 197-1303-1311, GMT 197-1351-1353

B-9
 National Aeronautics and Space Administration EVA23Water Intrusion

Date of
File Name Description and Label
Document
EVA Suit MIB Video, GVS Downlink 4, GMT 197-1238-
EVA Suite MIB Video 8/11/2013
1240, GMT 197-1303-1311
EVA Suit MIB Video, MPEG Encoder, GMT 197-0548-
EVA Suite MIB Video 8/11/2013
0906
EVA Suit MIB Video, MPEG Encoder, GMT 197-0915-
EVA Suite MIB Video 8/11/2013
1138
EVA Suit MIB Video, MPEG Encoder, GMT 197-1311-
EVA Suite MIB Video 8/11/2013
1519
EVA Suite MIB Video 8/11/2013 EVA Suit MIB Video Downlink 1, GMT 197-1047-1440
EVA Suite MIB Video 8/11/2013 EVA Suit MIB Video, Downlink 2, GMT 196-1044-1304
EVA Suite MIB Video 8/11/2013 EVA Suit MIB Video, Downlink 2, GMT196-1306-1527
EVA Suit MIB Video. Downlink 2, GMT 196-1544,
EVA Suite MIB Video 8/11/2013
GMT 197-0942
EVA Suite MIB Video 8/11/2013 EVA Suit MIB Video, Downlink 2, GMT 197-0555-1238
EVA Suite MIB Video 8/11/2013 EVA Suit MIB Video, Downlink 2, GMT 197-1311-1441
EVA Suite MIB Video 8/11/2013 EVA Suit MIB Video, Downlink 3, GMT 196-1044-1301
EVA Suit MIB Video, Downlink 3, GMT 196-1345,
EVA Suite MIB Video 8/11/2013
GMT 197-1303
EVA Suite MIB Video 8/11/2013 EVA Suit MIB Video, Downlink 3, GMT 197-1311-1527
EVA Suite MIB Video 8/11/2013 EVA Suit MIB Video, Downlink 4, GMT 197-1220-1039
EVA Suite MIB Video 8/11/2013 EVA Suit MIB Video, Downlink 4, 197-1354-1303
EC/UTAS Fault Tree (separately maintained from FT
Excess water in EMU Fault Tree_EVA 23 Rev G 10/8/2013
developed by Karon Woods).
Most accurate and complete transcription of EVA 23
Console Loops - EV1, EV2, IV, Ground IV, FD, EVA,
EVA 23_Audio_Loops 10/11/2013
EVA Backroom controllers. Created by Human Factor
support group.
PRACAs for UIA FPU 10/9/2013 PRACAs for UIA FPU
IRMA XA risks Excel 10/9/2013 Table of XA IRMA Instances
Collection of XA IRMA Risks with detailed information
IRMA XA Risks - Detailed 10/9/2013
for each instance, with closure rationale, etc.
Statistics on Training Flows for different MOD EVA
Training Flows for MOD EVA Positions 10/3/2013
Positions
Folder of photos and video from Class III I-123 FPS
123 FPS Scans & Inspections (50 files) 11/4/2013 scans (X-Ray, N-Ray, etc). Class III was a dry-run for
actual flight unit.
MER EVA Console log from EVA 22 11/18/2013
MER Manager console log for EVA 22 11/18/2013
SSP_30234-RevG1 FMEA CIL Requirements 12/17/2013 ISS requirements on FMEA/CIL process
Station Program Implementation Plan , Real-Time
SPIP vol9cumulativedcn007 12/17/2013
Operations
SPIP Vol 8 SSP_50200-08-RevB-DCN008- Station Program Implementation Plan, Increment
12/17/2013
Collated_Master[1] Execution Preparation
Station Program Implementation Plan, Station
SPIP vol 1SSP-50200-01_RevD[1] 12/17/2013
Program Management Plan
SPIP VOL 10 dcn09 SSP_50200-10-Baseline- Station Program Implementation Plan, Sustaining
12/17/2013
DCN_009-Collated_Master[1] Engineering
Process governing criteria and process for opening a
PRACA FIAR Requirements JSC28035 12/17/2013
failure report
In flight Investigation opened in response to
PART_Record_8590-SOP IFI 12/17/2013
inadvertent activation of the SOP
PART_Record_8580-water intrusion in the In flight Investigation opened in response to water
12/17/2013
EMU IFI intrusion in the helmet of EMU 3011
NASA-STD-3001 12/17/2013 NASA Space Flight Human-System Standard

B-10
 National Aeronautics and Space Administration EVA23Water Intrusion

Date of
File Name Description and Label
Document
Volume 2: Human Factors, Habitability, and
Environmental Health
On-Orbit Anomaly Resolution Process
MGT-OA-019_Rev_A 12/17/2013
Work Instruction
JSC-22254 Rev B Methodology for Conduct of Process for the Space Shuttle Program to conduct
12/17/2013
Space Shuttle Program Hazard Analyses hazard analyses
jsc_36528_vol1_mod_srqa_plan_rev_a 12/17/2013 MOD SR&QA Plan
JSC Mishap Response Plan JPR8621.1B 12/17/2013 JSC center level mishap plan
ISS Mishap Plan SSP_50190-Rev_E-DCN003-
12/17/2013 ISS Program Mishap plan
Collated_Master[1]
ISS Flight Rules VolB_Sec1 12/17/2013 ISS generic flight rules including EVA
Flight Control Operations Handbook procedure
FCOH SOP_6_10_2_3 12/17/2013
concerning formation of a Team 4
Aeromed Flt Rules VolB_Sec13 12/17/2013 Aeromedical flight rules
EMU 3011 Root Cause TIM MOD presentation concerning water history of on-
12/04/13
EMU Water History orbit EMUs

B-11
 National Aeronautics and Space Administration EVA23Water Intrusion

Appendix C: Definitions

Term Definition
An event or condition that results in an effect. Anything that shapes or
Cause
influences the outcome.
An event in which there is no injury or only minor injury requiring first aid
and/or no equipment or property damage, or minor equipment or property
Close Call
damage of less than $20,000, or not injury or only minor injury requiring first
aid, but which possesses a potential to cause a mishap
An event or condition that may have contributed to the occurrence of an
undesired outcome but, if eliminated or modified, would not by itself have
Contributing Factor
prevented the occurrence. Contributing factors increase the probability that an
event or condition will occur.
A graphical representation of a mishap or close call that shows the undesired
outcome (problem or accident) at the top of the tree, depicts the logical
Event and Causal
sequence of events, illustrates all causal factor(s) (including condition[s] and
Factor Tree
events) necessary and sufficient for the undesired outcome (mishap or close
call) to occur, and depicts the root cause(s) at the bottom of the tree.
An analytical technique, whereby an undesired system state is specified and
the system is analyzed in the context of its environment and operation to find
all credible ways in which the undesired event can occur. This can be
Fault Tree
performed by way of a symbolic or graphical logic diagram showing the
cause-effect relationship between an undesired top event or failure and one or
more contributing causes.
A conclusion, positive or negative, based on facts established during the
Finding investigation by the investigating authority (i.e., cause, contributing factor,
and observation).
High Visibility Those particular mishaps or close calls, regardless of the amount of property
(Mishaps or Close damage or personnel injury, that the Administrator, Chief/OSMA, CD,
Calls) ED/OHO, or the Center SMA director judges to possess a high degree of
programmatic impact or public, media, or political interest including, but not
limited to, mishaps and close calls that impact flight hardware, flight
software, or completion of critical mission milestones
Incident An occurrence of a mishap or close call.
An event or condition that created the proximate cause that, if eliminated or
modified, would have prevented the proximate cause from occurring. There
Intermediate Cause may be one to many intermediate causes for a single proximate cause. The
intermediate cause is between the proximate cause and the root cause in the
causal chain.
An unplanned event that results in at least one of the following: a. Injury to
non-NASA personnel, caused by NASA operations. b. Damage to public or
Mishap private property (including foreign property), caused by NASA operations or
NASA-funded development or research projects. c. Occupational injury or
occupational illness to NASA personnel. d. Mission failure before the

C-1
 National Aeronautics and Space Administration EVA23Water Intrusion

Term Definition
scheduled completion of the planned primary mission. e. Destruction of, or
damage to, NASA property except for a malfunction or failure of component
parts that are normally subject to fair wear and tear and have a fixed useful
life that is less than the fixed useful life of the complete system or unit of
equipment, provided that the following are true: 1) there was adequate
preventative maintenance; and 2) the malfunction or failure was the only
damage and the sole action is to replace or repair that component.
A factor, event, or circumstance identified during the investigation that did
not contribute to the mishap or close call, but, if left uncorrected, has the
Observation
potential to cause a mishap or increase the severity of a mishap; or a factor,
event, or circumstance that is positive and should be noted.
The event(s) that occurred, including any condition(s) that existed
immediately before the undesired outcome, directly resulted in the occurrence
Proximate Cause
of the undesired outcome and, if eliminated or modified, would have
prevented the undesired outcome. Also known as the direct cause(s).
An action developed by the investigating authority to correct the cause or a
Recommendation
deficiency identified during the investigation.
One of multiple factors (events, conditions, that are organizational factors)
that contributed to or created the proximate cause and subsequent undesired
Root Cause
outcome and, if eliminated or modified, would have prevented the undesired
outcome. Typically, multiple root causes contribute to an undesired outcome.
A structured evaluation method that identifies the root causes for an undesired
Root Cause Analysis outcome and the actions adequate to prevent recurrence. RCA should continue
until organizational factors have been identified or until data are exhausted.
An undesired outcome in this context refers to any event or result that is
unwanted and is different from the desired and expected outcome. This can be
loss of productivity, poor quality, production of scrap, increased risk,
increased cost, delay in schedule, damage to property, harm to the
environment, or harm to personnel. Undesired outcomes may also include
Undesired Outcome
intangible costs such as loss of public confidence or a decline in motivation.
(When describing an undesired outcome for a mishap or close call
investigation, the description should focus on the reason it was classified as a
mishap or close call; e.g., property damage, mission failure, fatality,
permanent disability, lost-time case, first aid case, etc.)

C-2
 National Aeronautics and Space Administration EVA23Water Intrusion

Appendix D: Acronyms

Abbreviated Form Phrase or Word

A/L Airlock
ACFM Actual cubic feet per minute
AE Assembly Element
AERCAM Autonomous Eva Robotic Camera
AI Artificial Intelligence
ALCLR Airlock Cooling Loop Recovery
ANSI American National Standards Institute
BME Biomedical Engineer
BPR Baseline Program Review
BTU British Thermal Unit
CA Completion of Assembly or Corrective Action
CAPCOM Capsule Communicator
CB Astronaut Office
CC Cubic Centimeters
CCB Change/Configuration Control Board
CCC Contaminate Control Cartridge
CEI Cargo Element Integrator
CERT Certification
CFR Code of Federal Regulations
CIL Critical Items List
CO2 Carbon Dioxide
CoP Community of Practice
CORE Central Operations Room
COTR Contracting Officers Technical Representative
CPR Certification Planning and Reporting
CSA Canadian Space Agency
CSR Certification Status Report or Customer Support Room
CTSD Crew and Thermal Systems Division
DCM Display Control Module
DICES Digital Integrated Communications Electronic

D-1
National Aeronautics and Space Administration EVA23Water Intrusion

System
DIDB Disposable In-suit Drink Bag
DOD Department of Defense
DVD Digital Video Disc/Drive
DVICE Digital Voice Inter-Communications Equipment
DX Robotics & Crew Systems Ops Division
E&CFT/ECFT Event and Causal Factor Tree
EAR Export Administration Regulations
EC Crew & Thermal Systems Division
ECS Environmental Control System
ECWS Enhanced Caution and Warning System
EDDA EMU Don/Doff Assembly
EDT Eastern Daylight Time
EMU Extravehicular Mobility Unit
ESA European Space Agency
ESD Electrostatic Discharge
ESOC European Space Operations Center
ETA Engineering Test Article
ETHOS Environmental and thermal Operating System
EV1 Extravehicular Crewmember (Astronaut) 1
EV2 Extravehicular Crewmember (Astronaut) 2
EVA Extravehicular Activity
EVA TRD Extravehicular Activity Test Requirements Document
FCR Flight Control Room
FD Flight Day, Flight Director
FESRRP Flight Equipment Safety and Reliability Review Panel
FIARS Failure Investigation and Analysis Reports
FIG Figure
FLT Flight
FMEA Failure Modes and Effects Analysis
FPRs Flight Performance Reserve
FPU Fluid Pump Unit
FRRs Flight Readiness Reviews
FT Fault Tree

D-2
National Aeronautics and Space Administration EVA23Water Intrusion

GFE Government Furnished Equipment

GGR&C Generic Ground rules, Requirements, and Constraints
GMIP Government Mandatory Inspection Points
GMT Greenwich Mean Time
GSFC Goddard Space Flight Center
GVS Ground Video Spooler
HAP Helmet Absorption Pad
HEO High Earth Orbit
HF Human Factors
HFACS Human Factors Analysis & Classifications Systems
HPV Helmet Purge Valve
HQ NASA Headquarters
Hr Hour
HS Hamilton Sundstrand
HTV H-II Transfer Vehicle
HUT Hard Upper Torso
HVCC High Visibility Close Call
Hz Hertz
ICES International Conference on Environmental Systems
IFA Inflight Anomalies
IISTF ISS Independent Safety Task Force
IMC Image Motion Compensation
INC Installation Notification Card
IP Injured Person, International Partner
IPR Interagency Purchase Request
IR CO2 Infrared Carbon Dioxide
IRIS Incident Reporting Information System
IRMA ISS Risk Management Application
IRT Interim Response Team
ISS International Space Station
ISSP International Space Station Program
ITAR International Traffic in Arms Regulations
IV Intravehicular Officer
JAXA Japan Aerospace Exploration Agency

D-3
National Aeronautics and Space Administration EVA23Water Intrusion

JSC Johnson Space Center

KSC Kennedy Space Center
LCVG Liquid Cooling and Ventilation Garment
LiOH Lithium Hydroxide
LOS Loss of Signal
LOTO Lockout/Tagout
LSS Life Support System
MAG Maximum Absorbency Garment
MBSU Main Bus Switching Unit
MCC Mission Control Center
MDFR Mission Data Request Form
MER Mission Evaluation Room
MET Metabolic
METOX Metal Oxide
MIB Mishap Investigation Board
MISO Mishap Investigation Support Office
ML Milliliters
MOA Memorandum of Agreement
MOD Mission Operations Directorate
MPEG Motion Picture Experts Group
MPSR Multi-Purpose Support Room
MSFC Marshall Space Flight Center
MWAR Mishap Warning-Action-Response
NASA National Aeronautics and Space Administration
NBL Neutral Buoyancy Lab
NESC NASA Engineering Safety Council
NPR NASA Procedural Requirements
NRCs National Research Council
NSC NASA Safety Center
NSCKN NASA Safety Center Knowledge Now
NTP Notice to Proceed
O2 Oxygen
OFT Orbital Flight Test
OJT On The Job Training

D-4
National Aeronautics and Space Administration EVA23Water Intrusion

OOT On-Orbit Tracking

OSMA Office of Safety Mission Assurance
OSO R Operations Support Officer
PAO Public Affairs Office
PDF Portable Document Format
PET phase elapsed time
PIT Process Improvement Team
PLSS Portable Life Support System
PMC Power Management Controller
POCC Payload Operations Control Center
PPM Parts Per Million
PPPEE Paper, People, Part, Equipment & Environment
PPT Sensor, Positive Pressure
PRACA Problem Reporting and Corrective Action
PROC Procedure
PSIA Pounds per Square Inch Absolute
PV Procedures Verification
PWR Payload Water Reservoirs
R&R Remove and Replace
RASC Revolutionary aerospace Systems Concepts
RCAT Root Cause Analysis Tool
Roskosmos Russian Federal Space Agency
S&MA Safety & Mission Assurance
S/G or S/G Loop Space to Ground
SBU Sensitive But Unclassified
SCOF SOP Checkout Fixture
SCU Service and Cooling Umbilical
SEMU Short Extravehicular Mobility Unit
SGT Stinger Ghaffarian Technologies
SMA Safety and Mission Assurance
SOP Secondary Oxygen Pack
SPCE Servicing Performance and Checkout Equipment
SPCU Service and Performance Checkout Unit
SPCUHXFix SPCU Heat Exchanger Fix

D-5
National Aeronautics and Space Administration EVA23Water Intrusion

SR&QA Safety, Reliability, and Quality Assurance

SRB Standing Review Board
SRD Standard Reference Document
SSA Space Suit Assembly
SSATA Space Station Airlock Test Article
SSM Subsystem Manager
SSPCB Space Station Program Control Board
SSTF Space Station Training Facility
STG Stage
TBR To Be Resolved
TCV Temperature Control Valve
TMG Thermal Micro-meteoroid Garment
TOPO Trajectory Operations Officer
TP Test Port
Trng Rqmts Training Requirements
U.S. United States
USN (ret) United States Navy (Retired)
UTAS United Technologies Aerospace Systems
WI Work Instruction
WL Windsor Locks
XA JSC Extravehicular Project Office

D-6
 National Aeronautics and Space Administration EVA23Water Intrusion

Appendix E: Detailed Timeline

Date Time Description
1/1/1970 NASA decided to design new EMU suit for Shuttle Program
1/1/1990 Station Program authorized 25 EVA EMU Cert
1/1/1993 Station Program completed 25 EVA EMU Cert
2/1/1990 NASA decided to use Shuttle EMU for Space Station EVA versus developing new
EMU.
11/16/2009 3010 MEGA PLSS delivered to ISS
2/8/2010 3005 MEGA PLSS delivered to ISS
5/14/2010 3011 MEGA PLSS delivered to ISS
2/11/2011 STS-130 EV2 (SEMU 3005) Observed large water droplets in helmet and water at
feet
2/17/2011 STS-130 EV1 (SEMU 3005) observed large water droplets in helmet
5/20/2011 STS-134 CO2 sensor failure (SEMU 3005) resulted in reduced EVA timeline
5/25/2011 STS-134 EV (SEMU 3004) experienced Eye Irritation, likely from anti-fog agent.
7/8/2011 3015 MEGA PLSS delivered to ISS
8/1/2012 3015 MEGA PLSS failed due to sublimator loss (1 use)
8/30/2012 04:52:21 GMT EVA 18 executed (1st use of 3011 MEGA PLSS)
9/5/2012 EVA 19 executed (EMU 3011)
11/1/2012 EVA 20 executed (EMU 3011)
5/11/2013 EVA 21 executed (EMU 3011)
7/9/2013 12:02:00 GMT EVA 22 started (PET=0:00)
7/9/2013 06:10:00 GMT EVA 22 ended (repress started)
7/9/2013 06:50:00 GMT A lot of water was reported in EV2's helmet post EVA 22 repress
7/16/2013 11:57:00 GMT EVA 23 started (PET=0:00)
7/16/2013 12:06:00 GMT EV1 and EV2 egressed crewlock
7/16/2013 12:35:00 GMT EV2's CO2 Sensor went off scale high.
7/16/2013 12:36:00 GMT EV2 reported bad CO2 sensor on DCM.
7/16/2013 12:41:00 GMT EV2 reported "I feel a lot of water on the back of my head, but I dont think it is
from my bag."
7/16/2013 12:44:00 GMT EVA suggested to EMU that water may be from vent port (ground conversation)
7/16/2013 12:45:00 GMT Ground IV asked EV2 to try and identify water source and if it is increasing
7/16/2013 12:46:00 GMT EV2 reported "I still feel it and I cannot tell you the source. My only guess is that
it came out of my bag and then found its way over there in the back, but I dont
have any water in the front of the helmet."
7/16/2013 12:51:00 GMT EV2 reported "the leak is not from the water bag and it is increasing",
unprompted.
7/16/2013 12:51:30 GMT EVA asked EVA Backroom "do we want to terminate?"
7/16/2013 12:52:00 GMT EV2 drank remainder of water in Drink Bag
7/16/2013 12:53:00 GMT EV1 visually estimated 1/2 L of water at back of EV2's head
7/16/2013 12:54:00 GMT Flight Director stated to EVA "It sounds like his drink bag is empty, and thats the
source of where all this water came from."
7/16/2013 12:57:00 GMT EV2 stated "Im thinking that it might not be the water bag", unprompted.
7/16/2013 12:57:30 GMT EV1 questioned sweat or urine as leak source. "No other place for it to come."
7/16/2013 12:58:00 GMT EVA stated to FD "just water tanks or sweat" as other possible leak sources.
7/16/2013 13:00:00 GMT EV2 questioned LCVG as a possible leak source
7/16/2013 13:02:00 GMT EV2 stated the water is mostly at the back of his head
7/16/2013 13:03:00 GMT EV2 reported eyes and helmet to be completely covered in water
7/16/2013 13:04:00 GMT Flight and EVA agreed to terminate EVA
7/16/2013 13:05:00 GMT Ground IV called Terminate EVA
7/16/2013 13:06:00 GMT EV2 headed towards Airlock

E-1
 National Aeronautics and Space Administration EVA23Water Intrusion

7/16/2013 13:08:00 GMT Intermittent comm with EV2 began.

7/16/2013 13:09:00 GMT Start of night-pass (sunset)
7/16/2013 13:10:00 GMT MCC called for EV2 to NOT connect to SCU
7/16/2013 13:18:00 GMT EVA support team discussed moving from Terminate to ABORT
7/16/2013 13:20:00 GMT FD and EVA decided to remain in Terminate
7/16/2013 13:26:00 GMT Airlock hatch was closed and locked
7/16/2013 13:29:00 GMT EVA 23 ended (nominal repress started)
7/16/2013 13:30:00 GMT EV2 confirmed he is OK via hand signals to EV1
7/16/2013 13:35:00 GMT Crew lock hatch opened
7/16/2013 13:38:00 GMT Helmet off
7/16/2013 17:00:00 GMT 3011 MEGA PLSS failed due to water in helmet (6 uses)
8/31/2013 I-134 removed/replaced from EMU 3011
8/31/2013 EMU 3011 screening test performed
9/16/2013 Orbital 1 launch to ISS

E-2
 National Aeronautics and Space Administration EVA23Water Intrusion

Appendix F: EMU Schematics

F-1
National Aeronautics and Space Administration EVA23Water Intrusion

F-2
National Aeronautics and Space Administration EVA23Water Intrusion

F-3
 National Aeronautics and Space Administration EVA23Water Intrusion

Appendix G: ISS Investigation Team Fault Tree

G-1
National Aeronautics and Space Administration EVA23Water Intrusion

G-2
 National Aeronautics and Space Administration EVA23Water Intrusion

Appendix H: EMU Evolution History (H2O Oriented)

H-1
National Aeronautics and Space Administration EVA23Water Intrusion

H-2
National Aeronautics and Space Administration EVA23Water Intrusion

H-3
National Aeronautics and Space Administration EVA23Water Intrusion

H-4
National Aeronautics and Space Administration EVA23Water Intrusion

H-5
National Aeronautics and Space Administration EVA23Water Intrusion

H-6
National Aeronautics and Space Administration EVA23Water Intrusion

H-7
National Aeronautics and Space Administration EVA23Water Intrusion

H-8
National Aeronautics and Space Administration EVA23Water Intrusion

H-9
National Aeronautics and Space Administration EVA23Water Intrusion

H-10
 National Aeronautics and Space Administration EVA23Water Intrusion

Appendix I: EVENT AND CAUSAL FACTOR TREE

EVA crew member (EV2) exposed to

potential loss of life during EVA 23.

UO-1

ISS Program performed EVA 23 EMU 3011 Helmet had large quantitiy of Flight Control Team/Crew did not Flight Control Team sent EV2 back to
w ithout recognizing EMU failure on w ater during EVA 23. terminate EVA as soon as w ater w as Airlock unaccompanied during EVA
EVA 22. reported in the helmet. termination.
ECFT-1 ECFT-2 ECFT-3 ECFT-4

FCT/Crew incorrectly attributed Water Large quantity of w ater from vent loop Flight Control Team did not recognize
in helmet during EVA 22 to the DIDB. leaked into helmet. the severity of the situation.

ECFT-1.1 ECFT-2.1 ECFT-4.1

See ECFT-3
expansion on
following pages
Flight Control Team did not perform EV2 did not communicate severity of
investigation of w ater source. his condition after terminate call.

ECFT-1.1.1 ECFT-4.1.1

Flight Control Team accepted the Ground team allow ed time pressures FCT's perception of the anomaly report EV2's comm system failed due to
explanation that the w ater w as from of impending EVA to influence actions. process as being resource intensive w ater
the drink bag. made them reluctant to invoke it.
ECFT-1.1.1.1 ECFT-1.1.1.2 ECFT-1.1.1.3 ECFT-4.1.1.1

ISS Community Perception w as that Program Pressure w as to maximize

drink bags leak. crew time on orbit for utilization.

ECFT-1.1.1.1.1 ECFT-1.1.1.2.1

I-1
 National Aeronautics and Space Administration EVA23Water Intrusion

EVA crew member (EV2) exposed to

potential loss of life during EVA 23.

UO-1

of Flight Control Team/Crew did not

terminate EVA as soon as w ater w as
reported in the helmet.
ECFT-3

EVA 23 Crew did not immediately Ground Team did not immediately
recognize the severity of the event recognize severity of the event.

ECFT-3.1 ECFT-3.2

FCT focused on the drink bag as the Crew Member Training did not include Critical Information w as not Airflow Contamination Procedure did EV2 did not have the experinence
source of the w ater. this failure mode. communicated betw een Crew and not address the failure mode. base to recognize the severity of the
Ground Team situation.
ECFT-3.1.1 ECFT-3.1.2 ECFT-3.1.3 ECFT-3.1.4 ECFT-3.1.5

Team had Channelized Attentioned on Team's set of responses led to a delay MOD did not understand the failure
the drink bag as the primary source of in identifying the vent loop as the mode.
w ater in the helmet source of the w ater leak.
ECFT-3.1.1.1 ECFT-3.1.1.2 ECFT-3.1.2.1

FCT/Crew incorrectly attributed Water EMU Hazard Report did not identify the Minor amounts of w ater in the helmet Engineering Team did not understand Safety Team did not understand the
in helmet during EVA 22 to the drink hazard. w as normalized. the failure mode. failure mode.
bag.
ECFT-3.1.1.1.1 ECFT-3.1.2.1.1 ECFT-3.1.2.1.2 ECFT-3.1.2.1.3 ECFT-3.1.2.1.4

Flight Control Team did not perform FMEA/CIL did not effectively describe FMEA/CIL did not effectively quantify No one applied know ledge of the No one applied know ledge of the Minimal Formal Training on EMU
investigation of w ater source. the behavior of w ater entering the the amounts of w ater entering the vent physics of w ater behavior in zero g to physics of w ater behavior in zero g to function existed for the Safety Team.
vent loop from the PLSS. loop from the PLSS. w ater coming from the PLSS vent loop w ater coming from the PLSS vent loop
ECFT-3.1.1.1.1.1 ECFT-3.1.2.1.1.1 ECFT-3.1.2.1.1.2 ECFT-3.1.2.1.3.1 ECFT-3.1.2.1.4.1 ECFT-3.1.2.1.4.2

No one applied our know ledge of the FMEA/CIL did not undergo thorough Requirement for specific EVA/EMU
physics of w ater behavior in zero g to review and update periodically. trianing of Safety personnel did not
w ater coming from the PLSS vent loop exist.
ECFT-3.1.2.1.1.1.1 ECFT-3.1.2.1.1.1.2 ECFT-3.1.2.1.4.2.1

ISS program FMEA/CIL requirement did Program cut funding.

not require complete FMEA/CIL periodic
update.
ECFT-3.1.2.1.1.1.2.1 ECFT-3.1.2.1.1.1.2.2

Community had lost sight of the value

of the FMEA/CIL effort.

ECFT-3.1.2.1.1.1.2.2.1

I-2
 National Aeronautics and Space Administration EVA23Water Intrusion

EVA crew member (EV2) exposed to

potential loss of life during EVA 23.

UO-1

f Flight Control Team/Crew did not

terminate EVA as soon as w ater w as
reported in the helmet.
ECFT-3

EVA 23 Crew did not immediately Ground Team did not immediately
recognize the severity of the event recognize severity of the event.

ECFT-3.1 ECFT-3.2

FCT focused on the drink bag as the Crew Member Training did not include Critical Information w as not Airflow Contamination Procedure did EV2 did not have the experinence
source of the w ater. this failure mode. communicated betw een Crew and not address the failure mode. base to recognize the severity of the
Ground Team situation.
ECFT-3.1.1 ECFT-3.1.2 ECFT-3.1.3 ECFT-3.1.4 ECFT-3.1.5

EV1 initially focused on drink bag as Flight Rule to address this FM did not
source in EVA 23. exist.

ECFT-3.1.3.1 ECFT-3.1.4.1

Crew incorrectly attributed w ater in Crew w as not trained to recognize MOD did not understand the failure
EV2's helmet after EVA 22 to the drink this failure mode. mode.
bag.
ECFT-3.1.3.1.1 ECFT-3.1.3.1.2 ECFT-3.1.4.1.1

EMU Hazard Report did not identify the Minor amounts of w ater in the helmet Engineering Team did not understand Safety Team did not understand the
hazard. w as normalized. the failure mode. failure mode.

ECFT-3.1.4.1.1.1 ECFT-3.1.4.1.1.2 ECFT-3.1.4.1.1.3 ECFT-3.1.4.1.1.4

FMEA/CIL did not effectively describe FMEA/CIL did not effectively quantify No one applied know ledge of the No one applied know ledge of the Minimal Formal Training on EMU
the behavior of w ater entering the the amounts of w ater entering the vent physics of w ater behavior in zero g to physics of w ater behavior in zero g to function existed for the Safety Team.
vent loop from the PLSS. loop from the PLSS. w ater coming from the PLSS vent loop w ater coming from the PLSS vent loop
ECFT-3.1.4.1.1.1.1 ECFT-3.1.4.1.1.1.2 ECFT-3.1.4.1.1.3.1 ECFT-3.1.4.1.1.4.1 ECFT-3.1.4.1.1.4.2

No one applied our know ledge of the FMEA/CIL did not undergo thorough Requirement for specific EVA/EMU
physics of w ater behavior in zero g to review and update periodically. trianing of Safety personnel did not
w ater coming from the PLSS vent loop exist.
ECFT-3.1.4.1.1.1.1.1 ECFT-3.1.4.1.1.1.1.2 ECFT-3.1.4.1.1.4.2.1

ISS program FMEA/CIL requirement did Program cut funding.

not require complete FMEA/CIL periodic
update.
ECFT-3.1.4.1.1.1.1.2.1 ECFT-3.1.4.1.1.1.1.2.2

Community had lost sight of the value

of the FMEA/CIL effort.

ECFT-3.1.4.1.1.1.1.2.2.1

I-3
 National Aeronautics and Space Administration EVA23Water Intrusion

EVA crew member (EV2) exposed to

potential loss of life during EVA 23.

UO-1

of Flight Control Team/Crew did not F

terminate EVA as soon as w ater w as A
reported in the helmet.
ECFT-3

EVA 23 Crew did not immediately Ground Team did not immediately
recognize the severity of the event recognize severity of the event.

ECFT-3.1 ECFT-3.2

Flight Control Team did not understand Ground Team focused on the drink bag Critical Information w as not
the failure mode. as the source of the w ater. communicated betw een Crew and
Ground Team
ECFT-3.2.1 ECFT-3.2.2 ECFT-3.2.3

EMU Hazard Report did not identify the Engineering Team did not understand Safety Team did not understand the
hazard. the failure mode. failure mode.

ECFT-3.2.1.1 ECFT-3.2.1.2 ECFT-3.2.1.3

FMEA/CIL did not effectively describe FMEA/CIL did not effectively quantify No one applied know ledge of the No one applied our know ledge of the Minimal Formal Training on EMU
the behavior of w ater entering the the amounts of w ater entering the vent physics of w ater behavior in zero g to physics of w ater behavior in zero g to function existed for the Safety Team.
vent loop from the PLSS. loop from the PLSS. w ater coming from the PLSS vent loop w ater coming from the PLSS vent loop
ECFT-3.2.1.1.1 ECFT-3.2.1.1.2 ECFT-3.2.1.2.1 ECFT-3.2.1.3.1 ECFT-3.2.1.3.2

No one applied our know ledge of the FMEA/CIL did not undergo thourough Requirement for specific EVA/EMU
physics of w ater behavior in zero g to review and update periodically. trianing of Safety personnel did not
w ater coming from the PLSS vent loop exist.
ECFT-3.2.1.1.1.1 ECFT-3.2.1.1.1.2 ECFT-3.2.1.3.2.1

ISS program FMEA/CIL requirement did Program cut funding

not require complete FMEA/CIL periodic
update.
ECFT-3.2.1.1.1.2.1 ECFT-3.2.1.1.1.2.2

Community had lost sight of the value

of the FMEA/CIL effort.

ECFT-3.2.1.1.1.2.2.1

I-4
 National Aeronautics and Space Administration EVA23Water Intrusion

EVA crew member (EV2) exposed to

potential loss of life during EVA 23.

UO-1

Flight Control Team/Crew did not F

terminate EVA as soon as w ater w as A
reported in the helmet.
ECFT-3

Ground Team did not immediately

recognize severity of the event.

ECFT-3.2

nd Ground Team focused on the drink bag Critical Information w as not

as the source of the w ater. communicated betw een Crew and
Ground Team
ECFT-3.2.2 ECFT-3.2.3

The Team had Channelized Attention Team's set of responses led to a delay Ground team did not identify PLSS as Cognitive Task Oversaturation
on the drink bag as the primary source in identifying the vent loop as the actual source of w ater. contributed to the teams delayed
of w ater in the helmet source of the w ater leak. identification of the actual source of
ECFT-3.2.2.1 ECFT-3.2.2.2 ECFT-3.2.2.3 ECFT-3.2.2.4

FCT/Crew incorrectly attributed Water Critical Information w as not

in helmet during EVA 22 to the DIDB. communicated betw een Crew and
Ground Team
ECFT-3.2.2.1.1 ECFT-3.2.2.3.1

Flight Control Team did not perform EV1 initially focused on drink bag as
investigation of w ater source. source in EVA 23.

ECFT-3.2.2.1.1.1 ECFT-3.2.2.3.1.1

Flight Control Team accepted the Ground team allow ed time pressures Flight Control Team's perception of the Crew incorrectly attributed w ater in Crew w as not trained to recognize
explanation that the w ater w as from of impending EVA to influence actions. anomaly report process as being EV2's helmet after EVA 22 to the drink this failure mode.
the drink bag. resource intensive made them bag..
ECFT-3.2.2.1.1.1.1 ECFT-3.2.2.1.1.1.2 ECFT-3.2.2.1.1.1.3 ECFT-3.2.2.3.1.1.1 ECFT-3.2.2.3.1.1.2

ISS Community Perception w as that Program emphasis w as to maximize Flight Control Team did not perform MOD did not understand the failure
drink bags leak. crew time on orbit for utilization. investigation of w ater source. mode.

ECFT-3.2.2.1.1.1.1.1 ECFT-3.2.2.1.1.1.2.1 ECFT-3.2.2.3.1.1.1.1 ECFT-3.2.2.3.1.1.2.1

Flight Control Team accepted the Ground team allow ed pressures of FCT's perception of the anomaly report EMU Hazard Report did not identify the Minor amounts of w ater in the helmet Engineering Team did not understand Safety Team did not understand the
explanation that the w ater w as from impending EVA to influence actions. process as being resource intensive hazard. w as normalized. the failure mode. failure mode.
the drink bag. made them reluctant to invoke it.
ECFT-3.2.2.3.1.1.1.1.1 ECFT-3.2.2.3.1.1.1.1.2 ECFT-3.2.2.3.1.1.1.1.3 ECFT-3.2.2.3.1.1.2.1.1 ECFT-3.2.2.3.1.1.2.1.2 ECFT-3.2.2.3.1.1.2.1.3 ECFT-3.2.2.3.1.1.2.1.4

ISS Community Perception w as that Program Pressure w as to maximize FMEA/CIL did not effectively describe FMEA/CIL did not effectively quantify No one applied know ledge of the No one applied know ledge of the Minimal Formal Training on EMU
drink bags leak. crew time on orbit for utilization. the behavior of w ater entering the the amounts of w ater entering the vent physics of w ater behavior in zero g to physics of w ater behavior in zero g to function existed for the Safety Team.
vent loop from the PLSS. loop from the PLSS. w ater coming from the PLSS vent loop w ater coming from the PLSS vent loop
ECFT-3.2.2.3.1.1.1.1.1.1 ECFT-3.2.2.3.1.1.1.1.2.1 ECFT-3.2.2.3.1.1.2.1.1.1 ECFT-3.2.2.3.1.1.2.1.1.2 ECFT-3.2.2.3.1.1.2.1.3.1 ECFT-3.2.2.3.1.1.2.1.4.1 ECFT-3.2.2.3.1.1.2.1.4.2

No one applied our know ledge of the FMEA/CIL did not undergo thorough Requirement for specific EVA/EMU
physics of w ater behavior in zero g to review and update periodically. trianing of Safety personnel did not
w ater coming from the PLSS vent loop exist.
ECFT-3.2.2.3.1.1.2.1.1.1.1 ECFT-3.2.2.3.1.1.2.1.1.1.2 ECFT-3.2.2.3.1.1.2.1.4.2.1

ISS program FMEA/CIL requirement did Program cut funding.

not require complete FMEA/CIL periodic
update.
ECFT-3.2.2.3.1.1.2.1.1.1.2.1 ECFT-3.2.2.3.1.1.2.1.1.1.2.2

Community had lost sight of the value

of the FMEA/CIL effort.

ECFT-3.2.2.3.1.1.2.1.1.1.2.2.1

I-5
 National Aeronautics and Space Administration EVA23Water Intrusion

EVA crew member (EV2) exposed to

potential loss of life during EVA 23.

UO-1

Flight Control Team/Crew did not F

terminate EVA as soon as w ater w as A
reported in the helmet.
ECFT-3

EVA 23 Crew did not immediately Ground Team did not immediately
recognize the severity of the event recognize severity of the event.

ECFT-3.1 ECFT-3.2

Flight Control Team did not understand Ground Team focused on the drink bag Critical Information w as not
the failure mode. as the source of the w ater. communicated betw een Crew and
Ground Team
ECFT-3.2.1 ECFT-3.2.2 ECFT-3.2.3

EV1 initially focused on drink bag as

source in EVA 23.

ECFT-3.2.3.1

Crew attributed w ater in EV2's helmet Crew w as not trained to recognize

after EVA 22 to the drink bag. this failure mode.

ECFT-3.2.3.1.1 ECFT-3.2.3.1.2

Flight Control Team did not perform Crew Member Training did not include
investigation of w ater source. this failure mode.

ECFT-3.2.3.1.1.1 ECFT-3.2.3.1.2.1

Flight Control Team accepted the Ground team allow ed pressures of FCT's perception of the anomaly report MOD did not understand the failure
explanation that the w ater w as from impending EVA to influence actions. process as being resource intensive mode.
the drink bag. made them reluctant to invoke it.
ECFT-3.2.3.1.1.1.1 ECFT-3.2.3.1.1.1.2 ECFT-3.2.3.1.1.1.3 ECFT-3.2.3.1.2.1.1

ISS Community Perception w as that Program Pressure w as to maximize EMU Hazard Report did not identify the Minor amounts of w ater in the helmet Engineering Team did not understand Safety Team did not understand the
drink bags leak. crew time on orbit for utilization. hazard. w as normalized. the failure mode. failure mode.

ECFT-3.2.3.1.1.1.1.1 ECFT-3.2.3.1.1.1.2.1 ECFT-3.2.3.1.2.1.1.1 ECFT-3.2.3.1.2.1.1.2 ECFT-3.2.3.1.2.1.1.3 ECFT-3.2.3.1.2.1.1.4

FMEA/CIL did not effectively describe FMEA/CIL did not effectively quantify No one applied know ledge of the No one applied know ledge of the Minimal Formal Training on EMU
the behavior of w ater entering the the amounts of w ater entering the vent physics of w ater behavior in zero g to physics of w ater behavior in zero g to function existed for the Safety Team.
vent loop from the PLSS. loop from the PLSS. w ater coming from the PLSS vent loop w ater coming from the PLSS vent loop
ECFT-3.2.3.1.2.1.1.1.1 ECFT-3.2.3.1.2.1.1.1.2 ECFT-3.2.3.1.2.1.1.3.1 ECFT-3.2.3.1.2.1.1.4.1 ECFT-3.2.3.1.2.1.1.4.2

No one applied our know ledge of the FMEA/CIL did not undergo thorough Requirement for specific EVA/EMU
physics of w ater behavior in zero g to review and update periodically. trianing of Safety personnel did not
w ater coming from the PLSS vent loop exist.
ECFT-3.2.3.1.2.1.1.1.1.1 ECFT-3.2.3.1.2.1.1.1.1.2 ECFT-3.2.3.1.2.1.1.4.2.1

ISS program FMEA/CIL requirement did Program cut funding.

not require complete FMEA/CIL periodic
update.
ECFT-3.2.3.1.2.1.1.1.1.2.1 ECFT-3.2.3.1.2.1.1.1.1.2.2

Community had lost sight of the value

of the FMEA/CIL effort.

ECFT-3.2.3.1.2.1.1.1.1.2.2.1

I-6