g& redhat RH142 Linux Troubleshooting Techniques andToolsRH142 Linux Troubleshooting Techniques and Tools RH142-RHELS-0n-020000811Table of Contents RH142 - Linux Troubleshooting Techniques and Tools RH142 Linux Troubleshooting Techniques and Tools Copyright Welcome Participant Introductions Red Hat Enterprise Linux Red Hat Enterprise Linux Variants vii viii xi Red Hat Network xii Other Red Hat Supported Software xitl The Fedora Project xiv Objectives xv Audience and Prerequisites xvi Lab Exercises xvii Classroom Network xviii RH142 Lab System xix Notes on Internationalization xX Unit 1 - An Introduction to Troubleshooting Techniques and Procedures Objectives 2 Cause and Effect 3 Identifying Example: Boot Sequence 4 Techniques for Solving Problems 5 Best Practices 6 Log Analysis 7 Additional Information Gathering 8 Runlevels 9 Pre-emptive Planning W Baseline Monitoring 12 Monitoring with Nagios 13 Disaster Recovery 14 Writing a Plan 15 End of Unit 1 16 Lab 1: Basic troubleshooting practice 17 Sequence 1: Working with your virtual environment 18 Sequence 2: Booting problems 19 Sequence 3: New security policy 20 Unit 2 - Common Troubleshooting Tools Objectives 23 Advanced head and tail 24 Copyright © 2009 Red Hat, inc. RH142-RHELS-on-0-2009031 1, Table of Contents All rights reservedfind Selected find options find and Permissions Extracting Text by Keyword Output parsing Logwatch and Sysiog Rebuilding the RPM database Detecting file changes with aide Verifying with rom End of Unit 2 Lab 2: Working with common troubleshooting tools Sequence 1: Recovering from a corrupt RPM database Sequence 2: Setting up aide Sequence 3: Locating broken files Challenge Sequence 4: Output parsing with awk Unit 3 - Booting Issues Objectives Tools Troubleshooting and GRUB Finding the GRUB root Restoring the MBR using GRUB. Using a GRUB CDROM Advanced Filesystem Repair Filesystem Superblock Repair Dealing with Bad Blocks Advanced Maintenance (debugfs) Recovery of a removed file (ext2 only) Software RAID Recovery LVM Maintenance End of Unit 3 Lab 3: Troubleshooting boot and filesystem problems. Sequence 1: Working with severe filesystem corruption ‘Sequence 2: Booting Problem 1 Sequence 3: Booting Problem 2 Unit 4 - Revision Control Objectives Tools Revision Control Systems Starting with git Branching & Tagging Merging changes Finding problems with git bisect Git documentation Copyright © 2009 Red Hat, inc. All rights reserved 25 27 29 30 32 33 36 37 4 47 48 49 50 52 53 56 59 60 62 63 65 67 69 RH142-RHELS-en-0-200903111, Table of ContentsComparing Files 85 Duplicating File Changes 86 End of Unit 4 87 Lab 4: Working with git 88 Sequence 1: Troubleshooting configuration files with git 89 ‘Sequence 2: Working with git 90 Challenge Sequence 3: Branching & Tagging with git 91 Challenge Sequence 4: Storing your scripts in git 92 Unit 5 - Hardware Objectives 98 Getting hardware info 99 Testing for memory errors 100 Testing for hard drive errors 101 Introduction to sar 102 Working with sar 103 Device Node Examples 104 Managing /dev With udev 105 Adding Files Under /dev 106 Kemel/Userspace Interaction 107 /proc 108 /proc Example: 109 /sys 110 /sys Example: 111 End of Unit S 112 Lab 5: Checking for hardware failures 113 Optional Sequence 1: Working with memtest86+ 114 Optional Sequence 2: Working with smartct! 115 Sequence 3: Working with sar 116 Challenge Sequence 4: Combining sar and awk 117 Unit 6 - Applications Objectives: 122 Tools 123 Classes of Problem 124 Application Failure 125 Application Fails to Start 126 Unresponsive Applications 127 Poor Performance 128 Diagnosing Poor Performance 129 Apache Errors 130 Diagnosing + Fixing Apache Problems 131 Memory Leaks 132 Troubleshooting Memory Leaks 133 Broken Configuration File 134 ‘Copyright © 2009 Red Hat, Inc. RH142-RHELS-en-0-20090311, Table of Contents (Ai rights reserved iiAdvanced Troubleshooting Monitoring the kernel with SystemTap The stap command SystemTap architecture SystemTap script libraries End of Unit 6 Lab 6: Application Troubleshooting Sequence 1: Controlling memory usage Sequence 2: Troubleshooting applications Sequence 3: Troubleshooting applications Sequence 4: Troubleshooting applications Unit 7 - Networking Objectives Tools Network Configuration Virtual Interfaces Network Card Parameters Manually Assigning Static Network Settings Manually Requesting Dynamic Network Settings Multi NIC Systems and Advanced Setups Troubleshooting Networks Connections Connection Checking Packet Monitoring Kernel Tuning Graphical Network and Monitoring Tools Network Security End of Unit 7 Lab 7: Network Troubleshooting Sequence 1: Network troubleshooting ‘Sequence 2: Network troubleshooting Sequence 3: Network troubleshooting Sequence 4: Network troubleshooting Sequence 5: Network troubleshooting Sequence 6: Network troubleshooting Sequence 7: Investigating a Process Unit 8 - Security Objectives Tools Important Log Files ‘SELinux Basics ‘SELinux enforcement Changing file context ‘SELinux Operation ‘Copyright © 2009 Red Hat, Inc. Ailrights reserved 135 136 137 139 140 141 142 143 144 145 152 153 154 155, 158 159 160 161 163 165 167 168 170 71 172 173 174 175 176 177 178 194 195, 196 197 198 199 200 RH142-RHELS-en-0-20090311, Table of Contents wvUnderstanding SELinux errors Fixing SELinux errors Authentication problems Troubleshooting NIS Troubleshooting LDAP PAM Operation /etc/pam.d/ Files: Tests /etc/pam.d/ Files: Control Values Example: /etc/pam.d/login File The system-auth file PAM Troubleshooting End of Unit 8 Lab 8: Troubleshooting security issues Sequence 1: Troubleshooting NIS ‘Sequence 2: Troubleshooting LDAP ‘Sequence 3: More LDAP troubleshooting Sequence 4: Further security troubleshooting Sequence 5: Troubleshooting PAM Unit 9 - Red Hat Resources Objectives Red Hat Resources Knowledge base Production Support Initiating Support Issues Technical Account Manager Developer Support Bugzilla ‘Support Workflow Diagnostic and Information-gathering Tools kexec and kdump Installing kexec and kdump Introduction to Kemel Debugging with kdump End of Unit 9 Lab 9: Working with Red Hat Support Sequence 1: Red Hat Survey ‘Sequence 2: Exploring sosreport Optional Sequence 3: Configuring your system for kexec Optional Sequence 4: Kemel crash simulation and diagnosis Appendix A - Installing Software ‘Software Installation Appendix B - Working with virtual systems Working with virtual systems Copyright © 2000 Red Hat, Inc. Ad rights reserved 210 aii 212 213 214 215 216 217 218 231 232 234 235 237 238 239 240 241 242 243 245 246 247 248 253 256 RH142-RHELS-en-0-2009031 1, Table of Contents v| Introduction RH142 Linux Troubleshooti Hechniquesand Foose & redhat, ‘er une ont by student ered In aid Het vaiing course ugh by Rad Mt nc. of aR at Carte Trang Parte Mo part of hia pubeton may be "doping sored na reterleytam or obarwaeropraduced witout por writen canaent of Red Hint You blared Hat wang ial ‘tr ologtorepery ved, coped r aoe please eral or pane lie UBA) #1 (06) 628 204 #1010) 764 3700. Copyright © 2009 Red Hat, Inc. FH142-RHELS-en-0-20090011, Introduction All rights reserved "| Copyright * The contents of this course and all its modules and related materials, including handouts to audience members, are Copyright © 2008 Red Hat, Inc. + No part of this publication may be stored in a retrieval system, transmitted or reproduced in any way, including, but not limited to, photocopy, photograph, magnetic, electronic or other record, without the prior written permission of Red Hat, Inc. * This instructional program, including all material provided herein, is supplied without any guarantees from Red Hat, ne. Red Hat, Ic: assumes no liabily for damages or legal action arising from the use or misuse of contents or details contained he + Ityou belleve Red Hat training materials are being used, copied, or otherwise improperly distributed please email [email protected] or phone toll-free (USA) +1 866 626 2994 or +1 919 754 3700. <) redhat, Forma no tat oe a at eng ure ahh fi a reo Re it Crog Trang Pct, Nop of a abn Pe [etosopled, Guplonta, sored Youve yates oterele reproduced without peor writen consent of ed Ht ne You aloe Hat waring materia (Tiibeng propery wood copied altrbted plete eal cereistngeredac com> of poe lite (UGA) #1 (06) £28 004 r+ 10) 74 3700. Rareright © 2009 Rest Hal.ine. RH142-RHELS-en-0-2009031 1, introduction Ad rights reserved vil