NetScreen Troubleshooting Tools

Objectives

• Discuss the troubleshooting process

• Use ScreenOS commands to gather information for
troubleshooting
• Test connectivity using ping and trace-route
• Use the snoop and debug troubleshooting tools
• Configure filter and output options for
troubleshooting tools
Troubleshooting Methodology

Gather Form Apply Problem

YES DONE
Information Hypothesis Solution solved?

NO

YES More
theories?

NO
 Troubleshooting Methodologies

• Far-end focus
– Start at destination point in network and work back to the source
• Near-end focus
– Start at source point in network and work out to destination
• Working the OSI layers
– Start at Layer 1 and work up
 Documentation is Essential!

• Provides record of ongoing problems

– Intermittent problems impossible to troubleshoot without records
• Documentation required by NetScreen TAC for
problem reporting
• For partners, record of your work for both customer
and NetScreen
Example Network Topology

10.1.10.0/24 Private External

.1 .254
Zone Zone
A D
.254
10.1.10.5 10.1.1.0/24 200.5.5.5
e1 1.1.8.0/24
e8

e2
e7
10.1.20.0/24
10.1.2.0/24
1.1.7.0/24
1.1.70.0/24
B .1 .254
.254 .1
10.1.20.5
Public C
B
Zone 1.1.70.250
Interface Address
E1 10.1.1.1
E2 10.1.2.1
E7 1.1.7.1
E8 1.1.8.1
get system
ns208-> get sys
Product Name: NS208
Serial Number: 0043042002000115, Control Number: 00000000
Hardware Version: 0110(0)-(11), FPGA checksum: 00000000, VLAN1 IP (0.0.0.0)
Software Version: 5.1.0r1.0, Type: Firewall+VPN
Base Mac: 0010.db1d.2140
File Name: ns200.5.1.0r1.0, Checksum: 634852f6

Date 10/19/2004 09:27:15, Daylight Saving Time enabled

The Network Time Protocol is Disabled
Up 406 hours 44 minutes 56 seconds Since 2 Dec 2003 10:42:19
Total Device Resets: 1, Last Device Reset at: 10/13/2004 08:32:38

System in NAT/route mode.

Use interface IP, Config Port: 80

User Name: netscreen

Interface ethernet1:
number 0, if_info 0, if_index 0, mode route
link down, phy-link down
vsys Root, zone Untrust, vr trust-vr
get interface
ns208-> get interface

A - Active, I - Inactive, U - Up, D - Down, R - Ready

Interfaces in vsys Root:

Name IP Address Zone MAC VLAN State VSD
eth1 1.1.1.1/24 Group1 0010.db1d.1be0 - U -
eth2 1.1.2.1/24 Group2 0010.db1d.1be4 - D -
<output omitted>

ns208-> get int e1

Interface ethernet1:
number 0, if_info 0, if_index 0, mode route
link down, phy-link down
vsys Root, zone Untrust, vr trust-vr
dhcp client disabled
PPPoE disabled
*ip 1.1.1.10/24 mac 0010.db1d.2140
*manage ip 1.1.1.10, mac 0010.db1d.2140
ping enabled, telnet enabled, SSH enabled, SNMP enabled
web enabled, ident-reset disabled, SSL enabled
webauth disabled, webauth-ip 0.0.0.0
OSPF disabled BGP disabled RIP disabled
bandwidth: physical 0kbps, configured 0kbps, current 0kbps
total configured gbw 0kbps, total allocated gbw 0kbps
DHCP-Relay disabled
DHCP-server disabled
get zone
ns5xt-> get zone
Total 10 zones created in vsys Root - 5 are policy configurable.
Total policy configurable zones for Root is 5.
------------------------------------------------------------------------
ID Name Type Attr VR Default-IF VSYS
0 Null Null Shared untrust-vr hidden Root
1 Untrust Sec(L3) Shared trust-vr untrust Root
2 Trust Sec(L3) trust-vr trust Root
4 Self Func trust-vr self Root
5 MGT Func trust-vr null Root
10 Global Sec(L3) trust-vr null Root
11 V1-Untrust Sec(L2) trust-vr v1-untrust Root
12 V1-Trust Sec(L2) trust-vr v1-trust Root
14 VLAN Func trust-vr vlan1 Root
16 Untrust-Tun Tun trust-vr hidden.1 Root
------------------------------------------------------------------------
ns5xt-> get zone untrust
Zone name: Untrust, id: 1, type: Security(L3), vsys: Root, vrouter:trust-vr
Intra-zone block: On, attrib: Shared, flag:0x6491
TCP non SYN send reset: Off
IP/TCP reassembly for ALG on traffic from/to this zone: No
Policy Configurable: Yes
Interfaces bound:1. Designated ifp is untrust
interface untrust(0x1591340)
IP classification disabled
 get route

ns208-> get route

C - Connected, S - Static, A - Auto-Exported, I - Imported
iB - IBGP, eB - EBGP, R - RIP, O - OSPF, E1 - OSPF external type 1
E2 - OSPF external type 2

untrust-vr (0 entries)
======================

trust-vr (8 entries)
======================
ID IP-Prefix Interface Gateway P Pref Mtr Vsys

------------------------------------------------------------------------------
* 9 0.0.0.0/0 eth8 1.1.8.254 S 20 1 Root

* 8 1.1.70.0/24 eth7 1.1.7.254 S 20 1 Root

* 7 10.1.20.0/24 eth2 10.1.2.254 S 20 1 Root

* 2 10.1.1.0/24 eth1 0.0.0.0 C 0 0 Root

* 3 10.1.2.0/24 eth2 0.0.0.0 C 0 0 Root

<output omitted>
get route (cont.)

ns208-> get route ip 10.1.10.5

Destination Routes for 10.1.10.5
---------------------
trust-vr : => 10.1.10.0/24 (id=6) via 10.1.1.254 (vr: trust-vr)
Interface ethernet1 , metric 1

ns5xt-> get route id 2

route in vr trust-vr:
--------------------------
id: 2
IP address/mask: 1.1.1.10/24
next hop (gateway): 0.0.0.0
preference: 0
metric: 0
outgoing interface: untrust
vsys name/id: Root/0
tag: 0
flag: 00000200/00000008
type: connected
status: active (for 2 minutes 45 seconds)
get arp

ns208-> get arp

IP Mac VR/Interface State Age Retry PakQue
3.4.5.6 abc3241244dc trust-vr/trust STS 0 0
1.1.7.250 00065bd2ff42 trust-vr/trust VLD 1151 0 0
ARP Entry Number 2/1024, No Free Entry Count: 0
Arp always-on-dest: disabled
ns208->
Ping Options

ns208-> ping 10.1.10.5

Type escape sequence to abort

Sending 5, 100-byte ICMP Echos to 10.1.10.5, timeout is 2 seconds

!!!!!
Success Rate is 100 percent (5/5), round-trip time min/avg/max=2/3/9 ms

ns208-> ping
Target IP address:10.1.10.5
Repeat count [5]:
Datagram size [100]:
Timeout in seconds[2]:
Source interface:
Type escape sequence to abort

Sending 5, 100-byte ICMP Echos to 10.1.10.5, timeout is 2 seconds

!!!!!
Success Rate is 100 percent (5/5), round-trip time min/avg/max=2/3/4 ms
 Trace-route

10.1.10.0/24 1.1.70.0/24
.1 .254 .1 .1 .254 .1
A e1 e7 C
10.1.1.0/24 1.1.7.0/24 B
10.1.10.5 1.1.70.250

ns5xt-> trace-route 1.1.70.250

Type escape sequence to escape
Send ICMP echos to 1.1.70.250, timeout is 2 seconds, maximum hops are 32
1 2ms 15ms 1ms 10.1.1.1
2 6ms 2ms 2ms 1.1.7.254
3 4ms 2ms 2ms 1.1.70.250
Trace complete
ns5xt->
 Troubleshooting Tools

• Get session
• Output options
• Debug utility
• Snoop utility
 get session

• Basic session
ns208-> get session
alloc 5/max 128000, alloc failed 0
id 78/s**,vsys 0,flag 00000040/80/20,policy -1,time 89
0(21):10.1.10.5/4647->200.5.5.5/80,6,0010db12cea1,vlan 0,tun 0,vsd 0
3(00):10.1.10.5/4647<-200.5.5.5/80,6,000000000000,vlan 0,tun 0,vsd 0

• Session with translation

ns208-> get session
alloc 2/max 128000, alloc failed 0
id 38/s**,vsys 0,flag 04000010/00/00,policy 1,time 1
0(01):10.1.10.5/7936->200.5.5.5/512,1,0010db12cea1,vlan 0,tun 0,vsd 0
10(20):1.1.8.15/7936<-200.5.5.5/512,1,0010db21c041,vlan 0,tun 0,vsd 0
 Debug Utility

• Overview
– Utility to view internal operations of NetScreen devices
• Common debug types
– Debug flow basic
– Debug nat
– Debug arp
– Debug DHCP
– Debug ike detail
– Debug pki detail
• Output to debug buffer by default
 Debug Buffer

• Get dbuf info

– Displays debug buffer size in bytes
• Get dbuf stream
– Displays the contents of the debug buffer
• Get dbuf mem
– Hex dump of debug buffer contents w/ decode displayed to the right
of the output
• Set dbuf size
– Allocates system memory for the debug buffer
• Clear dbuf
– Clears the contents of the debug buffer
 Troubleshooting Tools – Output Options

• Debug buffer (dbuf)

– Default
– Most efficient method (recommended)
• Direct to console
unset console dbuf

ns5xt-> get console

Console timeout: 0(minute), Page size: 22/22, debug: buffer
privilege 250, config was changed and not saved!
ID State Duration Task Type Host
0 Logout 0 6824 Local
1 Login 25607 5120 Local
ns5xt-> unset console dbuf
ns5xt-> get console
Console timeout: 0(minute), Page size: 22/22, debug: console
privilege 250, config was changed and not saved!
ID State Duration Task Type Host
0 Logout 0 6824 Local
1 Login 25683 5120 Local
Packet Flow Review

Packet Packet
Received intact?
Forward
No
Yes packet

Existing Destination
No No Drop packet
Session? reachable?

Yes Create
No Yes session
entry
Yes Permitted by
policy?

ARP for Resolve

Yes
destination? address
Forward
packet
No
 Debug Procedure

1. debug flow basic

2. clear db
3. Begin testing
• Ping
• Traceroute
• Other traffic
4. undebug all
5. get dbuf stream
 Debug Flow Basic - Example

• Host C (1.1.70.250) pings host A (10.1.10.5)

10.1.10.0/24 Private External

.1 .254
Zone Zone
A D
.254
10.1.10.5 10.1.1.0/24 200.5.5.5
e1 e81.1.8.0/24

e7
e2
10.1.20.0/24
10.1.2.0/24
1.1.7.0/24
1.1.70.0/24
B .1 .254
.254 .1
10.1.20.5
Public C
B
Zone 1.1.70.250
 Debug Flow Basic Output

ns208-> get db stream

****** 06519.0: <public/ethernet7> packet received [128]******
ipid = 1843(0733), @d7816910

1 packet passed sanity check.

ethernet7:1.1.70.250/2000->1.1.7.250/1024,1(8/0)<Root>
2 chose interface ethernet1 as incoming nat if.
search route to (ethernet7, 1.1.70.250->10.1.10.5) in vr trust-vr for vsd-
0/flag-0/ifp-null
[Dest] 5.route 10.1.10.0->0.0.0.0, to ethernet1

3 routed (10.1.10.5, 0.0.0.0) from ethernet7 (ethernet7 in 0) to ethernet1

4 policy search from zone 1000-> zone 1003
No SW RPC rule match, search HW rule
5 Permitted by policy 288066
Debug Flow Basic Output (cont.)

No src xlate choose interface ethernet1 as outgoing phy if

no loop on ifp ethernet1.
session application type 0, name None, timeout 60sec
service lookup identified service 0.
existing vector list 1-6005700.
6 Session (id:998) created for first pak 1
route to 10.1.10.5

7 arp entry found for 10.1.10.5

nsp2 wing prepared, ready
cache mac in the session
flow got session.
flow session id 998
8 post addr xlation: 1.1.70.250->10.1.10.5
 Debug Flow Basic – Existing Session

• Not visible on ISG-2000 or 5000 series

****** 06519.0: <public/ethernet7> packet received [128]******

ipid = 1843(0733), @d7817110
packet passed sanity check.
ethernet7:1.1.70.250/2000->10.1.10.5/1024,1(8/0)<Root>
existing session found. sess token 14
flow got session.
flow session id 998
post addr xlation: 1.1.70.250->10.1.10.5.
No Route to Destination

ns208-> get db stream

****** 88622.0: <Public/ethernet7> packet received [60]******
ipid = 105(0069), @d78dc070
packet passed sanity check.
ethernet7:1.1.70.250/1280->10.1.10.5/512,1(8/0)<Root>
chose interface ethernet7 as incoming nat if.
IP classification from non-shared src if : vsys Root
search route to (1.1.70.250->10.1.10.5) in vr trust-vr for 0/0
packet dropped: no route to (1.1.70.250->10.1.10.5) in vr trust-vr/0
packet dropped, no route
 Denied by Policy

ns208-> get db stream

****** 88939.0: <Public/ethernet7> packet received [60]******
ipid = 117(0075), @d78c0070
packet passed sanity check.
ethernet7:1.1.70.250/3328->10.1.10.5/512,1(8/0)<Root>
chose interface ethernet7 as incoming nat if.
IP classification from non-shared src if : vsys Root
search route to (1.1.70.250->10.1.10.5) in vr trust-vr for 0/0
route 10.1.10.5->10.1.1.254, to ethernet1
routed (10.1.10.5, 0.0.0.0) from ethernet7 (ethernet7 in 0) to ethernet1
IP classification from non-shared dst if : vsys Root
policy search from zone 1002-> zone 1000
vsys Root: ethernet7->ethernet1, policy zone 1002->1000(1000), 1.1.70.250-
>10.1.10.5
Searching global policy.
packet dropped, denied by policy
 Set Flow Filter

• Filter provide more selective output

• Set ffilter
– IP address (source or destination)
– IP protocol number
– TCP/UDP port number (source or destination)

ns208-> set ffilter ?

<return>
dst-ip flow filter dst ip
dst-port flow filter dst port
ip-proto flow filter ip proto
src-ip flow filter src ip
src-port flow filter src port
 Flow Filter Options

• Logical AND
– Enter options on the same line
– All conditions must be present
ns208-> set ffilter src-ip 1.1.7.250 dst-ip 10.1.10.5 ip-prot 6
filter added

• Logical OR
– Options entered on separate lines
– Any condition may be present
ns208-> set ffilter src-ip 1.1.7.250 dst-ip 10.1.10.5 ip-prot 6
filter added
ns208-> set ffilter src-ip 10.1.1.1
filter added
ns208-> set ffilter dst-ip 1.1.70.1
filter added
ns208-> set ffilter dst-port 80
filter added
 Viewing Flow Filter

• To view flow filters

ns208-> get ffilter
Flow filter based on:
id:0 src ip 1.1.7.250 dst ip 10.1.10.5 ip proto 6
id:1 src ip 10.1.1.1
id:2 dst ip 1.1.70.1
id:3 dst port 80

• To remove filters
ns208-> unset ffilter 1
filter 1 removed
ns208-> get ffilter
Flow filter based on:
id:0 src ip 1.1.7.250 dst ip 10.1.10.5 ip proto 6
id:1 dst ip 1.1.70.1
id:2 dst port 80
 Snoop Utility

• Allows viewing of packet details

– Layer 2 to layer 4
– Only packets handled by CPU are viewable
• Packets viewed as they enter and leave
• Filters can be applied
• Recommended that output be directed to the debug
buffer
• CPU intensive utility, use with caution
Snoop Enable/Disable

ns208-> snoop info

Snoop: OFF
Filters Defined: 0, Active Filters 0
Detail: OFF, Detail Display length: 96
Ns208>
ns208-> snoop
Start Snoop, type ESC or 'snoop off' to stop, continue? [y]/n y
ns208->
ns208-> snoop info
Snoop: ON
Filters Defined: 0, Active Filters 0
Detail: OFF, Detail Display length: 96

No Snoop filter defined or no active filters present.

Listening on ALL interfaces for BOTH directions.
ns208-> snoop off
Snoop off
ns208->
Snoop Filter Options

ns208-> snoop filter ?

delete delete snoop filter
ethernet snoop specified ethernet
id snoop filter id
ip snoop ip packet
off turn off snoop filter
on turn on snoop filter
tcp snoop tcp packet
udp snoop udp packet
ns208-> snoop filter ethernet ?
arp snoop arp packet
direction snoop direction
interface interface name
nsrp snoop nsrp packet
vlan snoop vlan packet
<number> snoop specified ethernet type
except snoop all but the specified ethernet type
offset ethernet offset
Snoop Settings

ns208-> snoop filter ethernet 0x0800

snoop filter added
ns208-> snoop filter ethernet interface eth1
snoop filter added
ns208-> snoop info
Snoop: OFF
Filters Defined: 2, Active Filters 2
Detail: OFF, Detail Display length: 96
Snoop filter based on:
id 1(on): Ether EtherType 0800 dir(B)
id 2(on): Ether on ethernet1 dir(B)
Snoop Filters - IP

ns208-> snoop filter ip ?

<return>
direction snoop direction
dst-ip snoop filter dst ip
dst-port snoop filter dst port
interface interface name
ip-proto snoop filter ip proto
port src or dst port
src-ip snoop filter src ip
src-port snoop filter src port
<a.b.c.d> IP Address
offset ip offset
ns208-> snoop filter ip dst-ip 10.1.1.254
snoop filter added
ns208-> snoop info
Snoop: OFF
Filters Defined: 3, Active Filters 3
Detail: OFF, Detail Display length: 96
Snoop filter based on:
id 1(on): Ether EtherType 0800 dir(B)
id 2(on): Ether on ethernet1 dir(B)
id 3(on): IP dst-ip 10.1.1.254 dir(B)
Snoop Output Example - ping

ns208-> snoop
Start Snoop, type ESC or 'snoop off' to stop, continue? [y]/n y
ns208-> clear db
ns208-> ping 10.1.1.254
Type escape sequence to abort
Sending 5, 100-byte ICMP Echos to 10.1.1.254, timeout is 2 seconds
!!!!!
Success Rate is 100 percent (5/5), round-trip time min/avg/max=100/100/100 ms
ns208-> get db stream
04470.0: 0(o):0010db19a4e0->0010db3aed41/0800
10.1.1.1->10.1.1.254/1, tlen=128
vhl=45, tos=00, id=1260, frag=0000, ttl=64
icmp:type=8, code=0

04470.0: 0(i):0010db3aed41->0010db19a4e0/0800
10.1.1.254->10.1.1.1/1, tlen=128
vhl=45, tos=00, id=1264, frag=0000, ttl=64
icmp:type=0, code=0
 Snoop Output Example – HTTP

ns208-> snoop info

Snoop: ON
Filters Defined: 1, Active Filters 1
Detail: ON, Detail Display length: 96
Snoop filter based on:
id 1(on): IP port 80 dir(B)
ns208-> clear db
ns208-> get db stream
24471.0: 0(o):0010db19a4e0->0010db3aed41/0800
10.1.1.1->10.1.1.254/6, tlen=44
vhl=45, tos=00, id=5751, frag=0000, ttl=64
tcp:ports 80->1060, seq=3125794986, ack=465589291, flag=6012/SYN
00 10 db 3a ed 41 00 10 db 19 a4 e0 08 00 45 00 ...:.A........E.
00 2c 16 77 00 00 40 06 4d 55 0a 01 01 01 0a 01 .,[email protected]......
01 fe 00 50 04 24 ba 4f d8 aa 1b c0 54 2b 60 12 ...P.$.O....T+`.
10 00 6a 0c 00 00 02 04 05 64 ..j......d
 Summary

• In this module we covered:

– Using get commands to gather system information
– Using ping and trace-route to test connectivity
– Using the debug utility to analyze packet flow through a NetScreen
device
– Using the snoop utility analyze packet contents
 Review Questions

1. Where is the interface number information useful?

2. Give the sequence of events that occurs when the
NetScreen is processing a packet
3. Give the command to set up a debug flow filter that
looks for a packet with a source IP address of
10.1.10.5 carrying an HTTP request
4. What are two ways to turn off snoop?