Commands to troubleshoot connectivity through a vShield Edge

Packet Capture

debug packet display interface INTERFACE host_EXTERNALIP-OF-DESTINATION-EDGE_and_tcp_port_PORT eg:
debug packet display interface any host_11.22.33.44_and_tcp_port_80
debug packet display interface vNic_0 udp
debug packet display interface vNic_0 icmp
debug packet display interface vNic_0 host_10.10.10.10
debug packet display interface vNic_0 tcp_src_port_53
debug packet display interface vNic_0 host_10.10.10.10
debug packet display interface any host_10.10.10.10_or_host_11.22.33.44

Useful Commands

list (lists all available commands)
show log follow
show tech-support
show nat
show flowtable topN 5NAT
show ip route

show system cpu
show system memory
show system network-stats
show system storage
show system uptime
show tech-support
show version

Ping external address from vShield Edge console (Note*** must use IP that’s assigned to edge)

ping interface addr <external IP> 8.8.8.8
ping interface addr 11.22.33.44 8.8.8.8

Scroll within a vShield Edge console session

SHIFT + [PgUp / PgDn]

Commands to troubleshoot connectivity through a Cisco ASA

Packet Tracer

packet-tracer input <INTERFACE> <PROTOCOL> <SOURCE-IP> <SOURCE-PORT> <DESTINATION-IP> <DESTINATION-PORT>
packet-tracer input external tcp 11.22.33.44 1010 55.66.77.88 80

VPN Example

packet-tracer input internal udp 10.10.10.10 500 10.20.20.20 500
packet-tracer input internal udp 10.20.20.20 500 10.10.10.10 500

Packet Capture

capture *NAME_OF_CAPTURE* interface *INTERFACE_NAME* match *TRANSPORT_PROTOCOL (TCP / UDP / IP)* host X.X.X.X (This is the source IP address) host Y.Y.Y.Y (This is the destination IP address) eq *PORT_NUMBER*
capture cap1 interface external match tcp host 11.22.33.44 host 55.66.77.88 eq 80

Capture all syslog traffic to 10.10.10.10

capture cap2 interface internal real-time match udp any host 10.10.10.10 eq 514

View capture

show capture cap1

Show access-list

show access-list acl_external

Find access-list entries including port 514

show access-list | inc 514

Find log entries including port 514

show logging | inc 514

Show arp for specific interface

show arp | inc internal

SSPI handshake failed with error code 0x80090308, state 14

Problem

You cannot login to SQL Server, and see the following error in the logs:

SSPI handshake failed with error code 0x80090308, state 14 while establishing a connection with integrated security; the connection has been closed. Reason: AcceptSecurityContext failed. The Windows error code indicates the cause of failure.  [CLIENT: [IP ADDRESS]].

Solution

There can be several reasons for this error, so rather than listing them all, read the following useful links:

--Show details of all connections
select auth_scheme, * from sys.dm_exec_connections

SQL Server full and transaction log backups completing but differential backups failing because of Veeam B&R configuration

Scenario

I recently refreshed our SQL Server Maintenance Plans, but System Centre Essentials has been intermittently reporting failures of the daily differential backups, with an error like this:

Executing the query “BACKUP DATABASE [DBNAME] TO DISK = N’…” failed with the following error: “Cannot perform a differential backup for database “DBNAME”, because a current database backup does not exist. Perform a full database backup by reissuing BACKUP DATABASE, omitting the WITH DIFFERENTIAL option. BACKUP DATABASE is terminating abnormally.”. Possible failure reasons: Problems with the query, “ResultSet” property not set correctly, parameters not set correctly, or connection not established correctly.

The strange thing is that the logs show all Weekly Full backups have completed without error. If I run a manual Full Backup, the next Differential Backup will complete, then the following day will fail.

Solution

First, I used the following query to show more details on the backups (make sure you select the DB in SSMS first):

[Read more…]

When good Domain Controllers go bad!

Scenario

It’s a pleasant day and all is well with the world. Colleagues are skipping around the office with smiles on faces…until…duh duh daaa! One by one, services start failing:

  • Printers go offline:
    • First, for Win7 users
    • Then for all clients
    • Can still print from server though
  • File shares go offline
  • Active Directory replication fails
  • DNS console will not open

Basically, your main Domain Controller (DC) has just taken a dump…and so have you!

These are the steps I took to troubleshoot the issues and get everything back online.

[Read more…]

Hardcore troubleshooting using Logs, Process Monitor, PowerShell and Excel

I’m a huge fan of the Sysinternals Suite – Process Monitor being one of my faves – and am loving using PowerShell to do ever more, but this post is at another level: http://blogs.technet.com/b/askds/archive/2012/06/01/what-s-causing-that-dfsr-change-storm.aspx

Hopefully I’ll be able to use similar techniques for those really tricky problems that rear their ugly head from time to time. Bring it on!

Malware Hunting with the Sysinternals Tools

I’ve just finished watching the best instructional video to date on Malware Hunting with the Sysinternals Tools, part of the latest TechEd 2012 sessions. There is also a similar PDF available for many of these tips.

The Sysinternals Suite has been my one of my favourite toolkits for many years, and although I already use most of the techniques shown in the video above, there were a few new features for me to get stuck into. Highly recommended.

Other TechEd sessions I’ve enjoyed watching include videos from the following speakers:

 

Troubleshooting Slow Boot Slow Logon

Here’s some great posts on troubleshooting slow boot and slow logons:

Troubleshooting GPOs

A great post on troubleshooting Group Policy using event logs: http://technet.microsoft.com/en-us/library/cc749336(v=ws.10).aspx

I can’t remember where I copied the rest of these steps, but I refer to them quite often:

Collect GPMC log

1. On domain controller, click Start -> Run, type GPMC.MSC, it will load the GPMC console. If the GPMC snap-in is not installed.

2. Right click on “Group Policy Result” and choose wizard to generate a report for the problematic computer and user account (please place appropriately). (Choose computer and select the proper user in the wizard)

3. Right click the resulting group policy result and click the “Save Report…” => save report and upload it to the link I provided.

Collect gpresult output

1. Run the command gpresult /v >C:\gpresult.txt.

2. Upload the C:\gpresult.txt to the above link.

Collect Userenv.log

Subkey: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon

Entry: UserEnvDebugLevel

Type: REG_DWORD

Value data: 0x00030002 (Hexadecimal)

After the issue reoccurs, find and upload %windir%\Debug\UserMode\Userenv.log file.

Collect fdeploy.log

Subkey: Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Diagnostics

Entry: FdeployDebugLevel

Type: REG_DWORD

Value data: 0x0000000F (Hexadecimal)

The log file can be found at: %Systemroot%\debug\usermode\fdeploy.log.

Enable Group Policy Preference logging and tracing

Scenario

You’ve created and linked a new GPO that uses Group Policy Preferences to move some files. You’ve had this working before, but now find that the files aren’t being copied across on certain clients.

The Resultant Set of Policy (RSOP) tool does not show GP Preferences, so you need another way of troubleshooting this. Enter Group Policy Preference logging and tracing.

Follow the steps below to enable the GP Preferences log file(s):

[Read more…]