VMware

VMware Workstation 5.0

Features | Documentation | Knowledge Base | Discussion Forums

previous Prev   Contents   Last   Next next

Examples: Debugging over a Virtual Serial Port

You can use Debugging Tools for Windows (WinDbg) or Kernel Debugger (KD) to debug kernel code in a virtual machine over a virtual serial port. You can download Debugging Tools for Windows from the Windows DDK Web site at www.microsoft.com/whdc/devtools/debugging/default.mspx.

The following two examples illustrate how to use a virtual serial port to debug kernel code in a virtual machine:

  • With the debugging application on the VMware Workstation host (Windows hosts only)
  • With the debugging application in another virtual machine on the same VMware Workstation host (useful on a Linux host and can also be done on a Windows host)
  • Using either of these methods lets you debug kernel code on one system, without the need for two physical computers, a modem or serial cable.

    Debugging an Application in a Virtual Machine from the Windows Host

    In this example, you have kernel code to debug in a virtual machine (called the target virtual machine) and are running WinDbg or KD on your Windows host.

    To prepare the target virtual machine, follow the steps for a Windows host in Connecting an Application on the Host to a Virtual Machine. Make sure you configure the virtual machine's virtual serial port as follows:

  • Select This end is the server
  • Under I/O Mode, select the Yield CPU on poll check box, as the kernel in the target virtual machine uses the virtual serial port in polled mode, not interrupt mode
  • To prepare the host, make sure you have a recent version of Debugging Tools for Windows — one that supports debugging over a pipe. You need version 5.0.18.0 or higher.

    When you are ready to begin, complete the following steps:

    1. Power on the virtual machine.

    2. Check to make sure the serial port is connected. Choose choose VM > Removable Devices. On that menu, serial<n> should be reported as \\.\pipe\<namedpipe> (on Windows hosts) or /tmp/<socket> (on Linux hosts). If the serial port is not connected, choose the virtual serial port, then Connect.

    3. On the host, open a Command Prompt window and do one of the following:

  • If you are using WinDbg, type the following:
  • windbg -k com:port=\\.\pipe\<namedpipe>,pipe

  • If you are using KD, type the following:
  • kd -k com:port=\\.\pipe\<namedpipe>,pipe

    Then press Enter to start debugging.

    Debugging an Application in a Virtual Machine from another Virtual Machine

    In this situation, you have kernel code to debug in a virtual machine (called the target virtual machine) and are running Debugging Tools for Windows (WinDbg) or Kernel Debugger (KD) in another virtual machine (called the debugger virtual machine) on the same host.

    This is useful if you are running VMware Workstation on a Linux host. The debugger virtual machine must be running Debugging Tools for Windows (WinDbg) or Kernel Debugger (KD) in a Windows guest operating system.

    To prepare the target virtual machine, follow the steps for the server virtual machine for the appropriate host in Connecting Two Virtual Machines. Make sure when you configure the target virtual machine's virtual serial port that you select the Yield CPU on poll check box, as the kernel in the target virtual machine uses the virtual serial port in polled mode, not interrupt mode.

    To prepare the debugger virtual machine, make sure you have downloaded Debugging Tools for Windows. Then follow the steps for the client virtual machine in Connecting Two Virtual Machines.

    When you are ready to begin, complete the following steps:

    1. Power on both virtual machines.

    2. Check to make sure the serial port is connected. Choose choose VM > Removable Devices. If the serial port is not connected, choose the virtual serial port, then Connect.

    3. In the debugger virtual machine, start debugging with WinDbg or KD normally.

    previous Prev   Contents   Last   Next next