Architecting a vRealize Log Insight Solution : Understanding vSphere Logs : 2.1 ESXi Log Files
   
2.1 ESXi Log Files
In vSphere 5.x, logging was significantly improved over earlier releases, making it far more straightforward to navigate and access logs, which in turn allows for improved troubleshooting and investigative analysis. In ESXi, all logs are now stored in the /var/log directory.
The physical location where logs are written depends on the device used during the ESXi installation. When the ESXi installation device is an SD card, USB key, or remote boot from an SAN environment, a local scratch partition is not created on the installation media automatically during the deployment. Despite its size, ESXi 6.x always sees this type of installation as remote, and as such, logs are stored in RAM disk (disk drive that is made up of a block of volatile memory) and lost when the host is rebooted.
The reason for this is that USB and SD devices are sensitive to high amounts of I/O, so the installer will not place the scratch partition on this type of device. The ESXi installer first scans for a local 4 GB VFAT partition. If it is unable to find one, it will then scan for a local VMFS volume to use to create a scratch directory. If no local VFAT partition or VMFS volume is found, the last resort is to put the scratch partition in the /tmp/scratch location on the local RAM disk.
After this type of installation, you will see a warning on the ESXi hosts in vCenter Server indicating that their log files are stored on non-persistent storage. (See Syslog not configured messages on ESXi host console or in logs (1032460) at http://kb.vmware.com/kb/1032460.
When this is the case, configure scratch space manually on the ESXi host using the VMware vSphere Web Client or CLI, or as part of a scripted installation procedure.
Because log messages that are stored on RAM disk are not retained after a reboot, troubleshooting information contained within the logs and core files will also be lost. If a persistent scratch location on the host is not configured properly, you might experience intermittent issues due to lack of space for temporary files, and the log files will not be updated. This can be problematic in low-memory hosts, but is not typically a critical issue for ESXi operation.
If the installation device is considered local during deployment, the ESXi host does not usually need to be manually configured with a scratch partition. The ESXi Installer creates a 4 GB FAT16 partition on the target device during the installation, if there is sufficient space to do so. If persistent scratch space is configured, most of these logs (see the following figure) are located on the scratch volume and the /var/log/ directory contains symlinks (symbolic links) to the persistent storage location.
Figure 1. Host Log Files
Note A symlink is a special type of file that contains a reference to another file in the form of an absolute or relative path.
For more information, see Creating a persistent scratch location for 4.x/5.x/6.0 (1033696), available at http://kb.vmware.com/kb/1033696.
The local ESXi logs can be accessed and inspected in one of several ways directly on each host:
• VMware vSphere Client™ or VMware vSphere Web Client
• DCUI (View system logs option)
• Web browser (example: https://HostnameOrIPAddress/host)
• Power-CLI “Get-Log” cmdlets
• ESXi CLI (example: cat /var/log/hostd.log)
For more information about accessing ESXi logs, see Location of ESXi 5.0 log files (2004201) at http://kb.vmware.com/kb/2004201.
The following table provides a comprehensive list of ESXi 6.0 host logs, their persistent location, and a description. Many different log files are generated automatically by different ESXi components and services.
Table 1. Table Host Log Descriptions
Log File
Persistent Location
Description
/var/log/auth.log
/scratch/log/auth.log
ESXi Shell authentication information such as success and failures.
/var/log/dhclient
.log
/scratch/log/dhclient.log
DHCP client log, including discovery, address lease requests and renewals.
/var/log/esxupdate
.log
/scratch/log/esxupdate.log
ESXi patch and update logs (useful if you need to know why a patch failed).
/var/log/hostd.log
/scratch/log/hostd.log
Host management service logs includes virtual machine and ESXi host task and events, communication with the vSphere Client and VMware vCenter Server vpxa agent, and SDK connections.
/var/log/shell.log
/scratch/log/shell.log
ESXi shell usage logs that track commands that were run. ESXi shell usage logs include enable/disable, and every command entered.
/var/log/sysboot.log
 
VMkernel startup and module loading.
/var/log/boot.gz
 
A compressed file that contains boot log information. Can be read without unzipping, by using zcat.
/var/log/syslog.log
/scratch/log/syslog.log
Management service initialization, watchdogs, scheduled tasks and DCUI use.
/var/log/usb.log
/scratch/log/usb.log
USB device information such as discovery and pass-through to virtual machines.
/var/log/vobd.log
/scratch/log/vobd.log
VMkernel observation events. VOBD is a daemon that VMware and third-party applications use for monitoring and troubleshooting.
/var/log/vmkernel
.log
/scratch/log/vmkernel.log
Core VMkernel logs, including device discovery, storage and networking device and driver information, and virtual machine startup information and driver events.
/var/log/vmkwarning
.log
/scratch/log/vmkwarning.
log
A summary of warnings and alert log messages excerpted from the VMkernel logs.
/var/log/vmksummary
.log
/scratch/log/vmksummary.
log
ESXi host startup/shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service resource consumption.
/var/log/vpxa.log
/scratch/log/vpxa.log
Present when a host is connected to a vCenter Server. vCenter Server VPXA agent logs, including communication with vCenter Server and the host management hostd agent.
/var/log/fdm.log
 
/scratch/log/fdm.log
Present when a host is connected to a vCenter Server. VMware vSphere High Availability logs, produced by the fdm service.
/var/log/lacp.log
/scratch/log/lacp.log
Link aggregation control protocol logs. ESXi 5.1 onwards only.
/var/log/hostd-probe.log
/scratch/log/hostd-probe.log
Host management service responsiveness checker.
/var/log/rhttpproxy
.log
/scratch/log/rhttpproxy.
log
HTTP connections proxied on behalf of other ESXi host web services.
/var/log/Xorg.log
/scratch/log/Xorg.log
Video acceleration.
/var/log/clomd.log
/scratch/log/clomd.log
CLOM daemon. Cluster level object manager (CLOM) logs. These logs are part of the VMware vSAN™ feature.
/var/log/esxcli.log
 
 
/var/log/osfsd.log
/scratch/log/osfsd.log
OSFSD daemon. Object storage file system (OSFS) logs. These logs are part of the vSAN feature.
/var/log/
sdrsinjector.log
/scratch/log/sdrsinjector.log
VMware vSphere Storage DRS™ injector log. Profiles the capabilities of the datastores for vSphere Storage DRS.
/var/log/swapobjd
.log
/scratch/log/swapobjd.log
SwapObj daemon logs.
/var/log/sysboot.log
 
Early VMkernel startup and module loading.
/var/log/vmamqpd.log
 
VMware AMQP daemon log.
/var/log/vmkeventd
.log
/scratch/log/vmkeventd.log
Capture of VMkernel events.
/var/log/vprobe.log
/scratch/log/vprobe.log
Output of VMware VProbes™, which provides a facility for transparently instrumenting a powered-on guest operating system, its currently running processes, and VMware virtualization software.
/var/log/vsanvpd.log
/scratch/log/vsanvpd.log
vSAN logs.
 
The exact list of ESXi host system component logs can vary from the files listed in the table, depending on the version of hypervisor installed. This extensive logging on every single host in the infrastructure can become unmanageable for both operational teams and administrators. This is where the benefits of providing a centralized, consolidated, and searchable syslog solution can be appreciated.
In addition to the listed ESXi 6.x host logs, a number of additional component source logs are, by design, not forwarded to a remote syslog server, but instead exist only on local persistent storage or in RAM disk. The following is list of ESXi component log sources that are not part of the syslog mechanism:
• configRP.log ‒ Resource pool changes
• hostprofiletrace.log ‒ Host profiles information
• smbios.bin ‒ Communication with smbios to provide hardware information
• storagerm.log ‒ Storage resource management, including I/O throttling
• vmauthd.log ‒ Authd information
• vmkdevmgr.log ‒ Hardware device detection
• vprobed.log ‒ Control and data logging