Working with Log Files¶

Using Standard Formats¶

The standard log formats include Squid, Netscape Common, Netscape extended, and Netscape Extended-2. The standard log file formats can be analyzed with a wide variety of off-the-shelf log-analysis packages. You should use one of the standard event log formats unless you need information that these formats do not provide. Refer to Using the Custom Format.

Set standard log file format options by following the steps below:

1. In the records.config file, edit the following variables
2. Edit the following variables to use the Squid format:
   • proxy.config.log.squid_log_enabled
   • proxy.config.log.squid_log_is_ascii
   • proxy.config.log.squid_log_name
   • proxy.config.log.squid_log_header
3. To use the Netscape Common format, edit the following variables:
   • proxy.config.log.common_log_enabled
   • proxy.config.log.common_log_is_ascii
   • proxy.config.log.common_log_name
   • proxy.config.log.common_log_header
4. To use the Netscape Extended format, edit the following variables:
   • proxy.config.log.extended_log_enabled
   • proxy.config.log.extended_log_is_ascii
   • proxy.config.log.extended_log_name
   • proxy.config.log.extended_log_header
5. To use the Netscape Extended-2 format, edit the following variables:
   • proxy.config.log.extended2_log_enabled
   • proxy.config.log.extended2_log_is_ascii
   • proxy.config.log.extended2_log_name
   • proxy.config.log.extended2_log_header
6. Run the command traffic_line -x to apply the configuration changes.

Using the Custom Format¶

The XML-based custom log format is more flexible then the standard log file formats and gives you more control over the type of information recorded in log files. You should create a custom log format if you need data for analysis that’s not available in the standard formats. You can decide what information to record for each Traffic Server transaction and create filters that specify which transactions to log.

The heart of the XML-based custom logging feature is the XML-based logging configuration file (logs_xml.config) that enables you to create very modular descriptions of logging objects. The logs_xml.config file uses three types of objects to create custom log files, as detailed below. To generate a custom log format, you must specify at least one LogObject definition (one log file is produced for each LogObject definition).

• The LogFormat object defines the content of the log file using printf-style format strings.
• The LogFilter object defines a filter so that you include or exclude certain information from the log file.
• The LogObject object specifies all the information needed to produce a log file.
  • The name of the log file. (required)
  • The format to be used (required). This can be a standard format (Squid or Netscape) or
  • a previously-defined custom format (i.e., a previously-defined LogFormat object).
  • The file mode: ASCII, Binary, or ASCII_PIPE. The default is ASCII. The ASCII_PIPE mode writes log entries to a UNIX-named pipe (a buffer in memory); other processes can then read the data using standard I/O functions. The advantage of this option is that Traffic Server does not have to write to disk, which frees disk space and bandwidth for other tasks. When the buffer is full, Traffic Server drops log entries and issues an error message indicating how many entries were dropped. Because Traffic Server only writes complete log entries to the pipe, only full records are dropped.
  • Any filters you want to use (i.e., previously-defined LogFilter objects).
  • The collation servers that are to receive the log files.
  • The protocols you want to log. If the protocols tag is used, then Traffic Server will only log transactions from the protocols listed; otherwise, all transactions for all protocols are logged.
  • The origin servers you want to log. If the servers tag is used, then Traffic Server will only log transactions for the origin servers listed; otherwise, transactions for all origin servers are logged.
  • The header text you want the log files to contain. The header text appears at the beginning of the log file, just before the first record.
  • The log file rolling options.

In order to accomplish this, we

1. edit the following variables in the records.config file:
2. proxy.config.log.custom_logs_enabled
3. In the logs_xml.config file
4. Add LogFormat, LogFilters, and LogObject specifications to the configuration file.
5. Save and close the logs_xml.config file.
6. Run the command traffic_line -x to apply your configuration changes.

Creating Summary Log Files¶

Traffic Server performs several hundred operations per second; therefore, event log files can quickly grow to large sizes. Using SQL-like aggregate operators, you can configure Traffic Server to create summary log files that summarize a set of log entries over a specified period of time. This can significantly reduce the size of the log files generated.

To generate a summary log file, create a LogFormat object in the XML-based logging configuration file (logs_xml.config) using the SQL-like aggregate operators below. You can apply each of these operators to specific fields, over a specified interval.

• COUNT
• SUM
• AVERAGE
• FIRST
• LAST

To create a summary log file format, we

1. Define the format of the log file in logs_xml.config as follows:

   <LogFormat>
     <Name = "summary"/>
     <Format = "%<operator(field)> : %<operator(field)>"/>
     <Interval = "n"/>
   </LogFormat>
   

   where operator is one of the five aggregate operators (COUNT, SUM, AVERAGE, FIRST, LAST), field is the logging field you want to aggregate, and n is the interval (in seconds) between summary log entries. You can specify more than one operator in the format line. For more information, refer to logs_xml.config.

2. Run the command traffic_line -x to apply configuration changes .

The following example format generates one entry every 10 seconds. Each entry contains the timestamp of the last entry of the interval, a count of the number of entries seen within that 10-second interval, and the sum of all bytes sent to the client:

<LogFormat>
  <Name = "summary"/>
  <Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)>"/>
  <Interval = "10"/>
</LogFormat>

<Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)> : %<cqu>"/>

Choosing Binary or ASCII¶

You can configure Traffic Server to create event log files in either of the following:

• ASCII

  These files are human-readable and can be processed using standard, off-the-shelf log analysis tools. However, Traffic Server must perform additional processing to create the files in ASCII, which mildly impacts system overhead. ASCII files also tend to be larger than the equivalent binary files. By default, ASCII log files have a .log filename extension.

• Binary

  These files generate lower system overhead and generally occupy less space on the disk than ASCII files (depending on the type of information being logged). However, you must use a converter application before you can read or analyze binary files via standard tools. By default, binary log files use a .blog filename extension.

While binary log files typically require less disk space, there are exceptions.

For example: the value 0 (zero) requires only one byte to store in ASCII, but requires four bytes when stored as a binary integer. Conversely: if you define a custom format that logs IP addresses, then a binary log file would only require four bytes of storage per 32-bit address. However, the same IP address stored in dot notation would require around 15 characters (bytes) in an ASCII log file. Therefore, it’s wise to consider the type of data that will be logged before you select ASCII or binary for your log files. For example, you might try logging for one day using ASCII and then another day using binary. If the number of requests is roughly the same for both days, then you can calculate a rough metric that compares the two formats.

For standard log formats, select Binary or ASCII (refer to Setting Standard Log File Format Options). For the custom log format, specify ASCII or Binary mode in the LogObject (refer to Using the Custom Format). In addition to the ASCII and binary options, you can also write custom log entries to a UNIX-named pipe (i.e., a buffer in memory). Other processes can then read the data using standard I/O functions. The advantage of using this option is that Traffic Server does not have to write to disk, which frees disk space and bandwidth for other tasks. In addition, writing to a pipe does not stop when logging space is exhausted because the pipe does not use disk space. Refer to logs_xml.config for more information about the ASCII_PIPE option.

Rolled Log Filename Format¶

Traffic Server provides a consistent naming scheme for rolled log files that enables you to easily identify log files. When Traffic Server rolls a log file, it saves and closes the old file before it starts a new file. Traffic Server renames the old file to include the following information:

• The format of the file (such as squid.log).
• The hostname of the Traffic Server that generated the log file.
• Two timestamps separated by a hyphen (-). The first timestamp is a lower bound for the timestamp of the first record in the log file. The lower bound is the time when the new buffer for log records is created. Under low load, the first timestamp in the filename can be different from the timestamp of the first entry. Under normal load, the first timestamp in the filename and the timestamp of the first entry are similar. The second timestamp is an upper bound for the timestamp of the last record in the log file (this is normally the rolling time).
• The suffix .old, which makes it easy for automated scripts to find rolled log files.

Timestamps have the following format:

%Y%M%D.%Hh%Mm%Ss-%Y%M%D.%Hh%Mm%Ss

The following table describes the format:

%Y

The year in four-digit format. For example: 2000.

%M

The month in two-digit format, from 01-12. For example: 07.

%D

The day in two-digit format, from 01-31. For example: 19.

%H

The hour in two-digit format, from 00-23. For example: 21.

%M

The minute in two-digit format, from 00-59. For example: 52.

%S

The second in two-digit format, from 00-59. For example: 36.

The following is an example of a rolled log filename:

squid.log.mymachine.20110912.12h00m00s-20000913.12h00m00s.old

The logging system buffers log records before writing them to disk. When a log file is rolled, the log buffer might be partially full. If it is, then the first entry in the new log file will have a timestamp earlier than the time of rolling. When the new log file is rolled, its first timestamp will be a lower bound for the timestamp of the first entry.

For example, suppose logs are rolled every three hours, and the first rolled log file is:

squid.log.mymachine.20110912.12h00m00s-19980912.03h00m00s.old

If the lower bound for the first entry in the log buffer at 3:00:00 is 2:59:47, then the next log file will have the following timestamp when rolled:

squid.log.mymachine.20110912.02h59m47s-19980912.06h00m00s.old

The contents of a log file are always between the two timestamps. Log files do not contain overlapping entries, even if successive timestamps appear to overlap.

Rolling Intervals¶

Log files are rolled at specific intervals relative to a given hour of the day. Two options control when log files are rolled:

• The offset hour, which is an hour between 0 (midnight) and 23
• The rolling interval

Both the offset hour and the rolling interval determine when log file rolling starts. Rolling occurs every rolling interval and at the offset hour. For example, if the rolling interval is six hours and the offset hour is 0 (midnight), then the logs will roll at midnight (00:00), 06:00, 12:00, and 18:00 each day. If the rolling interval is 12 hours and the offset hour is 3, then logs will roll at 03:00 and 15:00 each day.

Setting Log File Rolling Options¶

To set log file rolling options and/or configure Traffic Server to roll log files when they reach a certain size, follow the steps below:

1. In the records.config file, edit the following variables
   • proxy.config.log.rolling_enabled
   • proxy.config.log.rolling_size_mb
   • proxy.config.log.rolling_offset_hr
   • proxy.config.log.rolling_interval_sec
2. Run the command traffic_line -x to apply the configuration changes.

You can fine-tune log file rolling settings for a custom log file in the LogObject specification in the logs_xml.config file. The custom log file uses the rolling settings in its LogObject, which override the default settings you specify in Traffic Manager or the records.config file described above.

ICP Log Splitting¶

When ICP log splitting is enabled, Traffic Server records ICP transactions in a separate log file with a name that contains icp. For example: if you enable the Squid format, then all ICP transactions are recorded in the squid-icp.log file. When you disable ICP log splitting, Traffic Server records all ICP transactions in the same log file as HTTP transactions.

HTTP Host Log Splitting¶

HTTP host log splitting enables you to record HTTP transactions for different origin servers in separate log files. When HTTP host log splitting is enabled, Traffic Server creates a separate log file for each origin server that’s listed in the log_hosts.config file. When both ICP and HTTP host log splitting are enabled, Traffic Server generates separate log files for HTTP transactions (based on the origin server) and places all ICP transactions in their own respective log files. For example, if the log_hosts.config file contains the two origin servers uni.edu and company.com and Squid format is enabled, then Traffic Server generates the following log files:

squid-uni.edu.log

All HTTP transactions for uni.edu

squid-company.com.log

All HTTP transactions for company.com

squid-icp.log

All ICP transactions for all hosts

squid.log

All HTTP transactions for other hosts

If you disable ICP log splitting, then ICP transactions are placed in the same log file as HTTP transactions. Using the hosts and log format from the previous example, Traffic Server generates the log files below:

squid-uni.edu.log

All entries for uni.edu

squid-company.com.log

All entries for company.com

squid.log

All other entries

Traffic Server also enables you to create XML-based Custom Log Formats that offer even greater control over log file generation.

Setting Log Splitting Options¶

To set log splitting options, follow the steps below:

1. In the records.config file, edit the following variables
   • proxy.config.log.separate_icp_logs
   • proxy.config.log.separate_host_logs
2. Run the command traffic_line -x to apply the configuration changes.

Editing the log_hosts.config File¶

The default log_hosts.config file is located in the Traffic Server config directory. To record HTTP transactions for different origin servers in separate log files, you must specify the hostname of each origin server on a separate line in the log_hosts.config file. For example, if you specify the keyword sports, then Traffic Server records all HTTP transactions from sports.yahoo.com and www.foxsports.com in a log file called squid-sports.log (if the Squid format is enabled).

To edit the log_hosts.config file follow the steps below:

1. In the log_hosts.config file, enter the hostname of each origin server on a separate line in the file, e.g.:

   webserver1
   webserver2
   webserver3
   

2. Run the command traffic_line -x to apply the configuration changes.

Configuring Traffic Server to Be a Collation Server¶

To configure a Traffic Server node to be a collation server, simply edit a configuration file via the steps below.

1. In the records.config file, edit the following variables
   • proxy.local.log.collation_mode (1 for server mode)
   • proxy.config.log.collation_port
   • proxy.config.log.collation_secret
2. Run the command traffic_line -x to apply the configuration changes.

Using a Standalone Collator¶

If you do not want the log collation server to be a Traffic Server node, then you can install and configure a standalone collator (SAC) that will dedicate more of its power to collecting, processing, and writing log files.

To install and configure a standalone collator:

1. Configure your Traffic Server nodes as log collation clients; refer to Configuring Traffic Server to Be a Collation Client.

2. Copy the traffic_sac binary from the Traffic Server bin directory and

3. Copy the libtsutil.so libraries from the Traffic Server lib directory to the machine serving as the standalone collator.

4. Create a directory called config in the directory that contains the traffic_sac binary.

5. Create a directory called internal in the config directory you created in Step 4 (above). This directory is used internally by the standalone collator to store lock files.

6. Copy the records.config file from a Traffic Server node configured to be a log collation client to the config directory you created in Step 4 on the standalone collator. The records.config file contains the log collation secret and the port you specified when configuring Traffic Server nodes to be collation clients. The collation port and secret must be the same for all collation clients and servers.

7. In the records.config file, edit the following variable

   • proxy.config.log.logfile_dir

8. Enter the following command:

   traffic_sac -c config
   

Configuring Traffic Server to Be a Collation Client¶

To configure a Traffic Server node to be a collation client, follow the steps below. If you modify the collation_port or secret after connections between the collation clients and the collation server have been established, then you must restart Traffic Server.

1. In the records.config file, edit the following variables:
   • proxy.local.log.collation_mode: 2 to configure this node as log collation client and send standard formatted log entries to the collation server. For XML-based formatted log entries, see logs_xml.config file; refer to Using the Custom Format.
   • proxy.config.log.collation_host
   • proxy.config.log.collation_port
   • proxy.config.log.collation_secret
   • proxy.config.log.collation_host_tagged
   • proxy.config.log.max_space_mb_for_orphan_logs
2. Run the command traffic_line -x to apply the configuration changes.

Collating Custom Event Log Files¶

If you use custom event log files, then you must edit the logs_xml.config file (in addition to configuring a collation server and collation clients).

To collate custom event log files

1. On each collation client, edit the :file:`logs_xml.config

2. Add the CollationHost attribute to the LogObject specification:

   <LogObject>
     <Format = "squid"/>
     <Filename = "squid"/>
     <CollationHosts="ipaddress:port"/>
   </LogObject>
   

   where ipaddress is the hostname or IP address of the collation server to which all log entries (for this object) are forwarded, and port is the port number for communication between the collation server and collation clients.

3. Run the command traffic_line -L to restart Traffic Server on the local node or traffic_line -M to restart Traffic Server on all the nodes in a cluster.

Squid log in XML¶

This is the equivalent XML configuration for the log above:

<LogFormat>
  <Name = "squid"/>
  <Format = "%<cqtq> %<ttms> %<chi> %<crc>/%<pssc> %<psql> %<cqhm> %<cquc>
             %<caun> %<phr>/%<pqsn> %<psct>"/>
</LogFormat>

Netscape Common in XML¶

This is the equivalent XML configuration for the log above:

<LogFormat>
  <Name = "common"/>
  <Format = "%<chi> - %<caun> [%<cqtn>] \"%<cqtx>\" %<pssc> %<pscl>"/>
</LogFormat>

Netscape Extended in XML¶

This is the equivalent XML configuration for the log above:

<LogFormat>
  <Name = "extended"/>
  <Format = "%<chi> - %<caun> [%<cqtn>] \"%<cqtx>\" %<pssc> %<pscl>
     %<sssc> %<sscl> %<cqbl> %<pqbl> %<cqhl> %<pshl> %<pqhl> %<sshl> %<tts>"/>
</LogFormat>

Netscape Extended2 in XML¶

This is the equivalent XML configuration for the log above:

<LogFormat>
  <Name = "extended2"/>
  <Format = "%<chi> - %<caun> [%<cqtn>] \"%<cqtx>\" %<pssc> %<pscl>
             %<sssc> %<sscl> %<cqbl> %<pqbl> %<cqhl> %<pshl> %<pqhl> %<sshl> %<tts> %<phr> %<cfsc> %<pfsc> %<crc>"/>
</LogFormat>