Traffic Server Cluster¶
Traffic Server scales from a single node to multiple nodes that form a cluster allowing you to improve system performance and reliability.
Understanding Traffic Server Clusters¶
A Traffic Server cluster consists of multiple Traffic Server nodes. The nodes in a cluster share configuration information and can form a single logical cache. Traffic Server detects the addition and deletion of nodes in the cluster automatically and can detect when a node is unavailable. Traffic Server uses its own protocol for clustering, which is multicast for node location and heartbeat, but unicast for all data exchange within the cluster. Traffic Server has two clustering modes:
- Management-only mode; refer to Management-Only Clustering below.
- Full-clustering mode; refer to Full Clustering
Management-Only Clustering¶
In management-only clustering mode, Traffic Server cluster nodes share configuration information. You can administer all the nodes at the same time. Traffic Server uses a multicast management protocol to provide a single system image of your Traffic Server cluster. Information about cluster membership, configuration, and exceptions is shared across all nodes, and the traffic_manager process automatically propagates configuration changes to all the nodes.
Full Clustering¶
In full-clustering mode, as well as sharing configuration information, a Traffic Server cluster distributes its cache across its nodes into a single, virtual object store, rather than replicating the cache node by node. Traffic Server can provide an enormous aggregate cache size and can maximize cache hit rate by storing objects only once across the entire cluster.
A fully clustered Traffic Server maps objects to specific nodes in the cluster. When a node receives a request, it checks to see if the request is a hit somewhere in the cluster. If the request is a hit on a different node, the node handling the request obtains the object from the hit node and serves it to the client. Traffic Server uses its own communication protocol to obtain an object from sibling cluster nodes.
If a node fails or is shut down and removed, Traffic Server removes references to the missing node on all nodes in the cluster.
Full clustering recommends a dedicated network interface for cluster communication to get better performance.
Enabling Clustering Mode¶
Before you put a node into a cluster, please make sure the following things are in order:
- You are using the same operation system on all nodes:
- Using the same distribution, e.g.: RHEL 5.5
- Have same kernel, e.g.: 2.6.18-194.17.1.el5
- The same architecture, e.g.: x86_64
- You have the same version of Traffic Server installed
- The same hardware
- On the same switch or same VLAN.
Traffic Server does not apply the clustering mode change to all the nodes in the cluster. You must change the clustering mode on each node individually. You may following these instructions:
- Setup the same cluster name, with proxy.config.proxy_name, e.g. MyCluster.
- Set proxy.local.cluster.type to 1, to enable cluster mode. The following values of this configuration are valid
| Value | Description |
|---|---|
| 1 | full-clustering mode |
| 2 | management-only mode |
| 3 | no clustering (default) |
Setup a proxy.config.cluster.ethernet_interface, e.g.: eth0. This should be replaced by your real interface; we recommends a dedicated interface here. Refer to proxy.local.cluster.type for a full description.
Enable configuration changes:
traffic_line -x
Restart traffic server:
traffic_line -L
The traffic_server and traffic_manager processes will need to restart after the change of proxy.local.cluster.type and proxy.config.cluster.ethernet_interface have taken place.
Traffic Server will join the cluster in about 10 seconds, and you can run traffic_line -r proxy.process.cluster.nodes to check the hosts in the cluster, or check out the cluster.config in the configuration directory. This configuration is generated by the system, and should not be edited. It contains a list of the machines that are currently members of the cluster, for example:
# 3
127.1.2.3:80
127.1.2.4:80
127.1.2.5:80
After successfully joining of a cluster, all changes of global configurations on any node, will take effect on all nodes. This means you can make changes on any cluster node member, and they are automatically distributed to all members.
Deleting Nodes from a Cluster¶
To delete a node from the Traffic Server cluster, just roll back proxy.local.cluster.type to the default value 3 and reload.
Common issues for Cluster setup¶
- The Cluster member auto discovery is build upon from multi-casting of the UDP, so imposible to setup where multi-casting is not avaliable, such as AWS EC2.
- The Cluster will depend on some internal ports: 8088 8089 and 8086, you should make sure the firewall will not ban them.
Performance tweak for busy Cluster¶
Starting from v3.2.0, Apache Traffic Server can handle multiple internal cluster connections, and we can tweak the number of Cluster threads. Each of the thread will keep one connection to all of peering cluster machines.
Increasing Cluster threads¶
In the cluster environment, the current performance of the cluster threads will consume the same cpu usage as a normal network thread. It’s reasonable to keep roughly the same number of cluster threads as network threads. For example, if you are running a system with 10 network processing threads, you can set the number of cluster threads by modifying proxy.config.cluster.threads to 10. E.g.:
traffic_line -s proxy.config.cluster.threads -v 10