TL;DR: Installing XBMC on Debian Wheezy is quite easy: it almost works out of the box. The big difficulty is the configuration of the remote control: either it works as you expect or you will have to scratch your head over the pile of layers needed to work with a remote control.

The configuration of my HTPC is as follows:

• an Antec Micro Fusion 350 housing featuring a SoundGraph iMON IR receiver and LCD screen¹,
• a Zotac ION-ITX-D-E motherboard with an Intel Atom 330 CPU and an nVidia ION chipset,
• some Onkyo AV receiver connected through HDMI,
• a Logitech Harmony 555 as remote control.

Installation

Installing Debian Wheezy

Installing Debian Wheezy² is pretty easy. Nowadays, getting a bootable USB key from a netinst image of Debian Installer for Wheezy is simplified:

$ sudo dd if=debian-testing-i386-netinst.iso \
>         of=/dev/disk/by-id/usb...

The installation was smooth with the exception of GRUB which was unable to install itself on the disk. This is a known bug when dealing with LVM and it comes with a simple workaround. I hope it will be corrected in time for Wheezy release.

While this has little to do with the installation of XBMC, I wanted to test systemd which may become the default init in Debian (at least in Debian GNU/Linux). From README.Debian:

systemd can be installed alongside sysvinit and will not change the behaviour of the system out of the box. This is intentional. To test systemd, add init=/bin/systemd to the kernel command line and then rebooting, or install the systemd-sysv package.

The final system boots in about 15 seconds.

Configuring X

Because video decoding in nouveau driver is still a work in progress, the use of the proprietary NVIDIA drivers is mandatory to be able to watch high resolution videos. Therefore, /etc/apt/sources.list should be completed with contrib and non-free repository. Then, you can install the appropriate packages: xserver-xorg-video-nvidia, nvidia-vdpau-driver and xserver-xorg.

Here is my /etc/X11/xorg.conf.d/nvidia.conf:

Section "Device"
    Identifier     "NVidia ION"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    Option         "HWCursor" "False"
    Option         "NoFlip" "False"
    Option         "FlatPanelProperties" "Scaling = Native"
    Option         "DynamicTwinView" "False"
    Option         "ConnectedMonitor" "DFP-1"
    Option         "CustomEDID" "DFP-1:/etc/X11/edid.bin"
    Option         "NoLogo" "True"
EndSection

Section "Extensions"
    Option "Composite" "false"
EndSection

The CustomEDID option allows the driver to get an appropriate EDID even when the AV receiver is off. You can get yours, free of charge, with get-edid from read-edid package.

Installing XBMC

Thanks to the work of Andrés Mejía, XBMC is now available in Debian Wheezy. To install it, just type aptitude install xbmc. I have dropped the following xbmc.service in /etc/systemd/system directory:

[Unit]
Description = XBMC media center
After = syslog.target

[Service]
User = xbmc
Group = xbmc
Type = simple
ExecStart = /usr/bin/xinit /usr/bin/xbmc-standalone -- :0
Restart = on-failure

[Install]
WantedBy = multi-user.target

Enable this service on boot with systemctl enable xbmc.service. You need to allow xbmc user to run X. The simplest way is to run dpkg-reconfigure -plow x11-common and to allow anybody to run X. sudo may be an alternative.

Configuration

Sound

While I wanted to use PulseAudio, I want the AV receiver to be able to upmix stereo streams itself. With PulseAudio, it would always receive a 6-channel signal. Therefore, I directly use ALSA.

First, unmute the appropriate outputs:

$ amixer scontrols | grep IEC958
Simple mixer control 'IEC958',0
Simple mixer control 'IEC958 Default PCM',0
Simple mixer control 'IEC958',1
$ amixer sset 'IEC958',0 unmute
$ amixer sset 'IEC958 Default PCM',0 unmute
$ amixer sset 'IEC958',1 unmute
$ sudo systemctl stop alsa-utils.service

The order of channels is incorrect. With the following /etc/asound.conf, we declare a new output, hdmi2, with a different mapping:

pcm.hdmi2 {
  type asym
  playback.pcm {
    type plug
    slave.pcm "remap-surround51"
  }
}

pcm.!remap-surround51 {
  type route
  slave.pcm "hdmi"
  ttable {
    0.0= 1
    1.1= 1
    2.4= 1
    3.5= 1
    4.2= 1
    5.3= 1
  }
}

In XBMC, this output should be used instead of the default one. hdmi should still be used for passthrough. To check if each speaker is mapped correctly, one can use speaker-test -D hdmi2 -c 6.

LCD display

The LCD display integrated into the SoundGraph iMON is supported by the imon kernel module and the lcdproc package. I have only modified a few lines of /etc/LCDd.conf to make it work:

[server]
Driver=imonlcd
ServerScreen=off

[imonlcd]
Protocol=1
OnExit=2
Contrast=400

UPDATED: The use of the LCD screen triggers various bugs including segfaults of ir-keytable and freezes. Since its readibility is less than ideal, I don’t use it any more. I have uninstalled lcdproc and I am using this simple script to switch it off on start:

#!/usr/bin/python

import struct
open("/dev/lcd0", "w").write(struct.pack("Q",0x8800000000000008))

This script is triggered by the following rule for udev:

# Disable LCD screen
ACTION=="add", NAME=="lcd0",
   RUN+="/usr/local/bin/disable_lcd"

Remote control

This is the most difficult part. I have a Logitech Harmony remote which is a great universal remote. Its support in Linux is acceptable: you can configure through Logitech website and use congruity to push the new configuration.

Remote controls and Linux

Before Linux 2.6.36, most remote controls would need LIRC to work:

• The driver receives the signal from the IR receiver and make it available through /dev/lirc.
• lircd, with the help of a configuration file describing the protocol used by the remote control, will read the signal and turn it into the appropriate LIRC code.
• XBMC connects to lircd and receives incoming LIRC codes. It will translate them to an XBMC command. This translation is specified in Lircmap.xml.
• XBMC maps each command to an action (like Play, Fullscreen, …) using a keymap. This keymap can handle commands received by a remote control, but also by a keyboard, a mouse or a joystick.

Since Linux 2.6.36, remote controls will be mapped as a generic input device (just like a keyboard):

• The driver receives the signal from the IR receiver.
• The signal will be handled by a decoder. The configuration of this decoder is done in userland by ir-keytable. The decoder will turn the signal into the appropriate event (usually, some keypress).
• X will listen to those events and turn them into X key events.
• XBMC will receive them and use the appropriate keymap to turn them into actions.

And to add more complexity to the mix, in this last case, you can still use LIRC: lircd will listen to events generated by the kernel and turn them into LIRC codes. This can be very confusing.

Moreover, the SoundGraph iMON IR receiver accepts two IR protocols: the iMON protocol and the RC-6 one. The Linux driver accepts both of them but uses the first one by default. The RC-6 protocol is the protocol used by many MCE remote controls.

I hope you are still with me here.

The easy way

To get a reasonable configuration out of the box, here is how to configure each layer:

Logitech Harmony remote

Configure it as a Microsoft branded Media Center PC: Windows Media Center SE.

iMON IR receiver

It must use RC-6 protocol. See below for more details.

LIRC

In /etc/lirc/hardware.conf, put DEVICE=/dev/input/by-id/usb-15c2_0038-event-if00 and DRIVER=devinput. In /etc/lirc/lircd.conf, just put include "/usr/share/lirc/remotes/devinput/lircd.conf.devinput".

XBMC

With the previous bits done, it should just work out of the box.

To switch to RC-6 protocol, install the ir-keytable package and use the following commands:

$ sudo modprobe rc-imon-mce
$ sudo ir-keytable -s rc0 -p rc-6 -c -w /lib/udev/rc_keymaps/imon_mce
Read imon_mce table
Old keytable cleared
Wrote 77 keycode(s) to driver
Protocols changed to RC-6

To make the change permanent, add the rc-imon-mce module to /etc/modules and create /etc/udev/rules.d/90-imon.rules with the following content:

# Override the keytable for iMON
ACTION=="add|change", SUBSYSTEM=="rc", DRV_NAME="imon", \
   RUN+="/usr/bin/ir-keytable -s $name -p rc-6 -c -w /lib/udev/rc_keymaps/imon_mce"

The hard way

Now, you may want to bind custom actions to some (physical or virtual) buttons. Basically, you are left with two solutions:

1. Start from the basic configuration with LIRC and add more buttons at each levels (there are five of them!).
2. Remove LIRC and start with the Logitech Harmony acting as a Microsoft MCE keyboard.

The first option can be quite difficult. You need to find an unused code for the Logitech Harmony. You can try to make it learn a new code if you have some RC-6 remote control. Then, you need to ensure that this code will be present in the keytable used by ir-keytable. If not, you need to add it. That’s not easy since you need a to enable some debug stuff in the kernel to find the appropriate scancode. After that, the code needs to be translated in lircd.conf. You will then have to translate it again in Lircmap.xml. At least, you need to add it to a keymap in XBMC.

The other way is not ideal but seems less cumbersome. The first step is to configure the Logitech Harmony as a Microsoft MCE keyboard: it has a lot of available keys. Because of the lack of multimedia keys, let’s match the keyboard configuration of XBMC:

Unfortunately, the keytables provided with ir-keytable are not complete enough. I have built a more complete table³. With this table and the bindings described above, most functions will work out of the box without LIRC.

Additional keys can be configured in a dedicated keymap⁴. Here is an excerpt of mine:

<keymap>
 <global>
  <keyboard>
    <end>XBMC.ShutDown()</end>
    <f1>XBMC.ActivateWindow(MusicLibrary)</f1>
    <f2>XBMC.ActivateWindow(Videos,TvShowTitles)</f2>
    <f3>XBMC.ActivateWindow(Videos,MovieTitles)</f3>
    <f4>XBMC.ActivateWindow(Weather)</f4>
  </keyboard>
 </global>
 <FullscreenVideo>
  <keyboard>
   <opensquarebracket>SubtitleDelayMinus</opensquarebracket>
   <closesquarebracket>SubtitleDelayPlus</closesquarebracket>
   <f6>xbmc.runscript(script.xbmc.subtitles)</f6>
  </keyboard>
 </FullscreenVideo>
</keymap>

FTP

Instead of using SSH, I prefer to drop new files with anonymous FTP. vsftpd fits this purpose. Here is my configuration file:

listen=YES
xferlog_enable=YES
use_localtime=YES
setproctitle_enable=YES

secure_chroot_dir=/var/run/vsftpd/empty
nopriv_user=ftp
ftpd_banner=XBMC
hide_ids=YES

ftp_username=xbmc
anon_umask=022
anon_root=/home/xbmc/media
anonymous_enable=YES
write_enable=YES
anon_upload_enable=YES
anon_world_readable_only=YES

It is currently not compatible with systemd (see bug #670308). I have removed the symlink in /etc/rc2.d and I have used the following unit file:

[Unit]
Description=Vsftpd ftp daemon
After=syslog.target network.target

[Service]
Type=simple
ExecStart=/usr/sbin/vsftpd /etc/vsftpd.conf
ExecReload=/bin/kill -HUP $MAINPID
ExecStartPre=-/bin/mkdir -p /var/run/vsftpd/empty

[Install]
WantedBy=multi-user.target

Miscellaneous

1. In /etc/default/grub, reduce TIMEOUT to 0 to shorten the boot time.

2. Enabling dirty regions can help speed up XBMC.

3. aptitude install upower pm-utils to be able to shutdown/suspend from XBMC. Since XBMC was configured to be started outside any session, you need to explicitely give the appropriate rights by creating the following /var/lib/polkit-1/localauthority/50-local.d/xbmc.pkla:

   [Actions for xbmc user]
   Identity=unix-user:xbmc
   Action=org.freedesktop.upower.*;org.freedesktop.consolekit.system.*
   ResultAny=yes
   ResultInactive=yes
   ResultActive=yes
   

TL;DR: An application asynchronously leveraging Net-SNMP AgentX implementation has many opportunities to get stuck when the master agent becomes temporarily unavailable. A mostly efficient workaround is to dedicate a master agent for AgentX and delegate MIB handling to a subagent.

Pinging the master agent

Net-SNMP uses an event-based model. It also provides some synchronous functions but they are built on top of the asynchronous ones. It is shipped with its own event loop. Therefore, Net-SNMP AgentX protocol implementation seems a fine choice to add SNMP support to an event-based program, for example in Keepalived, a VRRP daemon and a monitoring daemon for LVS clusters.

Unfortunately, an AgentX subagent is expected to check, at a regular interval, if the master agent is alive . This is handled automatically with Net-SNMP implementation as long as you call run_alarms() function which will call agentx_check_session():

/*
 * check a session validity for connectivity to the master agent.  If
 * not functioning, close and start attempts to reopen the session 
 */
void
agentx_check_session(unsigned int clientreg, void *clientarg)
{
    /* [...] */
    DEBUGMSGTL(("agentx/subagent", "checking status of session %p\n", ss));

    if (!agentx_send_ping(ss)) {
        snmp_log(LOG_WARNING,
                 "AgentX master agent failed to respond to ping.  Attempting to re-register.\n");
        /*
         * master agent disappeared?  Try and re-register.
         * close first, just to be sure .
         */
        agentx_unregister_callbacks(ss);
        agentx_close_session(ss, AGENTX_CLOSE_TIMEOUT);
        /* [...] */
        if (main_session != NULL) {
            /* [...] */
            main_session = NULL;
            agentx_reopen_session(0, NULL);
        }
        else {
            snmp_close(main_session);
            main_session = NULL;
        }
    } else {
        DEBUGMSGTL(("agentx/subagent", "session %p responded to ping\n",
                    ss));
    }
}

This piece of code will end up calling (directly or indirectly) the following functions¹ which are using agentx_synch_response():

• agentx_send_ping()
• agentx_open_session()
• agentx_reopen_session()
• agentx_close_session()
• agentx_register()
• agentx_unregister()
• agentx_register_index()
• agentx_unregister_index()
• agentx_add_agentcaps()
• agentx_remove_agentcaps()

agentx_synch_response() will synchronously wait for an answer from the master agent. Your event-based program will just sit here doing nothing while this happens. For example, keepalived may be unable to send VRRP probes resulting in some serious havoc in your cluster.

Functions prefixed by agentx_ are not part of NetSNMP API and could therefore be rewritten to do their work asynchronously. Unfortunately, talk is cheap. Moreover, there exists other deep paths that will block a subagent. For example, the low-level connection to the master agent (using TCP or a Unix socket, but not with UDP) will use a blocking connect() call. It seems quite difficult to fix this.

Here are some (partial) workarounds:

• Call agentx_check_session() less often. The default interval is 15 seconds. Disabling it completely or using a very high value is not advised because this function is the only way to reestablish a connection to the master agent when it is, for example, restarted. Here is how to change the delay to 120 seconds:

  netsnmp_ds_set_int(NETSNMP_DS_APPLICATION_ID,
                     NETSNMP_DS_AGENT_AGENTX_PING_INTERVAL, 120);
  

• Don’t call agentx_check_session() if we know the master agent is alive. For example, if it just sent us a new request, no need to ping it. I have written a patch implementing such a workaround. The ABI is left untouched, therefore, you only need to recompile Net-SNMP libraries.

• Use a lower timeout and no retry. By default, the timeout for synchronous requests is 1 second and a failed request will be retried 5 times. Because several synchronous functions are used in a row, you may be trapped for as long as 30 seconds with those settings. Here is how² to disable the retry mechanism and to set the timeout to a very low value:

  static int
  snmp_setup_session_cb(int majorID, int minorID,
                void *serverarg, void *clientarg)
  {
      netsnmp_session *sess = serverarg;
      sess->timeout = ONE_SEC / 3;
      sess->retries = 0;
      return 0;
  }
  
  void some_init_function()
  {
      /* [...] */
      snmp_register_callback(SNMP_CALLBACK_LIBRARY,
         SNMP_CALLBACK_SESSION_INIT,
         snmp_setup_session_cb, NULL);
      /* [...] */
  }
  

Unresponsive master agent

Why is the master agent unresponsive? snmpd is also an event-based program and, trust me, there are a lot of places where it will become unresponsive. Here are two examples:

• When using the pass or the pass_persist directive, Net-SNMP will block until the delegated command outputs a complete line.

  /* var_extensible_pass() in agent/mibgroup/ucd-snmp/pass.c */
  /*
   * valid call.  Exec and get output 
   */
  if ((fd = get_exec_output(passthru)) != -1) {
      file = fdopen(fd, "r");
      if (fgets(buf, sizeof(buf), file) == NULL) {
          fclose(file);
          wait_on_exec(passthru);
          /* [...] */
          continue;
      }
      /* [...] */
      fclose(file);
      wait_on_exec(passthru);
      /* [...] */
  }
  

• DISMAN-PING-MIB implementation will just hang while pinging the remote host. If the host is down and you have requested 5 probes with a timeout of 1 second, snmpd will be unresponsive for 5 seconds. A similar problem exists with DISMAN-TRACEROUTE-MIB.

Minimal master agent

More and more code is running in the main agent. You cannot expect it to not block. But we could add a minimal master agent to the mix: it will handle only a few MIB modules by itself while most MIB modules will be delagated to a complete agent running as a subagent. Your own subagent will connect to the minimal master agent.

It is quite easy to turn snmpd into a subagent: just use -X flag. We should also disable unwanted MIB modules for each agent with -I flag. For the minimal master agent, we only enable some modules. All other modules³ will be served by snmpd acting as a subagent.

$ MODS="snmp_mib,sysORTable,usmConf,usmStats,usmUser,vacm_conf,vacm_context,vacm_vars"
$ snmpd -Lsd -Lf /dev/null -u snmp -g snmp \
>   -C -c /etc/snmp/snmpd.master.conf \
>   -p /var/run/snmpd.master.pid \
>   -I $MODS
$ snmpd -Lsd -Lf /dev/null -u snmp -g snmp \
>   -p /var/run/snmpd.pid -X \
>   -I -$MODS

Access control configuration should be placed in /etc/snmp/snmpd.master.conf, as well as master agentx directive. Other directives (like pass, pass_persist, load, sysname, …) in /etc/snmp/snmpd.conf.

The minimal master agent is very unlikely to block and while the ground cause is still here, it is now very much mitigated. You may extend this solution by splicing snmpd into several subagents: one handling safe modules, one handling pass and pass_persist stuff and a last one for unsafe modules (like DISMAN-PING-MIB).

tl;dr: in my opinion, extending Net-SNMP should be done with either of those methods:

• using extend directive;
• using pass_persist directive;
• using AgentX protocol.

SNMP in a nutshell

The variables accessible with SNMP through the agent are organized in a tree. Each variable is typed and associated to an object identifier (or OID). For example, .1.3.6.1.2.1.31.1.1.1.1.2 is the OID of the name of the second network card.

Usually, SNMP is operated over UDP, port 161. For the purpose of this article, we consider a manager can issue three kinds of requests:

GET

retrieve a variable.

GETNEXT

retrieve the next variable, in lexical order of their OID. This operation enables an SNMP manager to walk through all available variables, for example to discover the list of interfaces of an agent. This operation is what makes an SNMP agent difficult to implement but this is also a critical feature.

SET

request the modification of a variable.

Since such an OID may be difficult to handle by humans, available variables are described by a MIB module which is a file that should be parsed to get a mapping between OID and variable names and types. For example, the above OID can also be referred as IF-MIB::ifName.2. Those MIB modules are described using a subset of ASN.1 defined in RFC 2578.

A MIB module allows to group several variables into a conceptual table. For example, IF-MIB::ifDescr, IF-MIB::ifType, IF-MIB::ifSpeed are columns of IF-MIB::ifTable. Each row is associated to an index. In the case of IF-MIB::ifTable, it is the interface index. Net-SNMP includes snmptable allowing one to display such a table in a natural manner¹:

$ snmptable localhost IF-MIB::ifTable
SNMP table: IF-MIB::ifTable

 ifIndex ifDescr           ifType ifMtu   ifSpeed
       1      lo softwareLoopback 16436  10000000
       2    eth0   ethernetCsmacd  1500 100000000
       3    eth1   ethernetCsmacd  1500  10000000
       4     br0   ethernetCsmacd  1500         0

Variables inside a table are called columnar objects while those outside are scalar objects. SNMP, as a protocol, does not care of this distinction since it does not know the concept of MIB modules.

Other useful tools from Net-SNMP includes snmpget to get a specific variable, snmpwalk to discover available variables (with GETNEXT operation) from an OID and snmptranslate to translate between object names and OID.

$ snmpget localhost IF-MIB::ifDescr.2
IF-MIB::ifDescr.2 = STRING: eth0
$ snmpwalk localhost IF-MIB::ifDescr
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
IF-MIB::ifDescr.4 = STRING: br0
$ snmpwalk -On public localhost .1.3.6.1.2.1.2.2.1.2
.1.3.6.1.2.1.2.2.1.2.1 = STRING: lo
.1.3.6.1.2.1.2.2.1.2.2 = STRING: eth0
.1.3.6.1.2.1.2.2.1.2.3 = STRING: eth1
.1.3.6.1.2.1.2.2.1.2.4 = STRING: br0
$ snmptranslate -On IF-MIB::ifDescr.3
.1.3.6.1.2.1.2.2.1.2.3

Extending Net-SNMP

ethtool -S allows us to access various statistics from a network card, like the number of octets received and transmitted or the number of collisions. While some of those statistics are defined in IF-MIB, in RMON-MIB and in EtherLike-MIB, some of them are not available and very specific. Therefore, we would like to export those statistics, unmodified, through SNMP.

To the best of my knowledge, no MIB module exists for such an usage. The first task is to write one. It can be called ETHTOOL-MIB. It only contains one table indexed by the interface index and the name of the statistic. Here is the expected output:

$ snmpwalk localhost ETHTOOL-MIB::ethtoolStat.2
ethtoolStat.2.'align_errors' = Counter64: 0
ethtoolStat.2.'broadcast' = Counter64: 25
ethtoolStat.2.'multicast' = Counter64: 345
ethtoolStat.2.'rx_errors' = Counter64: 0
ethtoolStat.2.'rx_missed' = Counter64: 0
ethtoolStat.2.'rx_packets' = Counter64: 262011
ethtoolStat.2.'tx_aborted' = Counter64: 0
ethtoolStat.2.'tx_errors' = Counter64: 0
ethtoolStat.2.'tx_multi_collisions' = Counter64: 0
ethtoolStat.2.'tx_packets' = Counter64: 296170
ethtoolStat.2.'tx_single_collisions' = Counter64: 0
ethtoolStat.2.'tx_underrun' = Counter64: 0
ethtoolStat.2.'unicast' = Counter64: 261641

I have coded various implementations of this MIB module and made them available in a GitHub repository.

With arbitrary commands

The extend directive allows the agent to execute an arbitrary command and to provide its output and status through NET-SNMP-EXTEND-MIB module. If we want to export the statistics for eth0, we could add something like this in snmpd.conf, using a custom script to format the output of ethtool -S in a way that is more suitable for our use:

# Export eth0 statistics
extend eth0-names  /full/path/to/ethtool-stats eth0 names
extend eth0-values /full/path/to/ethtool-stats eth0 values

The result can be retrieved through NET-SNMP-EXTEND-MIB::‌nsExtendOutput2Table:

$ snmpwalk localhost NET-SNMP-EXTEND-MIB::nsExtendOutput2Table
nsExtendOutLine."eth0-names".1 = STRING: tx_packets
nsExtendOutLine."eth0-names".2 = STRING: rx_packets
nsExtendOutLine."eth0-names".3 = STRING: tx_errors
nsExtendOutLine."eth0-names".4 = STRING: rx_errors
nsExtendOutLine."eth0-names".5 = STRING: rx_missed
nsExtendOutLine."eth0-names".6 = STRING: align_errors
nsExtendOutLine."eth0-names".7 = STRING: tx_single_collisions
nsExtendOutLine."eth0-names".8 = STRING: tx_multi_collisions
nsExtendOutLine."eth0-values".1 = STRING: 246309
nsExtendOutLine."eth0-values".2 = STRING: 223941
nsExtendOutLine."eth0-values".3 = STRING: 0
nsExtendOutLine."eth0-values".4 = STRING: 0
nsExtendOutLine."eth0-values".5 = STRING: 0
nsExtendOutLine."eth0-values".6 = STRING: 0
nsExtendOutLine."eth0-values".7 = STRING: 0
nsExtendOutLine."eth0-values".8 = STRING: 0

It is also possible to configure such a command using SET requests. For example, if we want to add statistics for eth2, we could issue the following command:

$ snmpset -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendStatus."eth2-names"'  = createAndGo \
>    'nsExtendCommand."eth2-names"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-names"'    = 'eth2 names' \
>    'nsExtendStatus."eth2-values"'  = createAndGo \
>    'nsExtendCommand."eth2-values"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-values"'    = 'eth2 values'
nsExtendStatus."eth2-names" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-names" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-names" = STRING: eth2 names
nsExtendStatus."eth2-values" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-values" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-values" = STRING: eth2 values
$ snmpgetnext -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendOutLine."eth2-names"'
nsExtendOutLine."eth2-names".1 = STRING: tx_packets

This can be very convenient but also a huge security risk. You should disable this functionality or enable it only for some SNMPv3 user.

Using extend does not allow to implement an arbitrary MIB module. If the cache is not disabled, extend is a good solution for various simple needs, like providing the content of a one-line file or the output of some Nagios plugin. However, exporting anything tabular, like interface statistics, is not a good fit.

With pass-through scripts

Net-SNMP allows one to extend the agent by delegating some sub-tree to a script defined by the pass_persist directive. Such a script will receive requests on its standard input with a very simple line-oriented protocol described in the manual page of snmpd.conf. Here is an example of interaction:

→  PING
←  PONG
→  get
→  .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
←  .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
←  counter64
←  223941
→  getnext
→  .1.3.6.1.4.1.39178.100.1.1.1.2.2
←  .1.3.6.1.4.1.39178.100.1.1.1.2.2.97.108.105.103.110.95.101.114
←  counter64
←  0

The protocol is simple enough to implement it from scratch in any scripting language. However, in Perl, you may prefer to use SNMP::‌ExtensionSNMP::‌PassPersist extension from Sébastien Aperghis-Tramoni. You can find the complete example on Github.

use SNMP::Extension::PassPersist;
my $extsnmp = SNMP::Extension::PassPersist->new(
    backend_collect => \&update_tree
    );
$extsnmp->run;

sub update_tree {
    my @interfaces = </sys/class/net/*>;
    foreach my $interface (@interfaces) {
        $interface =~ s/^.*\///;
        open(ETHTOOL, "ethtool -S $interface 2>/dev/null |") or next;
        while (<ETHTOOL>) {
            /^\s+(\w+): (\d+)$/ or next;
            my $name  = $1;
            my $value = int($2);
            my $oid   = oid_compute($interface, $name);
            $extsnmp->add_oid_entry($oid, "counter", $value);
        }
        close(ETHTOOL);
    }
}

With Python, snmp-passpersist module, from Nicolas Agius, provides a similar interface. Look at the complete example on GitHub.

pass_persist allows you to implement an arbitrary MIB module with honest performances. Here are some important things to know about this directive:

• Unless you have a recent version of Net-SNMP, 64bit types are converted to 32bit equivalents. The counters may overflow quickly.
• While handling a request, you are not allowed to pause (for example, to fetch a remote information): the agent would become unresponsive. If you need slow-to-get data or data from the agent, grab them in a separate thread and ensure you have them beforehand.

With AgentX protocol

AgentX is a protocol defined in RFC 2741 allowing a master agent to be extended by independent sub-agents. For snmpd to become a master agent, you just need to add master agentx in snmpd.conf. Here are some of the benefits of an AgentX sub-agent over pass_persist:

1. No configuration is needed for the master agent to accept an additional sub-agent. A sub-agent registers to the master agent the MIB modules (or part of them) it wants to take care of.
2. A sub-agent is decoupled from the master agent. It can run with a different identity or be integrated into another daemon to export its internal metrics, send traps or allow remote configuration through SNMP.
3. AgentX protocol can be carried over TCP. Sub-agents can therefore run on a foreign host or in a jailed environment.
4. 64bit types are fully supported. Traps are also supported.

Perl and Python

Net-SNMP provides a Perl module, NetSNMP::agent, to write a sub-agent using AgentX protocol. This module is not as convenient as the module for pass_persist described above since it does not provide a convenient layer to put required information into some datastore. Basically, it mimics the available C API. Here are a few examples of its use:

• An implementation of ETHTOOL-MIB I have done in the same way as the pass_persist version. My Perl skills are a bit rusty and the code has many rough edges.
• An implementation of a sub-agent for Redis servers.
• An implementation of BRIDGE-MIB shipped with Net-SNMP.

There is no similar binding for Python but there is an agentx extension using ctypes. It provides a more high-level view. I did not write an example for this one but you can look at the example provided by the author.

C with “new” API

To write a sub-agent in C, you have to choose between two API:

1. the API inherited from UCD-SNMP also known as the “traditional” API;
2. the new API developed for Net-SNMP 5.x.

With both API, an handler is registered to take care of one or several sub-trees. With the traditional API, the handler is invoked with very little information. It is then up to you to locate the appropriate variable by mapping the index part of the OID to the objects you want to export. It is quite simple for scalar objects but columnar objects with complex indexes are a pain with such an API.

The new API provides various helpers. Some of them allow you to copy objects into an array or a list and let Net-SNMP library do the hard work of serving them. Another helper will provide methods to serve objects without putting them in a special structure but by providing an iterator. Some of those helpers can also help you to cache some variables.

On top of this new API, Net-SNMP comes with mib2c, a tool that will help you convert a MIB module into C code: after a few questions, you will get a skeleton C code you will have to complete with the logic to retrieve real objects. A generated file also provides directions on what to do (with some magic command to have a step-by-step cookbook).

I was able to implement ETHTOOL-MIB by just running it through mib2c, choosing the table container helper with cache and adding some code in ethtoolStatTable_data_access.c. You can look at the result on Github. Here is a stripped-down version² of what has been modified:

/* Socket for ioctl */
skfd = socket(AF_INET, SOCK_DGRAM, 0);

/* Iterate through all interfaces */
getifaddrs(&ifap);
for (ifa = ifap; ifa != NULL; ifa = ifa->ifa_next) {
    /* Grab statistics name and values */
    strcpy(ifr.ifr_name, ifa->ifa_name);
    drvinfo.cmd         = ETHTOOL_GDRVINFO;
    ifr.ifr_data        = (caddr_t) &drvinfo;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    n_stats             = drvinfo.n_stats;
    strings->cmd        = ETHTOOL_GSTRINGS;
    strings->string_set = ETH_SS_STATS;
    strings->len        = n_stats;
    ifr.ifr_data        = (caddr_t) strings;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    stats->cmd          = ETHTOOL_GSTATS;
    stats->n_stats      = n_stats;
    ifr.ifr_data        = (caddr_t) stats;
    ioctl(skfd, SIOCETHTOOL, &ifr);

    ifIndex = if_nametoindex(ifa->ifa_name);

    /* Iterate through statistics */
    for (i = 0; i < n_stats; i++) {

        /* Declare the appropriate index */
        strncpy(ethtoolStatName,
                (char *)&strings->data[i * ETH_GSTRING_LEN],
                ETH_GSTRING_LEN);
        ethtoolStatName[sizeof(ethtoolStatName) - 1] = '\0';
        ethtoolStatName_len = strlen(ethtoolStatName);

        /* Append this new variable to the container. */
        rowreq_ctx = ethtoolStatTable_allocate_rowreq_ctx();
        ethtoolStatTable_indexes_set(rowreq_ctx,
                    ifIndex,
                    ethtoolStatName, ethtoolStatName_len);
        rowreq_ctx->data.ethtoolStat.high = stats->data[i] >> 32;
        rowreq_ctx->data.ethtoolStat.low = (uint32_t)stats->data[i];

        CONTAINER_INSERT(container, rowreq_ctx);
    }
}

See? It is pretty easy. I have only enumerated all the objects I wanted to expose. I didn’t even have to convert statistics names to an OID, mib2c built a ethtoolStatTable_indexes_set() for this purpose. OpenHPI project provides an extensive documentation to write a sub-agent this way.

Now, on the downside, to implement a single table, I have 14 generated files whose code style is usually unfit to integrate into an existing project : some people, me included, do not like automatically generated code, all the more when it is so verbose. The solution would be to use the new API without mib2c; unfortunately, while the documentation exists, all provided examples rely exclusively on mib2c tool. Unless you are able to grasp the new API without it, I would restrict its use to two cases:

• Writing a new module for inclusion into Net-SNMP. Since most modules are now written with the help of mib2c, there is no coding style problem.
• Writing a standalone sub-agent for some MIB. As a standalone project, you won’t run into coding style problems either.

For other uses, let’s have a look at the traditional API.

C with “traditional” API

The traditional API is way simpler and even if it is quite old, it is easier to find associated examples. Here is how to initialize the agent, again, in a stripped-down version (the complete version is on GitHub):

static oid ethtool_oid[] = {1, 3, 6, 1, 4, 1, 39178, 100, 1};
static struct variable3 ethtool_vars[] = {
  {1, ASN_COUNTER64, RONLY, ethtool_stat, 3, {1, 1, 2}}
};

int main()
{
  netsnmp_enable_subagent();
  snmp_disable_log();
  snmp_enable_stderrlog();
  init_agent("ethtoolAgent");
  REGISTER_MIB("ethtoolStatTable", ethtool_vars,
               variable3, ethtool_oid);
  init_snmp("ethtoolAgent");
  while (1)
    agent_check_and_process(1);
}

The REGISTER_MIB() macro is used to register the provided root OID (ethtool_oid) to the master agent. It also registers handlers associated to various sub-OID through the ethtool_vars[] array. Each member of this array features the following fields:

• magic is an integer allowing one to discriminate OID handled by the same handler. For a table, this allows to handle several columns with the same handler since they share a common index.
• type is the type of the variable that will be returned.
• acl tells if the object is read-only (RONLY) or read-write (RWRITE).
• findVar is the handler associated to the object. Its task will be to find the appropriate object and return its value.
• The two last fields are the relative OID this entry is responsible for along with its length.

The prototype of ethtool_stat() handler function is the following:

static u_char*
ethtool_stat(struct variable *vp, oid *name, size_t *length,
             int exact, size_t *var_len, WriteMethod **write_method);

This function should return the value of the requested variable or NULL if no appropriate object was found. The value can be statically stored since it will be copied. Here is the description of the arguments:

• struct variable *vp is the structure the handle was associated with, except that vp->name is now a full OID, not a relative one. You can access vp->magic to determine which column or which scalar to handle if this handler is common to several objects.
• oid *name and size_t *length describe the OID requested. If you want to return another OID (for a GETNEXT request for example), you can modify them. name points to an OID array large enough to fit any OID you want to return.
• int exact tells if the requested OID should be exactly matched (case of GET or SET) or not (GETNEXT).
• size_t *var_len is used to tell the size of the returned object.
• WriteMethod **write_method is only used if the variable is modifiable. When a SET request is received, the API will call the handler with exact set to 1 and expects to get a pointer to the function that will handle the write operation³.

To implement this function, you are on your own. When relying on externally fetched data, you need to handle a cache yourself. I have used a red-black tree for this purpose (with code stolen from OpenBSD). This also enables an easy implementation of the GETNEXT operation.

Here are three projects using this API to provide SNMP support:

• Keepalived is a VRRP daemon and a monitoring daemon for LVS clusters. I have added a complete SNMP support. The most interesting files are core/snmp.c, check/check_snmp.c and vrrp/vrrp_snmp.c. It features a tight integration into Keepalived event loop that should be reusable by other projects. It includes read-only support of all data structures (configuration, state and statistics), ability to send traps on events and a write support for some values.
• lldpd is an implementation of 802.1AB (LLDP) which is a protocol allowing an equipment to advertise itself to its neighbors on Ethernet level. 802.1AB comes with a huge MIB module and many extensions. lldpd provides a fairly complete read-only implementation of this MIB module. Some tables have complex indexes. Have a look at agent.c. There is currently no write support and no traps but lldpd features privilege separation and some code allows it to still use an Unix socket to speak with the master agent.
• Asterisk, an open source PBX, includes an AgentX sub-agent allowing one to grab some metrics. Look at res/snmp/agent.c.

UPDATED: If it is critical for your project to not block, you should be aware that using NetSNMP implementation of AgentX has some issues. Have a look at my other post on this topic.

Other solutions

The manual page for snmpd.conf explores other ways to extend Net-SNMP agent under the section “Extending agent functionality“:

• exec and sh are deprecated. Use extend instead.
• pass is a bit simpler than pass_persist. The command is run once for each request. Such a command can easily be turned into a pass_persist script. Moreover, Net-SNMP will not answer any request while a pass command is running.
• proxy redirects requests to another SNMP agent. Unless you have some code that can only act as a complete SNMP agent, it is better to use AgentX instead. For programs using Net-SNMP API, turning an agent into an AgentX sub-agent is just one line of code. Since a sub-agent involves less code than a complete agent, it is more efficient and more reliable.
• smux has been superseeded by AgentX.
• perl and dlmod allow one to load and execute external code inside the running agent. Unless you have important performance requirements, it is better to use a sub-agent instead: if some bug happens, only the sub-agent will be affected, not the master agent. A sub-agent sticks more closely to the Unix philosophy: “write programs that do one thing and do it well”. However, Net-SNMP FAQ says:

Implementing the module in C within the main agent (directly or via dlmod) is probably the most efficient and reliable, closely followed by embedded perl (or python) extensions. These have the advantage of minimal overheads between the code implementing the MIB module, and the agent framework, and no inter-process communication issues.

The documentation on this component is scarce and it is difficult to find up-to-date bits on how the route cache works and how to tune it. The book Understanding Linux Network Internals from O’Reilly is an exception and contains valuable information on how the route cache works. Even if the book is targeted at 2.6.12, the part on the route cache is still quite accurate. Unfortunately, it fails to provide appropriate tips on how to monitor and tune the route cache.

I hope to provide here a concise view of the route cache, as it is implemented in Linux 3.1¹. It is protocol-dependent² and I will cover only the IPv4 version here.

Overview of the route cache subsystem

To handle an incoming or outgoing IP datagram, the kernel needs to issue a lookup in the route tables. While it seems to be quite trivial, several questions need to be answered:

• Does the source address and the destination address appear to be valid?
• Is the source address a martian address³?
• Is the destination address mine or should I forward the packet?
• Which route table should I use?
• Does the destination address match this route? And this one?
• Can I currently contact the gateway that I should use?

Those checks can be a bit time consuming, even for small route tables. To avoid them for each packets, Linux maintains a route cache which is queried before doing a regular lookup and updated after each one. You can dump it with ip -s route show cache:

$ ip -s route show cache
198.51.100.7 from 203.0.113.2 via 192.0.2.1 dev eth1
  cache used 7 age 2sec ipid 0x1fce rtt 131ms rttvar 45ms cwnd 10
198.51.100.17 from 203.0.113.15 via 192.0.2.1 dev eth1
  cache used 3 age 0sec ipid 0xc3bd
local 192.0.2.18 from 203.0.113.15 dev lo src 192.0.2.18
  cache <local> used 154 age 1sec iif eth0

Here are two examples:

• If Linux receives a packet from 198.51.100.17 to 203.0.113.15, it will find this flow in the route cache. Therefore, it already knows that it should forward the packet to 192.0.2.1. No checks needed.
• If it receives a packet from 198.51.100.16 to 203.0.113.15, there is no appropriate entry in the route cache and therefore, the system will have to look at the route tables. It is likely to use the exact same entry than if the source was 198.51.100.17 but maybe there is some policy routing requesting the use of a special route table or 198.51.100.16 is a local address and the packet will therefore be classified as martian.

The schema below shows how this cache is implemented⁴. It uses a separate hash table (BSD systems keep the cache in the routing table). Each bucket is a chained list of route cache entries.

Once an entry has been added to the route cache, there are several ways to remove it. Most entries are removed by the garbage collector which will scan the route cache and remove invalid and older entries. It will be triggered when the route cache is full or at regular interval, once a certain threshold has been met.

Available knobs

There are several values you can tune. Most of them are available in sysctl.

• rhash_entries is the size of the hash table⁵. If you don’t specify it on the kernel command line, it is computed dynamically based on the memory available on your system. You can view its value by looking at something like IP route cache hash table entries: 262144 (order: 9, 2097152 bytes) in the kernel logs.
• net.ipv4.route.max_size is the maximum number of entries in the route cache. Except under exceptional circumstances, this value is never exceeded.
• net.ipv4.route.gc_elasticity is the target average length of a chain in the route cache. The garbage collector will work harder if this value is exceeded. This means that if you multiply this value by the value of rhash_entries, you will get the target average number of entries in the route cache.
• net.ipv4.route.gc_min_interval_ms is the minimum delay between two runs of the garbage collector, except when the cache is full. The default value should be fine.
• net.ipv4.route.gc_thresh is a threshold triggering the garbage collector every net.ipv4.route.gc_min_interval_ms milliseconds.
• net.ipv4.route.gc_timeout is the base value to determine if an entry is old enough to be removed or not. Whatever its value, the garbage collector will attempt to remove the same number of entries. However, this value could potentially influence its efficiency. See below for more details on this.

You may find documentation about those obsolete sysctl values:

• net.ipv4.route.secret_interval has been removed in Linux 2.6.35; it was used to trigger an asynchronous flush at fixed interval to avoid to fill the cache.
• net.ipv4.route.gc_interval has been removed in Linux 2.6.38. It is still present until Linux 3.2 but has no effect. It was used to trigger an asynchronous cleanup of the route cache. The garbage collector is now considered efficient enough for the job.

UPDATED: net.ipv4.route.gc_interval is back for Linux 3.2. It is still needed to avoid exhausting the neighbour cache because it allows to cleanup the cache periodically and not only above a given threshold. Keep it to its default value of 60.

Statistics & monitoring

Linux maintains some statistics about the use of the route cache. You can find them in /proc/net/stat/rt_cache. The command lnstat can print them nicely for you:

$ lnstat -s1 -i1 -c-1 -f rt_cache
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
 entries|  in_hit|in_slow_|in_slow_|in_no_ro|  in_brd|in_marti|in_marti| out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis|
        |        |     tot|      mc|     ute|        |  an_dst|  an_src|        |    _tot|     _mc|        |      ed|    miss| verflow| _search|t_search|
 3096848|   42309|     686|       0|       0|       0|       0|       0|       3|       0|       0|     674|     672|       0|       0|   27644|       8|
 3096984|   41405|     636|       0|       0|       0|       0|       0|       3|       0|       0|     623|     621|       0|       0|   28189|       8|
 3097160|   42483|     700|       0|       0|       0|       0|       0|       5|       0|       0|     694|     692|       0|       0|   29506|      12|

Except for the first column, lnstat outputs values in units per second. Let’s review some of those values:

• rt_cache_entries is the number of entries in the route cache. You should compare it with net.ipv4.route.max_size and ensure that the cache is never full to avoid triggering the garbage collector too often.
• rt_cache_in_hit and rt_cache_out_hit are the number of regular lookups avoided because the result was found in the cache for incoming and outgoing packets, respectively. When the system is a router, most lookups only happen on the incoming side. You should compare this value with rt_cache_in_slow_tot and rt_cache_in_out_slow_tot which are the number of lookups in the route tables. On this system, the efficiency of the route cache is more than 98% which is quite good.
• rt_cache_gc_total is how often the garbage collector was requested to be triggered while rt_cache_gc_ignored corresponds to how often it was finally not run because it has already been triggered shortly beforehand (less than net.ipv4.route.gc_min_interval_ms milliseconds). The difference between those two values should be very small to ensure that the garbage collector does not run more a handful of times per second.
• rt_cache_gc_goal_miss is how often the garbage collector was not able to fulfill its goal. This value should rarely be different of 0.
• rt_cache_gc_dst_overflow is how often the route cache is bigger than the allowed maximum size. This should never happen except when you try to shrink the cache.
• rt_cache_in_hlist_search and rt_cache_out_hlist_search is how often a lookup in the cache had to look at the next entry in the chain for the computed bucket: each time Linux has to follow the next pointer in a cache entry, it increments one of those counters. It is a clue on the average length of chains in the route cache hash table. Compare those values with the number of cache lookups (hit and miss).

As an illustration, here is a plot of the various statistics exposed above for a router whose route cache receives about 1000 routes par second:

rhash_entries is 1,048,576, as is net.ipv4.route.gc_thresh. Therefore, the garbage collector was requested to be run when the number of entries goes above this level. Because the cache is not full, it is only really run twice per seconds (the value of net.ipv4.route.gc_interval_ms is 500). net.ipv4.route.gc_elasticity is set to 3. This explains why the garbage collector is aggressive when the number of entries reached 3,145,728.

As you can see, the efficiency is near 100% all the time. The percentage of collisions is rt_cache_in_hlist_search ratio to the sum of rt_cache_in_hit and rt_cache_in_slow_tot.

UPDATED: The plot above was with a 2.6.39 kernel. For a kernel between 2.6.35 and 2.6.37 (included) or a 3.2 kernel or more recent, the cleanup triggered every net.ipv4.route.gc_interval seconds will expire up to rhash_entries entries. If the pace at which routes are added to the cache is less than this rate, the number of entries may stop climbing, even when net.ipv4.route.gc_threshold is not met. For example, here are the same statistics with a 2.6.35 kernel for a router with about 2,500 new entries per second but with net.ipv4.route.gc_interval enabled; the threshold of 1,048,576 is never met:

Tuning

Do you need to modify any of those values? You have two questions to ask yourself:

1. How much efficiency do you want to get from the route cache?
2. How much memory do you want to dedicate to the route cache?

As a rule of thumb, two millions entries eat about 500 MB of memory on a 64bit system. You should be able to compute the average memory usage and the maximum memory usage from the values of net.ipv4.route.max_size, rhash_entries and net.ipv4.route.gc_elasticity. For example, if the route cache hash table has 262,144 buckets, the maximum allowed number of entries in the cache is 4,194,304 and net.ipv4.route.gc_elasticity is set to 8, the memory usage will be 500 MB on average and 1 GB max. If this is too much, you will need to lower some values.

Look at the previous section to compute the current efficiency of your route cache. It should be above 90%. If you are dissatisfied with that, you could increase the cache size.

If you want to double the number of entries, double net.ipv4.route.max_size, net.ipv4.route.gc_thresh and rhash_entries but keep net.ipv4.route.gc_elasticity to 8.

For the garbage collector to be efficient, the route cache should not be filled too fast. The garbage collector should be able to cope with this situation but this may impact performance because it needs to walk several times the route cache to find entries to expire. Watching gc_goal_miss may give you a hint about this: if this value starts to be different of 0, lower net.ipv4.route.gc_timeout.

UPDATED: For a kernel where net.ipv4.route.gc_interval matters, things become more complicated. Because the cleanup algorithm will expire entries at a regular interval, the average number of entries may stay low unless the number of new entries per second is high enough. Therefore, the average number of entries may be lower than the theoretical value computed above. Monitoring the appropriate metrics is the key to a good tuning. If you feel that entries are expired too fast, you may want to double net.ipv4.route.gc_timeout.

In-depth look into the garbage collector

Those different pieces of advice may have puzzled you. We need to understand how the garbage collector works to better cope with them. The garbage collector is triggered when a new entry needs to be added to the cache and the number of entries is superior to net.ipv4.route.gc_thresh. It is implemented in rt_garbage_collect() function. It will do nothing if it has been called less than net.ipv4.route.gc_interval_ms milliseconds and the cache is not full (more than net.ipv4.route.max_size entries).

Setting a goal

The garbage collector will first assess the situation. It will look how the number of entries currently in the cache compares to the product of rhash_entries and net.ipv4.route.gc_elasticity:

1. Above this limit, it will try to remove at least rhash_entries entries.
2. Otherwise, it will try to remove no more than half the difference.

If you look at the previous plot, you can easily see the difference when the garbage collector is not aggressive (above 1,048,576 entries but below 3,145,728) and when it is (above 3,145,728 entries). If we assume the garbage collector is able to meet its goal, we can easily simulate its algorithm:

The first plot shows what would happen if about 2000 routes are added per second with rhash_entries equal to 262,144 and net.ipv4.route.gc_elasticity set to 8. When net.ipv4.route.gc_threshold is met, the garbage collector has almost no effect. However, when the number of entries reaches 2,097,152, the garbage collector sets its goal to 262,144. From this point, the number of entries oscillates around 2 millions entries.

On the second plot, rhash_entries is now equal to 1,048,576 but net.ipv4.route.gc_elasticity has been set to 2. Therefore, the aggressive part of the garbage collectors kicks at the same threshold than for the previous plot. However, its goal is now four times larger and the oscillations have a larger amplitude. The fact that the aggressive part of the garbage collector kicks less often is nullified by the fact that it needs to remove more entries each time. Because of the slight impact on cache efficiency, it seems better to keep net.ipv4.route.gc_elasticity around 8, or 4 if we want to keep shorter chains (but there seems to be no improvement to do so).

The other plots show what happens when there is a sudden surge in the number of routes added or when there is a pause.

Meeting the goal

Now that we understand how rhash_entries, net.ipv4.route.gc_elasticity and net.ipv4.route.gc_threshold interact, let’s look at net.ipv4.route.gc_timeout. Once the garbage collector has set its goal and if it is positive, it needs to choose which entries to remove from the cache.

It walks the hash table from the position of its last run and remove entries until its goal is met. If the entry is not current anymore (for example, the network interface associated to it has changed its IP configuration), it is removed. Otherwise, the system looks at the age of the entry and its position in the chain. If the entry is the first in the chain, it is kept only if its age is below net.ipv4.route.gc_timeout. If it is second, it is kept only if its age is below half of net.ipv4.route.gc_timeout. If it is third, the threshold is a quarter of net.ipv4.route.gc_timeout, and so on. The garbage collector will favor short chains.

If after a full run of the hash table, the garbage collector was not able to meet its goal, it will start again but will behave more aggressively, as if net.ipv4.route.gc_timeout is set to half of its value. It will do as many passes as necessary until its goal is met or until there is no way to remove any entry (or until it has spent too much time). Once the garbage collector has switched to this more aggressive behavior, it will keep being aggressive for a few cycles (a bit less for each cycle).

With a very large value of net.ipv4.route.gc_timeout, the garbage collector will have a hard time to find entries to expire. It will need to do several passes until it is able to expire some entries. On the other hand, if you choose a very small value, the garbage collector may remove entries that were just added, even if there are older entries further in the hash table.

For a more in-depth coverage of how the route cache works, look at chapter 33 of Understanding Linux Network Internals. Just be aware that multipath route caching has been removed (and was never really used) and asynchronous cleanup (rt_periodic_timer) does not exist anymore.

UPDATED: As stated earlier, net.ipv4.route.gc_interval has been reinstantiated in Linux 3.2. The cleanup is a bit different of what is done by the garbage collector but have some similarities. It is run every net.ipv4.route.gc_interval seconds (even when there is less than net.ipv4.route.gc_threshold entries). It will set a goal proportional to net.ipv4.route.gc_interval and inversely proportional to net.ipv4.route.gc_timeout. It cannot be greater than rhash_entries. If net.ipv4.route.gc_interval and net.ipv4.route.gc_timeout are equal, the goal is exactly rhash_entries. It represents the number of entries the cleanup procedure will look at (and not the number of entries it will try to expire). If an entry is not valid anymore or is old enough to be removed (with the same criteria as for the garbage collector), it will be removed. Another important thing about this cleanup algorithm is that it will modify the maximum allowed length of a chain to the average length it has observed plus four times the standard deviation with a maximum equal to net.ipv4.route.gc_elasticity. Without this algorithm, the maximum allowed length is 20. When inserting a new entry, if a chain with more than net.ipv4.route.gc_elasticity entries is selected, the kernel will try to remove an element before inserting a new one. Then, if the chain is still longer than the maximum allowed length (longer than what is allowed by net.ipv4.route.gc_elasticity or too long compared to other chains⁶), all cache entries for the current interface are invalidated.

Conclusion

When tuning the route cache, rhash_entries, net.ipv4.route.gc_elasticity, net.ipv4.route.max_size and net.ipv4.route.gc_threshold are related and should not be modified independently. net.ipv4.route.gc_thresh should be below the product of net.ipv4.route.gc_elasticity and rhash_entries while net.ipv4.route.max_size should be above this value.

Linux exposes several interesting metrics related to the route cache. Monitoring them allows one to watch the efficiency of the route cache and may uncover some anomalies (garbage collector running too often, difficulty to remove routes from the cache, …).

The route cache subsystem still evolves and some old behaviors have been dismissed. The best documentation is, unfortunately, still the code and other documentations tend to become obsolete. The IPv6 route cache is quite different of the one presented here. Don’t blindly apply the same tuning for IPv6.

Forward secrecy allows today information to be kept secret even if the private key is compromised in the future. Achieving this property is usually costly and therefore, most web servers do not enable it on purpose. Google recently announced support of forward secrecy on their HTTPS sites. Adam Langley wrote a post with more details on what was achieved to increase efficiency of such a mechanism: with a few fellow people, he wrote an efficient implementation of some elliptic curve cryptography for OpenSSL.

Without forward secrecy

To understand the problem when forward secrecy is absent, let’s look at the classic TLS handshake when using a cipher suite like AES128-SHA. During this handshake, the server will present its certificate and both the client and the server will agree on a master secret.

This secret is built from a 48byte premaster secret generated and encrypted by the client with the public key of the server. It is then sent in a Client Key Exchange message to the server during the third step of the TLS handshake. The master secret is derived from this premaster secret and random values sent in clear-text with Client Hello and Server Hello messages.

This scheme is secure as long as only the server is able to decrypt the premaster secret (with its private key) sent by the client. Let’s suppose that an attacker records all exchanges between the server and clients during a year. Two years later, the server is decommissioned and sent for recycling. The attacker is able to recover the hard drive with the private key. She can now decrypt any session she recorded: the encrypted premaster secret sent by a client is decrypted with the private key and the master secret is derived from it. The attacker can now recover passwords and other sensitive information that can still be valuable today.

The main problem lies in the fact that the private key is used for two purposes: authentication of the server and encryption of a shared secret. Authentication only matters while the communication is established, but encryption is expected to last for years.

Diffie-Hellman with discrete logarithm

One way to solve the problem is to keep using the private key for authentication but uses an independent mechanism to agree on a shared secret. Hopefully, there exists a well-known protocol for this: the Diffie-Hellman key exchange. It is a method of exchanging keys without any prior knowledge. Here is how it works in TLS:

1. The server needs to generate once (for example, with openssl dhparam command):
   • ·p·, a large prime number,
   • ·g·, a primitive root modulo ·p· (for every integer ·a· coprime to ·p·, there exists an integer ·k· such that ·g^k=a\mod p·).
2. The server picks a random integer ·a· and compute ·g^a \mod p·. After sending its regular Certificate message, it will also send a Server Key Exchange message (not included in the handshake depicted above) containing, unencrypted but signed with its private key for authentication purpose:
   • random value from the Client Hello message,
   • random value from the Server Hello message,
   • ·p·, ·g·,
   • ·g^a\mod p=A·.
3. The client checks that the signature is correct. It also picks a random integer ·b· and sends ·g^b \mod p=B· in a Client Key Exchange message. It will also compute ·A^b\mod p=g^{ab}\mod p· which is the premaster secret from which the master secret is derived.
4. The server will receive ·B· and compute ·B^a\mod p=g^{ab}\mod p· which is the same premaster secret known by the client.

Again, the private key is only used for authentication purpose. An eavesdropper will only know ·p·, ·g·, ·g^a\mod p· and ·g^b\mod p·. Computing ·g^{ab}\mod p· from those values is the discrete logarithm problem for which there is no known efficient solution.

Because the Diffie-Hellman exchange described above always uses new random values ·a· and ·b·, it is called Ephemeral Diffie-Hellman (EDH or DHE). Cipher suites like DHE-RSA-AES128-SHA use this protocol to achieve perfect forward secrecy¹.

To achieve a good level of security, parameters of the same size as the key are usually used (the security provided by the discrete logarithm problem is about the same as the security provided by factorisation of two large prime numbers) and therefore, the exponentiation operations are pretty slow as we can see in the benchmark below:

Diffie-Hellman with elliptic curves

Fortunately, there exists another way to achieve a Diffie-Hellman key exchange with the help of elliptic curve cryptography which is based on the algebraic structure of elliptic curves over finite fields. To get some background on this, be sure to check first Wikipedia article on elliptic curves. Elliptic curve cryptography allows one to achieve the same level of security than RSA with smaller keys. For example, a 224bit elliptic curve is believed to be as secure as a 2048bit RSA key.

Some theory

Diffie-Hellman key exchange described above can easily be translated to elliptic curves. Instead of defining ·p· and ·g·, you get some elliptic curve, ·y^2=x^3+\alpha x+\beta·, a prime ·p· and a base point ·G·. All those parameters are public. In fact, while they can be generated by the server, this is a difficult operation and they are usually chosen among a set of published ones.

The use of elliptic curves is an extension of TLS described in RFC 4492. Unlike with the classic Diffie-Hellman key exchange, the client and the server need to agree on the various paremeters. Most of this agreement is done inside Client Hello and Server Hello messages. While it is possible to define some arbitrary parameters, web browsers will only support a handful of predefined curves, usually NIST P-256, P-384 and P-521. From here, the key exchange with elliptic curves is pretty similar to the classic Diffie-Hellman one:

1. The server picks a random integer ·a· and compute ·aG· which will be sent, unencrypted but signed with its private key for authentication purpose, in a Server Key Exchange message.
2. The client checks that the signature is correct. It also picks a random integer ·b· and sends ·bG· in a Client Key Exchange message. It will also compute ·b\cdot aG=abG· which is the premaster secret from which the master secret is derived.
3. The server will receive ·bG· and compute ·a\cdot bG=abG· which is the same premaster secret known by the client.

An eavesdropper will only see ·aG· and ·bG· and won’t be able to compute efficiently ·abG·.

Using ECDHE-RSA-AES128-SHA cipher suite (with P-256 for example) is already a huge speed improvement over DHE-RSA-AES128-SHA thanks to the reduced size of the various parameters involved.

Web browsers only support a handful of well-defined elliptic curves, chosen to ease an efficient implementation. Bodo Möller, Emilia Käsper and Adam Langley have provided 64bit optimized versions of NIST P-224, P-256 and P-521 for OpenSSL. To get even more details on the matter, you can read the end of the introduction on elliptic curves from Adam Langley, then a short paper from Emilia Käsper which presents a 64bit optimized implementation of the NIST elliptic curve NIST P-224.

In practice

First, keep in mind that elliptic curve cryptography is not supported by all browsers. Recent versions of Firefox and Chrome should handle NIST P-256, P-384 and P-521 but for most versions Internet Explorer, you are currently out of luck. Therefore, you need to keep accepting other cipher suites.

You need a recent version of OpenSSL. Support for ECDHE cipher suites has been added in OpenSSL 1.0.0. Check with openssl ciphers ECDH that your version supports them. If you want to use the 64bit optimized version, you need to run a snapshot of OpenSSL 1.0.1, configured with enable-ec_nistp_64_gcc_128 option. A recent GCC is also required in this case.

Next, you need to choose the appropriate cipher suites. If forward secrecy is an option for you, you can opt for ECDHE-RSA-AES128-SHA:AES128-SHA:RC4-SHA cipher suites which should be compatible with most browsers. If you really need forward secrecy, you may opt for ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:EDH-DSS-DES-CBC3-SHA instead.

Then, you need to ensure the order of cipher suites is respected. On nginx, this is done with ssl_prefer_server_ciphers on. On Apache, this is SSLHonorCipherOrder on.

UPDATED: You need to check ECDHE support for your web server. For nginx, the support has been added in 1.0.6 and 1.1.0. The curve selected defaults to NIST P-256. You can specify another one with ssl_ecdh_curve directive. For Apache, it has been added in 2.3.3 and does not exist in the current stable branch. Adding support for ECDHE is quite easy. You can check how I added it in stud. This issue also exists for DHE cipher suites, in which case you also might have to specify DH parameters to use (generated with openssl dhparam) using some special directive or by appending the parameters to the certificate. Check Immerda Techblog article for more background about this point.

The implementation of TLS session tickets may be incompatible with forward secrecy, depending on how they are implemented. When they are protected by a random key generated at the start of the server, the same key could be used for months. Some implementations² may derive the key from the private key. In this case, forward secrecy is broken. If forward secrecy is a requirement for you, you need to either disable tickets or ensure that key rotation happens often.

Check with openssl s_client -tls1 -cipher ECDH -connect 127.0.0.1:443 that everything works as expected.

Some benchmarks

With the help of the micro-benchmark tool that I developed for my previous article, we can compare the efficiency of cipher suites providing forward secrecy:

I have used a snapshot of OpenSSL 1.0.1 (2011/11/25). The optimized version of ECDHE is the one you get by using enable-ec_nistp_64_gcc_128 option when configuring OpenSSL.

Let’s focus on the server part. Enabling DHE-RSA-AES128-SHA cipher suite hinders the performance of TLS handshakes by a factor of 3. Using ECDHE-RSA-AES128-SHA instead only adds an overhead of 27%. However, if we use the 64bit optimized version, the cost is only 15%. The overhead is only per full TLS handshake. If 3 out of 4 of your handshakes are resumed, you need to adjust the numbers.

Your mileage may vary but the computational cost for enabling perfect forward secrecy with an ECDHE cipher suite seems a small sacrifice for better security.

Typekit provides a subscription-based library of hosted, high-quality fonts to use on websites. I expose here my own, mostly uneducated¹ on the typographic side, look on this service.

Fonts embedding 101

There are several ways to embed fonts for the Web. Let’s have a look at the most common ones.

Self-hosting and @font-face

A first possibility is to host fonts yourself and provide a CSS file with the appropriate @font-face declarations. There are five important things to know:

1. There is no common format. You will need to host different versions of the same font to support most browsers.
2. Of course, Internet Explorer comes with its own quirks that you need to work around.
3. Most browsers enforce a same origin policy. You need to serve your fonts from the same domain as the requesting page. You can circumvent such a limitation by enabling cross-origin resource sharing: add an Access-Control-Allow-Origin header when serving fonts.
4. Fonts are unlikely to change. To leverage caching, configure an explicit expiration date far in the future. Something like 30 days.
5. There is no MIME types registered for fonts. Don’t bother setting one, even if Google Chrome likes to complain about them in the console.

One great resource is Font Squirrel which provides kits with fonts and the appropriate CSS declarations. You choose your font family (or upload one of your own), the formats that you want to use (check them all) and you will get a ZIP archive with all fonts and a CSS file. You may want to tweak it a bit to enable automatic use of the bold variant when needed and to use a local copy if available. For example, if you choose Crimson, an oldstyle typeface, you may want to declare the bold version with font-weight: bold and add local names:

@font-face {
    font-family: 'Crimson';
    src: url('Crimson-Bold-webfont.eot');
    src: local('Crimson Bold'), local('Crimson Text Bold'),
         url('Crimson-Bold-webfont.eot?#iefix') format('embedded-opentype'),
         url('Crimson-Bold-webfont.woff') format('woff'),
         url('Crimson-Bold-webfont.ttf') format('truetype'),
         url('Crimson-Bold-webfont.svg#CrimsonBold') format('svg');
    font-weight: bold;
    font-style: normal;
}

Google Web Fonts

For open-source fonts, you can also use Google Web Fonts instead of hosting the files yourself. Build a collection of fonts and choose the variants that you want to use for each of them. You will then be provided with a code snippet to paste in the <head> element of your page. For example, for Crimson:

<link href='http://fonts.googleapis.com/css?family=Crimson+Text:400,700,400italic'
      rel='stylesheet' type='text/css'>

Why choose this solution over self-hosting?

1. You don’t use your own bandwith to serve fonts. This may be important if you pay for it.
2. Google uses its own CDN to serve fonts. If you don’t use a CDN yourself, this can be a major performance improvement.
3. Caching is more efficient since many sites will use the same font.
4. It’s easier to do it right.

Unless you are concerned about Google tracking your users², I think this is a better solution than self-hosting. Quick delivery of fonts is pretty important since content rendering can be delayed by them.

Paid services

If you would like to use commercial fonts, like Adobe Minion Pro or Paratype Futura PT, a convenient option is to rent them. Typekit is service providing such an option. Pricing depends on volume and the number of fonts you want to have access to. There is a free plan and the first paid plan is US$24.99 per year. Other similar services exist. Feel free to compare.

Once registered, the workflow is similar to Google Web Fonts. You choose your fonts, assemble them in a “kit” and publish it. Instead of directly providing a CSS file, Typekit provides some Javascript which will in turn download a CSS file containing all fonts in the appropriate format for the browser (on most browsers, fonts are inlined).

<script type="text/javascript"
        src="//use.typekit.com/tyt0atd.js"></script>
<script type="text/javascript">
  try{Typekit.load();}catch(e){}
</script>

Typekit: the good parts…

So, why choose Typekit? First, there is the very large choice of fonts. The first paid plan comes with more than 500 fonts from about 80 foundries. There are about a hundred serif fonts and a couple hundred sans serif ones.

Typekit provides a convenient interface to browse fonts. You can either search fonts by name or by characteristics. In the last case, you can filter on properties like the classification (serif, sans serif, decorative, etc.), weight, width, contrast, x-height, recommended use or language support. Once a font is selected and added to your kit, you can easily choose what variants you want. The kit total size is displayed.

If you happen to meet any problem or have some questions, Typekit support is very responsive. I got clear and crisp answers in less than a few hours in the week-end.

…and the bad ones

Unfortunately, Typekit has also some shortcomings. One of the most commonly cited is that Javascript is mandatory. No Javascript, no fonts. If you want to display the appropriate fonts for users without Javascript, you should just not use Typekit.

Not free, nor free

While Typekit offers some open-source fonts, most of them are not. Moreover, only a few of them are available in the free plan. While I would prefer to use open-source fonts, I don’t consider this a strong requirement: if you are dissatisfied for some reason, switching back to open-source fonts (or another service) is fast and easy.

I like serif fonts a lot and they get less attention from the FLOSS world: it is difficult to find one whose readability on screen is excellent. Droid Serif is a nice font but is a bit too black for me. Maybe Cardo would be a good match.

You may disagree and in this case, Google Web Fonts or self-hosting seem the perfect match.

Difficulty to choose and test fonts

While browsing fonts is easy and smooth, choosing and testing them is another matter. For example, Typekit is missing some interface to compare font rendering for whole paragraphs at a given size. It also misses a way to check how two fonts render together, one for headings and one for paragraphs. Here, Google Web Fonts is more versatile.

Things become really difficult when you want to test your selected fonts for real. Except if you have invested into one of the expensive plans, you can only include a limited number of fonts in your kit. Each time you want to try another set of fonts, you need to modify your kit and wait for it to be distributed across Typekit network. You may have to wait five or ten minutes between each change!

Another inconvenient thing is that once you publish your site, any change to the kit will affect the published version. Therefore, you need to use a second kit for development and play back and forth with those two kits.

What would be convenient here is a special kit with all fonts from the library with restricted access to some IP to avoid abuse. This is something done by Fontdeck, a competitor.

Poor subset support

Because you want to keep the font size small (about 30 kB for each variant), Typekit usually proposes two subsets: the default one contains only latin characters and the complete one with all glyphs. If some glyph is absent from the default subset, you need to use the complete one which is roughly two to four times the size of the default one. This is a bit huge for only one character.

What would be nice is to have subsets for each language. In my case, I would take a French subset because I need the “œ” glyph which appears in French in words like “cœur” (heart or core) or “œil” (eye). I don’t want to select the complete subset just for this character.

When a glyph is absent, the next font in the CSS stack is used. For example, if the selected font is FF Tisa Web Pro and the next font in the stack is Droid Serif, here is how the word “cœur” may be rendered. On the first line, FF Tisa Web Pro contains the appropriate glyph. On the second line, the glyph is absent and taken from Droid Serif. Since x-height and contrast are a bit different, the replaced glyph seems out of tone. At small sizes, the difference accounts for more than 10%.

I have also added a third line where the ligature is built from “o” and “e” and the following style is applied to “e”:

.lf-ligature {
    margin-left: -0.172em;
    zoom: 1; /* Here to trigger hasLayout on IE7 */
}

As you can see, the result seems good. I have a piece of Javascript turning a ligature into the appropriate markup. It can be extended to handle more ligatures and could be adapted to handle other composed characters (like the ones in “Antonín Dvořák”).

Defective caching

Because a document can only be rendered after the fonts have been loaded³, it is pretty important to deliver fonts as fast as possible. Typekit uses Edgecast as a CDN and DynECT for DNS hosting.

Two files are downloaded. Both of them are hosted on use.typekit.com. The first one is a script that will act as a loader. Based on the characteristics of your browser, it will choose how to download the requested fonts for best rendering. The second file is generally a big CSS file containing the fonts.

Here are the headers returned for the first file:

HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=300
Content-Encoding: gzip
Content-Type: text/javascript
Date: Tue, 08 Nov 2011 18:34:16 GMT
ETag: "388467610+gzip"
Expires: Tue, 08 Nov 2011 18:39:16 GMT
Last-Modified: Sat, 05 Nov 2011 09:31:57 GMT
Server: ECS (cdg/D624)
Vary: Accept-Encoding
X-Cache: HIT
Content-Length: 7440

First, notice that the content is served compressed. Good point. The next headers to look at are Cache-Control, Date, ETag and Expires: the server requests the content to be cached for 5 minutes. When the browser notices that its copy is out-of-date, it will issue a new request including a If-None-Match: 388467610+gzip header. Unless the kit has been modified, the server will answer with 304 Not Modified and no content. The copy of the browser will be valid for another 5 minutes. This means that after 5 minutes, at least 3 round-trips are needed to check if our fonts are up-to-date.

Worse, sometimes, Typekit servers may be just slow. They can wait one second before answering, just to send a 304. Since the first file is a script embedded in <head>, the whole page is blocked until you get the answer. Here is an illustrative example with Chromium 14 where a page using Typekit is loaded while all static assets are in the navigator cache and things have gone wrong (which is pretty rare):

Since fonts are needed, the rendering has been paused for 3.5 seconds until Typekit was fully loaded. If we look more closely, we may also notice we got a status 200 from the server instead of 304. Here is an excerpt of the browser request:

GET /tyt0atd.js HTTP/1.1
Host: use.typekit.com
Connection: keep-alive
Accept-Encoding: gzip,deflate,sdch
If-None-Match: "2200251225+gzip"
If-Modified-Since: Thu, 10 Nov 2011 11:38:11 GMT

And here is an excerpt of the answer headers:

HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=300
Content-Type: text/javascript
Date: Fri, 11 Nov 2011 17:10:03 GMT
ETag: "2200251225"
Expires: Fri, 11 Nov 2011 17:15:03 GMT
Last-Modified: Thu, 10 Nov 2011 11:38:11 GMT
Server: EOS (lax001/54E5)
Content-Length: 25755

What happened? For some unknown reason, the server denied the access to the compressed version of the resource: there is no Content-Encoding: gzip while the browser did send Accept-Encoding: gzip. The tag provided by the browser (2200251225+gzip) came obviously from the compressed resource and therefore does not match. RFC 2616 requests If-Modified-Since header to be ignored if the tag does not match. Consequently, the server answers with 200 OK and resend the whole file. The same problem happened with the CSS file. Let me stress again how important the problem is: we had to download again the two files but we also downloaded the non-compressed version.

Now, let’s suppose that we did not run into this problem and the server answered with 304 Not Modified. The result is still not satisfactory: we would have waited 1.3 seconds to get both answers from the server. Keep in mind that most of the time, Typekit is pretty fast.

Here is how I think this should have been done: each time you modify your font kit, you get a new URL to use (containing the version of the font kit for example). This resource and the associated fonts (whose URL is also versioned) are requested to be cached for several days. As a sidenote, this also enables to modify a kit for your development environment without modifying the kit for your production site.

I suspect that Typekit did not go this way to track kit usage more accurately and enforce the appropriate pricing. Having multiple versions of the same kit would also allow us to use more fonts that we are paying for.

UPDATED: Typekit support knows the problem and is working on it:

You’ve identified a small bug in our CDN provider that we’re working with them to fix. Hopefully I can give an explanation of what’s happening.

The CDN edge servers have a cache of recently served kits on them. Any request that is served from that cache will be correctly Gzipped and handle ETags. Unfortunately, if the content isn’t already in cache and needs to be fetched from the master origin server then the request won’t be gzipped correctly which has the side effect of breaking ETags.

[…]

We are working to on getting this bug fixed, but any change to a globally distributed CDN will take some time. We’re also working on having our content cached longer than 5 minutes whilst still allowing people to quickly update their site.

UPDATED: To avoid to block page loading if Typekit is slow, you can load it asynchronously with yepnope.js. You’ll get some FOUT instead.

Conclusion

I have mixed feelings about Typekit. While the difficulty to test fonts and the lack of language-related subsets are annoying, I have been able to work around them. However, the caching problem is more annoying. I am still considering going back to Google Web Fonts just because of this.