Tech on Bradley Falzon

A fork of /x/net/http2 providing Server Push for Go

Mon, 02 May 2016 08:54:54 +0930

Overview

Since version 1.6, Go has transparently supported HTTP/2 for both clients and servers when using TLS 1.2. The support was made available via the golang.org/x/net/http2 library for Go 1.5.

What’s being documented here is a fork of golang.org/x/net/http2 (see below) which adds preliminary HTTP/2 Server Push support (for Go applications), tested in Google Chrome 49, 50, 52 and Firefox 46.

See also CloudFlare’s recent announcement for more information on HTTP/2 Server Push, as well as the standard’s information page: http://httpwg.org/specs/rfc7540.html#PushResources

A simple demonstration is available here: https://bradleyf.id.au:8443 and the source code is available here: https://github.com/bradleyfalzon/h2push-demo (fork is available: https://github.com/bradleyfalzon/net)

If you’re on Twitter and interested in IETF Internet-Drafts and Protocols, check out https://twitter.com/rfcbot

Usage

Clients are required to get the fork using go get github.com/bradleyfalzon/net/http2 and import accordingly import "github.com/bradleyfalzon/net/http2".

An appropriate HTTP/2 server can be created by using the following example:

s := &http.Server{
    Addr:           ":3003",
    Handler:        nil,
    ReadTimeout:    30 * time.Second,
    WriteTimeout:   30 * time.Second,
    MaxHeaderBytes: 1 << 20,
}
http2.ConfigureServer(s, nil)

log.Fatal(s.ListenAndServeTLS("cert.pem", "key.pem"))

Within a http handler, to push a resource simply add the relative path of the resource using the Link header, for example:

w.Header().Add("Link", "</static/main.css>; rel=preload;")
w.Header().Add("Link", "</static/main.js>; rel=preload;")

Use Google Chrome 52 (currently a Canary release) to better view pushed resources (although earlier releases do support pushed resources, they just do not obviously indicate it). See the CloudFlare announcement for better information on viewing pushed resources.

Clients can push multiple resources
Resources are fetched by using the appropriate http handler, therefore static assets as well as dynamic assets can be pushed
In this implementation, a handler can detect if a resource was pushed by checking for presence of the h2push header. Note, this is not a trusted header, see discussion in implementation issues.

if _, ok := r.Header["H2push"]; ok {
    // Most likely a pushed request (but a client could have forged this header)
}

Implementation Details

You can quickly view the entire diff on GitHub.

HTTP/2 supports concurrent requests on a single TCP connection by multiplexing each request and response on their own stream.

HTTP/2 has different request and response types called frames. For example, a HEADERS frame contains either the request or response headers, a DATA frame contains the body for either request or response, and there are other frames that are unique to HTTP/2 such as SETTINGS, WINDOW_UPDATE and PUSH_PROMISE.

The HTTP/2 spec provides the implementation requirements and is available http://httpwg.org/specs/rfc7540.html, see the section on Server Push for specifics.

The first step is to detect which assets need to be pushed, this implementation detects resources to be pushed by the presence of a Link header when processing the response. This is a simple API, but this is (in this implementation at least) only detected after the initial response has been generated by the application. A server or proxy may immediately based on the request whether another resource should be pushed (such as the value or lack of a cookie, or if the resource is not cacheable), proxies may also wish to push resources before the initial response is ready. So future server implementations may chose other methods (discussed in closing notes).

Note, this implementation’s use of the Link header is semi-complete implementation of CloudFlare’s Server Push behaviour, whereby the application signals a resource to be pushed by setting a header containing the full path of the resource. A future implementation should either correctly follow the full Link draft (https://www.w3.org/Protocols/9707-link-header.html) or implement another API, with support for sending promises immediately without waiting for the request to finish processing.

Once the server knows of a resource to push, it must first send a PUSH_PROMISE frame to the client, before sending the response’s HEADER or DATA frame. See http://httpwg.org/specs/rfc7540.html#PushRequests for why this is the case.

There are rules on what type of resources can be pushed, and importantly, clients can disable server push; promises may be suitable for a web browser, but not for server to server communication, user-agents such as wget and curl, or potentially resource (bandwidth, CPU or battery) constrained devices.

These checks and detections can be seen: https://github.com/bradleyfalzon/net/commit/e5fbdb8434a6c8ca5b358cee38d2acb0070d8fb1#diff-51f54e5e768ac5a5b2539aebaf738475R1972

Once a resource has been chosen to be pushed, the PUSH_PROMISE frame needs to be created and sent on the existing stream. The frame contains a new stream ID which the resources will be sent on, as well as a mock of the request headers a client would have sent.

These mock headers gives the client an opportunity to reject the stream (by sending a RST_STREAM), potentially because the client already has the resource in its cache. Obviously some of the pushed resource may already have been sent, so it’s important to send the promise as early as possible, and only if it’s very likely the client doesn’t have the resource already, and the cost of addition network bandwidth (for both the client and server) doesn’t outweigh the performance gain.

Go’s /x/net/http2 didn’t originally contain the ability to create a stream, but by seeing how the existing processHeaders method creates a new stream from new requests, it’s possible to implement a limited (and incomplete) newStream method which sets up the internal state to support a new stream generated server-side. Future implementations will likely be able to remove some of the repetitive code.

Note, streams created by clients must be odd numbered (all streams start at stream ID 1, ID 0 is reserved for control frames), and servers create even numbered streams. New stream IDs must be greater than the last maximum. So a client creating a new stream with the ID 2^31-1 is probably going to have a bad time very quickly.

The newly created method responsible for building required frames, writePromise, modifies the http2.serverConn state - and it likely violates responsibility. See the TODO above the function definition for more information. This will likely cause issues but needs further investigation.

To send the PUSH_PROMISE frame a new writer struct was created in write.go which simply sends the frame.

This implementation’s generation of request headers only includes a minimal set of headers for the request to succeed (method, scheme, authority and path), it does not include Cookies, Accept etc. This is simply an implementation issue and will need to be fixed in production implementations.

Once the PUSH_PROMISE frame has been generated, and new set of request headers is generated (more repetition here) that will be sent to the http handler responsible for writing to the usual http.ResponseWriter - which in turn causes the response HEADER and DATA frames to be sent on the newly created stream ID.

The implementation currently available was designed to contain the minimum number of changes without refactoring the entire package - and production implementation will unlikely make this trade off as the developer will likely also have given themselves more than a weekend and already be familiar with the code.

To recap, the steps required are:

Detect assets to push
Ensure client has not disabled server push, and the assets can be sent
Create a new stream to send the response on
Create a fake set of request headers to send in PUSH_PROMISE
Send a PUSH_PROMISE frame on the existing stream and before the DATA of the initial request, this frame contains:
- new stream ID which will be used to send the headers and data of the resource
- faked request headers allowing the client to reject the resource (some data may already have been sent)
Call the appropriate http handler to write the response to the http.ResponseWriter
Sends response’s HEADER and DATA frames on the promised stream ID

Implementation Issues

This is not a complete nor correct implementation of HTTP/2 Server Push, nor is it suitable as a candidate for production implementation. Here be dragons.
A better implementation will likely focus on refactoring /x/net/http2 as well as possibly using the serve loop to listen for promises to be created (to safely mutate the http2.serverConn struct when creating streams).
This implementation does not handle fragmented headers, this is unlikely a problem in toy servers, but a requirement for a production implementation, however, it should currently send headers up to approximately 16K in size correctly.
This likely fails existing unit tests and does not include new nor does it update existing tests.
Doesn’t send promises for HEAD requests, this is by design as no testing has been done, but the HTTP/2 specification does not forbid it.
It currently adds a “h2push: true” header to pushed requests, this is only to enable the demo to differentiate pushed requests from standard requests, a production implementation would need to reconsider this approach as the headers are obviously easily forged by clients. The http.Request struct could be modified to include a Promised (or similar) bool field.
Only minimal http request headers are sent to the relevant http handlers, headers such as Cookie, User-Agent, Accept* etc are not current sent.
This does not check if any additional streams exceed the negotiated maximums.
This fork will not likely be updated from upstream nor maintained to a satisfactory level.
There maybe an issue with requesting the same pushed resource multiple times on the same TCP connection, more testing required.

Closing Notes

I would love to explore the possibility of implementing a cache based on HTTP/2 Server Push, either in a dedicated reverse proxy (similar to mod_pagespeed) or via a HTTP/2 Server such as Caddy via a plugin.

More investigation in exactly where Server Push is most beneficial needs to occur. HTTP/2 already provides great benefits via its multiplexing, perhaps pushed resources should only focus on pushing resources with the aim to reduce first paint events (style sheets) and to push critical scripts required for rendering.

Applications may need to ensure they don’t push resources in response to requests generated via XMLHttpRequest or similar methods.

The Link header method may not be the absolute best API for all applications, as mentioned already, it’s important for some promises to be sent very early, and using response header methods would require them to be processed before the request is finished processing or flushed. Response headers may provide a good method for intermediate proxies (which explains CloudFlare’s support).

Detecting when a resource is being push may also have it’s benefits (it’s not clear whether or how CloudFlare provides this).

Thanks

I would like to thank the Go authors for their initial implementation (you know who you are) and I look forward to a production implementation in later Go versions.

Thanks for SPDY team and those involved in the IETF WG for their work on HTTP/2 and supporting Server Push.

A big thanks to CloudFlare for giving me the idea for this weekend project, your open source contributions and staff are always inspirational.

Linux backups using Google's Nearline Storage

Sun, 22 Mar 2015 10:01:28 +1030

Overview

Sign up to Google Cloud Platform
Install gsutil command
Use gsutil to create our API credentials
Create our Nearline Bucket
Do an initial sync using gzutil’s rsync method

You should be redirected to console.developers.google.com, your first step is to create a project. I’ve called mine Linux Backup.

Don’t create any storage buckets or additional credentials yet, we’ll create them via gsutil.

Install gsutil

My instructions install gsutil to /usr/local/gsutil, change this path to any path you prefer.

# wget https://storage.googleapis.com/pub/gsutil.tar.gz
# tar xzf gsutil.tar.gz -C /usr/local/
# echo 'PATH=$PATH:/usr/local/gsutil' > /etc/profile.d/gsutil.sh

Configure gsutil

Running the gsutil config will provide a URL, open this URL in a browser and login using the Google account you’d like to use for storage and billing. You’ll be asked to authorise the request and once completed the website will give you a token to copy and paste back into gsutil. It’s straight forward.

# gsutil config
This command will create a boto config file at /root/.boto containing
your credentials, based on your responses to the following questions.
Please navigate your browser to the following URL:
https://accounts.google.com/o/oauth2/<snip>
In your browser you should see a page that requests you to authorize access to Google Cloud Platform APIs and Services
on your behalf. After you approve, an authorization code will be displayed.

Enter the authorization code: <some key>

Please navigate your browser to https://cloud.google.com/console#/project,
then find the project you will use, and copy the Project ID string from the
second column. Older projects do not have Project ID strings. For such projects,
click the project and then copy the Project Number listed under that project.

What is your project-id? symmetric-index-<some id>

Boto config file "/root/.boto" created. If you need to use a proxy to
access the Internet please see the instructions in that file.

Create Backups

Create a bucket, see gsutil help mb to see a complete list of options, such as specifying bucket region.

# gsutil mb -c nearline gs://<bucket_name>

Now, perform your initial rsync

# gsutil -m rsync -r /<directory> gs://<bucket_name>

The -m option runs a parallel rsync

For future backups, use the -q option to hide all output but errors, this is useful for cron, so it will only email if an error occurs.

# gsutil -qm rsync -r /<directory> gs://<bucket_name>
``

Faster CRC32 Checksums
======================

Note, by default it's likely rsync will use a slow method to calcualte CRC32 checksums. For a faster method it's
recommended to

$ gsutil ver -l | grep crcmod ```

If the output shows compiled crcmod: False, then install the compiled module by following the instructions in gsutil help crc32c - which essentially uses pip to install crcmod32.

Git, Push to Deploy & Hugo

Sun, 11 Jan 2015 14:26:45 +1030

Overview

Push to deploy is a mechanism to automate building and deploying code within a version control system (VCS) to a staging or production server. In this case, we’ll be using Git for our VCS, Git’s post-receive hooks for automating the builds and Hugo to build the blog itself.

This assumes you want two remotes for Git, one for your VCS purposes and the other a production server for deployments. You can expand the logic here to include a staging server or a more elaborate deployment to multiple servers. Of course, there’s other tools more suited to proper deployment processes.

Alternatively this design can be adjusted to support pushing to a continuous integration (CI) server that can be tasked to build environments, run tests deploy code to multiple servers.

Although this processes contains references to Hugo, this process can be used to deploy any code, however, as this is only designed to deploy a simple blog on the master branch, your mileage may vary.

Steps

Start with an existing repository with a remote called origin pointing to your favourite VCS (eg personal server, GitLab, GitHub / BitBucket etc).
On your production server, create a repo that will be used for hosting your live content.
Add post receive hook to the live remote which checks out master, runs Hugo and cuts over blog.
Add the new live remote to existing repo.
Push to live remote.

Starting Point

For this example, we’re starting with a straight forward git repo with only one remote.

$ git remote -v
origin  git@bitbucket.org:bradleyfalzon/bradleyf-blog.git (fetch)
origin  git@bitbucket.org:bradleyfalzon/bradleyf-blog.git (push)
$ git branch
* master

You can see I store my blog content on Atlassian’s BitBucket service. I’ve checked it out locally and I’m on the master branch. From now on, I’ll assume you’re not looking at tracking remote branches, but this process would only require minor modifications in that case.

Configure Production Remote

Here we’ll need to:

First we’ll too create a new bare repo on the server. An alternative technique is available in git-scm book.

$ cd /data/git
$ mkdir bradleyf-blog.git
$ cd bradleyf-blog.git
$ git init --bare
Initialized empty Git repository in /data/git/bradleyf-blog.git/

This has created an empty git repo with no working tree (thanks to --bare). A simple git clone would’ve create a working tree and checked out the master branch, which would stop a client from pushing to it whilst that branch is checked out.

Configure Post Receive Hooks

With our new (empty) git repo on the server, we need to configure our post-receive hook that will be executed after a push is successful. Whilst this hook is running, it will block the client from disconnecting, so don’t run anything too slow or background slow running scripts.

The post receive hook is stored in hooks/post-receive, and must be executable. Use hashbangs (#!) to specify which interpreter to execute for your script, in this case it’s BASH.

We’re using post-receive hooks, alternatively you could use a pre-receive hook which will give you the ability to exit with a non-zero status to indicate a failure and reject the push. This could be used to execute simple tests before accepting a push from a client.

Our post-receive hook looks like:

#!/bin/bash

# SYMDIR will be a symlink pointing to the current version
# of the content. In my case nginx has this directories public
# dir as the document_root.
SYMDIR=/data/www/bradleyf.id.au

# When there's failures, send emails to this address
EMAIL=user@example.com

# Store all logs here, being overriden each deploy
LOG=/tmp/blog-deploy.log

# Tell check that this other directory is the working tree, so
# checkout the content to this directory
GIT_WORK_TREE=$SYMDIR-`date +"%s"`

export GIT_WORK_TREE

# Simple checkErrors function, the first argument is a string to write to log
# if something happens.
function checkErrors() {
        if [ "$?" != "0" ]; then
                echo $1 >> $LOG
                cat $LOG | mail -s "Git deploy problems" $EMAIL
                exit 1
        fi
}

date > $LOG

# Create our working tree
rm -rf $GIT_WORK_TREE &> /dev/null
mkdir $GIT_WORK_TREE &>> $LOG
checkErrors "Could not mkdir $GIT_WORK_TREE"

# Checkout master to the working tree directory
git checkout -f master &>> $LOG
checkErrors "Could not checkout master to $GIT_WORK_TREE"

cd $GIT_WORK_TREE &>> $LOG
checkErrors "Could not change directory to $GIT_WORK_TREE"

# Run hugo to build the content and store in public/
hugo &>> $LOG
checkErrors "Could not use hugo to build blog"

# Atomically cut over the old working tree to the new, note
# using ln may not be the best method and mv should be considered.
ln -sfn $GIT_WORK_TREE $SYMDIR &>> $LOG
checkErrors "Could not create sym link from $GIT_WORK_TREE to $SYMDIR"

# Remove old versions (to revert use git-revert)
find $SYMDIR-* -maxdepth 0 -type d | grep -v $GIT_WORK_TREE | xargs rm -rf

Note, this is the first revision of this script your environment may require different options or flows depending on your use case.

Configure Clients

Once the server’s been configured, all clients will need to add this server as a remote. In my case, I’ve left BitBucket as my origin and added the production server, bradleyf.id.au, as a remote called live.

$ git remote add live root@bradleyf.id.au:/data/git/bradleyf-blog.git

Push to Production

Now a quick git push live would push my content to the production server and git’s post-receive hook will build the content and deploy for me (or email me if there’s a problem).

$ git push live

I wanted to manually push to live so I could control when I’m pushing as to whether it’s going into my VCS/origin (default) or live production server.

You must remember to push to both remotes when you’re ready, one for VCS, one for live. To make this two step process one, I manually created a third remote called all, so I could git push all which would push to the origin remote and live remote in one command.

$ tail -n 3 .git/config
[remote "all"]
    url = git@bitbucket.org:bradleyfalzon/bradleyf-blog.git
    url = root@bradleyf.id.au:/data/git/bradleyf-blog.git

In total, I have three options when I push:

$ git push
    - Push to origin remote only
$ git push live
    - Push to live production server only
$ git push all
    - Push to both BitBucket and production server

Additional Tips

Show the committed differences between current master and the remote live.

git diff master..live/master

Shaving your RTT with TCP Fast Open

Sat, 03 Jan 2015 12:13:48 +1030

TL;DR

TCP Fast Open (TFO) allows clients to send data in the initial SYN request, without waiting for a full handshake to occur. This removes an entire round trip almost transparently from the application.
Cookies are included in the TCP options header.
It’s available in Linux 3.7+, nginx and HAProxy has support already. Client support is lacking with only Chrome/Chromium on Linux, ChromeOS and Android 5.0 (Lollipop), and only if enabled manually.
There’s some potential issues with delivering duplicate data to the server’s application, which wouldn’t occur in normal TCP, although it is unlikely, some applications may not be compatible.
- It’s perfect for static content (CDNs),
- Every website which uses TLS (it’ll reduce that handshake time).

Overview

Standard TCP requires the client and server to establish a three way handshake (3WHS) before data can be delivered to the server’s listening application. This introduces latency of one round trip before the server’s application receives the data, and a total of two round trips by the time the server can respond. This can be significant for latency sensitive applications such as HTTP.

TCP Fast Open (TFO), defined by RFC 7413, still requires an initial 3WHS to be established before data can be sent for the first time. During this handshakes first SYN packet, the client can request a TFO cookie, and a compatible server responds with a cookie in its SYN-ACK response.

Once the client receives the SYN-ACK response it caches the cookie, responds with the application data (such as HTTP GET, or if it’s TLS, ClientHello) as per normal. At least two round trips were required, one for the connection setup (initial SYN and SYN-ACK) and the other for the application request and response.

But now the client has a TFO cookie, a new connection’s SYN packet sent to the same server now also includes the application data. The server validates the cookie and responds to the client with a SYN-ACK whilst also immediately sending the data to the server’s listening application for a response. Reducing the total round trips to one.

Support

Servers

Linux 3.7+ e.g. Red Hat Enterprise Linux (RHEL/CentOS) 7, Ubuntu 14.04 LTS
nginx 1.5.8
- nginx provided RPMs do not have support, other packages may have support
HAProxy 1.5

Clients

Chrome/Chromium on Linux, Chrome OS or Android (not sure if Chrome flags has it disabled, and if not, whether users need Linux 3.13 to have it enabled by default)
Linux 3.6+ e.g. Red Hat Enterprise Linux (RHEL/CentOS) 7, Fedora 18, Amazon Linux AMI 2014.03, Ubuntu 13.04
Chrome OS
Android Lollipop (5.0)

Websites

Google’s assets such as Google Search, YouTube, Blogger, DoubleClick etc

Detailed Process

Sending TFO Request

During the initial TCP connection, a client requests the kernel create a TCP connection with the TFO options enabled. In Linux, a client normally uses the connect() and write() system calls to create and send on a TCP connection. But to use TFO, these calls need to be combined - to allow application data to also be sent during the initial connection phase. Therefore, not only does the TCP stack the application is running on need to support TFO, but clients must also be written to support this.

API changes required to the application are more fully described in LWN’s article TCP Fast Open: expediting web services.

The SYN packet generated by the clients TCP stack will set the TCP Option 34 which corresponds to TCP Fast Open Cookie. Because no TFO connections have been established, the cookie is empty - indicating to the server the client supports TFO and would like a cookie. There is also no data within this SYN packet.

A SYN packet’s maximum TCP options length is 40 bytes, a Linux 3.18 kernel currently uses 20 (MSS 4 bytes, TCP SACK 2 bytes, Timestamps 10 bytes, NOP 1 byte and Window Scale 3 bytes). Adding TFO cookie increases this to 32. The TFO RFC states if the SYN packet does not have enough space to fit the TFO options, disable TFO for this connection. Note, it doesn’t state this for SYN-ACK replies, but one would assume this would also be true.

Receiving a TFO Request

During transit, some middleware may not understand this TCP option and instead drop the packet. In which case, the client should retransmit the SYN packet without a TFO option - i.e. attempt a standard TCP connection instead. The client should also cache the negative response, so future connections will not need to wait for a timeout to this destination, and instead try a normal TCP connection the first time.

The server application, much like the clients, must also be written to support TFO. In the server’s case, it must set the TFO_FASTOPEN socket option via the setsockopt() system call and provide a maximum queue length (discussed later in DoS Amplification Attacks).

An example client and server, using system calls in Go can be found github.com/bradleyfalzon/tcp-fast-open

When the server’s TCP stack receives the SYN packet, if it does not support TFO it will ignore the TFO option and establish a normal TCP connection. If the server supports TFO, and the listening socket has requested TFO support, it will generate a message authentication code (MAC) of the client’s IP address (Linux also includes the server’s IP address) using the server’s secret key.

If the received SYN packet does not contain a cookie, but contains TFO options, the generated cookie is sent to the client in the server’s SYN-ACK packet, using the same TFO option space and a normal TCP connection is established without any round trip savings.

Receiving a TFO Request Response

When the client receives the SYN-ACK packet with the TFO cookie set, it caches this cookie, the server’s IP address, and the server’s advertised maximum segment size (MSS). This cookie can now be used by the client when it tries to connect to the same IP address with client TFO support enabled (note, the cookie is bound to the server’s IP and doesn’t include the server’s destination port number).

Sending TFO+Data

When a client tries to connect to the same server again with TFO socket options set, the initial TCP SYN packet will be generated with the previously cached TFO cookie and application data up to the MSS size. If the application data exceeds the MSS size, additional data must wait to be sent until after the 3WHS is established - negating the use of TFO.

The default MSS for IPv4 is only 536 bytes (IPv6 is 1220 bytes), where as the typical MSS is closer to 1460 bytes, which is why it’s important to cache the MSS - to try and fit all the data in the initial request. So, keep request sizes within ~1460 bytes by reducing HTTP Cookie sizes.

Receiving TFO+Data

On the server’s side, the received SYN packet is checked for TFO cookies, and if so, the server generates a cookie. The generated cookie is compared to the cookie in the SYN packet. If the comparison is successful, the server replies with a SYN-ACK of the SYN + data and sends the data to the listening application. If the comparison fails, the server replies with a SYN-ACK only acknowledging the SYN (not data) and includes the generated TFO cookie for the client to use next time.

Receiving a TFO+Data Response

When the client receives the SYN-ACK packet, it checks if the data was acknowledged, if it was it ACKs the SYN-ACK and waits for further responses (if applicable) from the server.

If the data was not acknowledged in the received SYN-ACK packet, the client still ACKs the server’s SYN-ACK, but also sends another packet with the application data (as it would a normal TCP connection). If the received SYN-ACK packet contains a TFO cookie the client will cache it for use next time.

This system provides the following properties:

Connections benefit from TFO only after an initial TCP connection is established
Client application and kernel support is required, as well as server application and server kernel support
Clients can handle most bad middleware by retransmitting with TFO
Clients that suppport TFO that connect to servers that do not is handled gracefully by simply ignoring the TFO cookie request

Potential Issues

Applications that receive the initial SYN data must be tolerant of duplication, in many cases the application is TLS which is tolerant of duplicate data. Other applications that are not technically tolerant may simply accept the risk if the impact is low (e.g. the risk of a forum website double posting may be acceptable, but purchasing all items in the cart twice might not be).
If a client needs to send more than the cached MSS, TFO is unavailable. For example, my PayPal cookies are currently 3298 bytes, well above the 1460 MSS currently advertised.

Duplicate Data & Idempotency

Applications with TFO enabled sockets must be able to handle receiving duplicate data due to retransmitted SYN packets. This is only an issue for the data within the initial SYN packet. Consecutive packets sent after the initial TCP handshake (and non TFO TCP connections) detect duplicate data and drop the extra packets without delivery to the application.

Duplicate data is possible by at least two methods outlined by the RFC, and potentially and third discussed later.

The first condition, where a server “forgets” it received the initial SYN packet (e.g. due to a server reboot), has been discussed and two work arounds proposed:

Delay enabling TFO by a few minutes
Regenerate a new server key upon reboot, either randomly or based on a boot ID.

Solution 1 is a reasonable solution, with the obvious drawback of not having TFO enabled immediately and potentially(? - need to check) causing a client to clear its TFO cookie cache. Delayed start could be managed by user land tools, by simply disabling TFO by default and creating a service to enable TFO 5 minutes after server start (reboot).

Solution 2 is already implemented in the Linux as the server key is randomly generated each reboot (earlier versions before Linux 3.13 generated a new TFO key on boot, newer versions only once a socket sets the relevant TFO socket option). However, the use of TFO in some load balanced topologies, such as Direct Server Return (DSR), requires the servers to share the same TFO key, thereby allowing a client to reuse the same cached cookie on any server in the farm. By randomly generating the server key each reboot, all servers will need to change their key to the new key, which could reduce the effectiveness by needlessly increasing the amount of key rotation - which could be significant on larger farms.

The second and following conditions, have no obvious solutions provided by TFO itself, which means either TFO is incompatible with the application or, as for the third condition, a change must be made to the higher level architectures to support TFO.

A third condition which can cause duplicate data, that wasn’t obviously mentioned in the RFC, is a server farm where a retransmitted SYN packet arrives at a different server before the client receives acknowledgement (SYN-ACK) of the first packet. This scenario requires the SYN-ACK packet to be dropped or delayed and requires the load balanced topology to deliver the second SYN packet to a different server (either because no persistence is configured or because the server had been removed from the load balanced farm). Just like non TFO TCP connection, the server acknowledges the first SYN packet before it sends the application’s response, therefore, there’s no risk of slow application processing cause a retransmit.

A further note, one of the original developers of TFO states “today the client may send a non-idempotent request twice already with standard TCP.” - although it might be possible already, it’s not clear how much TFO increases this risk.

DoS & Amplification Attacks

There is a risk of amplification attacks, as described in more detail in the RFC. If an attacker can steal a valid cookie for the victim’s IP address, the attacker can generate a small request which may solicit a very large response sent directly to the victim.

The RFC further describes protections by requiring the server to automatically disable TFO (for that socket) in the event the TFO queue length exceeds an application defined max queue length. The max queue length defines the maximum number of outstanding unacknowledged SYN-ACK packets. Server applications are required to set this on listening sockets via the setsockopt system call. Exceeding the max queue length value will cause new connections to ignore TFO cookies and revert to a standard TCP handshake. The number of ignored TFO cookies can be monitored via the TCPFastOpenListenOverflow counter, it is not logged. See tcp_fastopen_queue_check in Linux 3.18.

Because the attacker’s requests are received by the application, standard DoS detection mechanisms can easily detect this, but in the case of TLS, the attacker can generate a small ClientHello request which will solicit a very large response from the server (containing the server’s certificate as well as intermediate certificates). Because the attack occurs at a lower level, within the TLS library, it’s less likely to be logged and may be unnoticed.

During a DoS attack the victim will receive SYN-ACK packets with TFO option bits set without seeing an earlier SYN request. A firewall or intrusion detection system (IDS) may be able to detect this type of attack. The victim and server would also see RST packets from the victim as the packets are rejected by the destination, providing additional clues of a possible TFO DoS attack.

An attacker that has reliable means of accessing the victims network and able to successfully obtaining valid cookies from a series of servers, maybe able to launch a larger or prolonged attack as each server will need to individually detect that it’s involved in an attack and individually disable TFO. A victims network may therefore decide to remove the TFO option from all outgoing SYN packets to prevent TFO on the network completely.

The max queue length counter is the total number of outstanding SYN-ACK packets, i.e. it is per socket not per client. Therefore when an attack is detected by the server, TFO is disabled for all clients (not just the victim). It may not be desirable for a single attacker to disable TFO for all customers, although this won’t cause a denial of service, it will disable the benefits of TFO and is trivially exploited.

Middleware Issues

Some middleware, such as firewalls and NAT boxes may cause issues with the new TCP option. Additionally, because the Linux continues to set the TFO option to 254, which is the experimental kind, it maybe more likely to be dropped.

It’s even been reported some middleware boxes, after detecting the TFO option in the initial SYN packet, drop subsequent SYN packets without the TFO option.

Also, if a device is behind a Carrier Grade NAT (CGN) with many public IP addresses constantly changing, a cookie may become invalidated often, reducing the effectiveness of TFO. High latency mobile devices which benefit the most from TFO are also most likely to be affected by changing public IP addresses due to CGNs. I currently have no data on this.

Minor Linux API Caveats

A strong benefit of TFO is the ability to reduce the requirement for long running, persistent, TCP connections (such as HTTP servers with keep alives). Persistent connections allows a client to send more data, at a later time, without needing to wait for a 3WHS - but this connection is not free to maintain, costing both clients and servers and is further described in the TFO RFC. Although TFO can immediately reduce the requirement for persistent connections (for connections established with TFO), Linux does not currently have an API for the application to determine whether the connection was negotiated with TFO. Persistent connections would only be enabled for non TFO connections - but because the application has no API to detect TFO connection, it cannot optionally disable persistent connections. Mobile clients may also wish the disable, or shorten, persistent connections and/or browser TCP preconnects when TFO connections are available.

Potentially the getsockopt system call could be capable of providing this information, as it provides other socket information such as whether the socket is listening, debugging etc.

TFO has been assigned TCP option 34 by IANA, however, the Linux (3.18 at the time of writing) currently uses the old experimental TCP option 254. This leads to an interesting thought, once the Linux supports the new TCP option number, will it continue to support the old experimental option number? Current Linux servers, such as Red Hat Enterprise Linux 7, and its derivatives, will continue to use set (in the case of a client) or check (in the case of a server) the older option number. Will the kernel developers simply stop checking the experimental number, legitimately breaking backwards compatibility - requiring a backport?

Linux API does not provide a mechanism for key rotation, once a new key is generated, cookies generated with the old key are immediately invalid. This benefit is briefly discussed in the RFC in Server Cookie Handling.

When a connection exceeds the max_qlen (the maximum number of unacknowledged TFO requests, used to reduce DoS attacks), log the event via pr_notice, printk or similar. Most people will probably never monitor the correct counters to detect this event, but will likely monitor kernel messages.

Enabling TFO in the Kernel

Linux supports configuring both overall client and server support via /proc/sys/net/ipv4/tcp_fastopen (net.ipv4.tcp_fastopen via sysctl). The options are a bit mask, where the first bit enables or disables client support (default on), 2nd bit sets server support (default off), 3rd bit sets whether data in SYN packet is permitted without TFO cookie option. Therefore a value of 1 TFO can only be enabled on outgoing connections (client only), value 2 allows TFO only on listening sockets (server only), and value 3 enables TFO for both client and server.

Note, even though these options maybe enabled, application level support must also be enabled.

To enabled TFO and be persistent across reboots, you can use sysctl like the following:

echo 'net.ipv4.tcp_fastopen=3' > /etc/sysctl.d/50-tcp_fastopen.conf
sysctl -p /etc/sysctl.d/50-tcp_fastopen.conf

Linux (from version 3.13) generates a key when an application sets the relevant setsockopt syscall options for the first time. Until a key is set, the proc value is all zeros, so don’t be alarmed.

By default, because the key does not persist between reboots, production use of TFO should include saving the key securely (generating random keys, setting restrictive file permissions) via sysctl. This will ensure clients can use the existing cookie without needing a new key generated.

To generate a new key and make persistent via sysctl:

RAND=$(openssl rand -hex 16)
NEWKEY=${RAND:0:8}-${RAND:8:8}-${RAND:16:8}-${RAND:24:8}
echo "net.ipv4.tcp_fastopen_key=$NEWKEY" > /etc/sysctl.d/50-tcp_fastopen_key.conf
chmod 600 /etc/sysctl.d/50-tcp_fastopen_key.conf; chown root /etc/sysctl.d/50-tcp_fastopen_key.conf
sysctl -p /etc/sysctl.d/50-tcp_fastopen_key.conf
unset RAND NEWKEY

Monitoring

To view client connection statistics for clients, ip tcp_metrics is available from iproute2 in version v3.7.0 (2012-12-11). This application can show you cached MSS as well as the TFO cookie used for a single IP or all IPs (exclude the show 127.0.0.1 option in that case).

$ ip tcp_metrics show 127.0.0.1
127.0.0.1 age 93935.839sec rtt 875us rttvar 500us cwnd 10 fo_mss 65495 fo_cookie cec297e8b2723c29

For both clients and servers the following counters are available in /proc/net/netstat, for an easy overview you can use the following (adjusting the column numbers if required):

grep '^TcpExt:' /proc/net/netstat | cut -d ' ' -f 87-92  | column -t

TCPFastOpenActive - number of successful outbound TFO connections.
TCPFastOpenActiveFail - number of SYN-ACK packets received that did not acknowledge data sent in the SYN packet and caused a retransmissions without SYN data. Note the original SYN packet contained a cookie + data, this is not number of connections to servers that didn’t support TFO.
TCPFastOpenPassive - number of successful inbound TFO connections.
TCPFastOpenPassiveFail - number of inbound SYN packets with TFO cookie that was invalid.
TCPFastOpenCookieReqd - number of inbound SYN packets requesting TFO with TFO set but no cookie
TCPFastOpenListenOverflow - number of inbound SYN packets that will have TFO disabled because the socket has exceeded the max queue length.

Additional options created that maybe useful: commit

TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
TCPOrigDataSent: number of outgoing packets with original data (excluding retransmission but including data-in-SYN). This counter is different from TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is more useful to track the TCP retransmission rate.

Rotating Keys

Linux provides /proc/sys/net/ipv4/tcp_fastopen_key (net.ipv4.tcp_fastopen_key via sysctl) which can be used to display the current key, as well as change the key to a new key. The key is 16 bytes, expressed as 32 character hex string, broken into 4 8 character blocks, separated by dashes.

Rotation of keys can be achieved in exactly the same way we generated keys in sysctl.

In a multi server server environment, you’ll want to randomly generate a key once, and set the same key on all servers.

Use Cases

Mobile Devices

TFO, being designed to reduce latency by removing an entire round trip, benefits mobile devices that often have a higher latency than other Internet connections provide.

Table data below, provided by the High Performance Browser Networking book, illustrates the typical latency for mobile networks.

Generation	Data Rate	Latency
2G	100–400 Kbps	300–1000 ms
3G	0.5–5 Mbps	100–500 ms
4G	1–50 Mbps	< 100 ms

As discussed in other sections, a theoretical benefit of TFO could be to reduce request and response time to half (if the response fits within TCP’s initial congestion window (initcwnd). A mobile device, with a latency of 500ms, would require two round trips using a standard, non-TFO, TCP connection. Assuming there is no server processing time, this connection would take 1000ms. With TFO, the first round trip is removed, so the connection time is now 500ms.

Static Site / CDN

Due to the possible, however unlikely, event of duplicate data, static content such as JavaScript, CSS, images, etc. have no idempotent requirements and would greatly benefit from reduced latency benefits of TFO.

Websites that are primarily static sites, or micro sites, have no idempotent requirements and therefore safe to use TFO.

For this type of content, which is also less than 14,600 bytes, the response may often fit within the server’s TCP initial congestion window (assuming a value of 10), the entire request and response time could be reduced by up to 50% (assuming the server has zero processing/fetching time).

Reducing TLS Latency

TLS introduces at least one additional round trip (for versions up to and including 1.2, TLS 1.3 addresses this), for new connections, two round trips are required.

Whether the TLS service supports session resumption via session tickets or session IDs, only the first TLS round trip penalty is required which contains the ClientHello request and server responses (which can be sent in multiple packets). New connections, or connections with expired/invalid tickets will have one more additional round trip.

With TLS resumption, only one round trip in total would be required to setup the connection (the same amount as a normal TCP connection without TFO and TLS). This works because the initial SYN packet will contain the ClientHello TLS data, as well as the session resumption ticket or ID. The second packet will contain the application data, which will drop retransmitted data - providing idempotence protection.

DNS Servers using TCP

DNS primarily uses UDP for transactions. It originally defined the maximum packet size of 512 bytes. A client would request a record with UDP, and if the response required a larger packet than 512 bytes, the response was truncated and a bit set instructing the client to retry with TCP. This would require a total of 3 round trips.

This was recognised and Extension mechanisms for DNS (EDNS) was released in 1999, and has since been superseded by RFC6891.

EDNS permits UDP responses to exceed the 512 limit, and allows responses to be sent over multiple UDP packets. Catering for protocols such as DNSSEC which may exceed this limit.

Because of EDNS, TCP usage by DNS resolvers is uncommon (0.61% reported by a medium ISP’s DNS resolver) for most transactions.

Regardless of when TCP maybe used, TFO can assist in reducing the round trips required for successful UDP downgrade to TCP and subsequent 3WHS. But the real world benefits will be minimal.

Reducing Tor Latency

Additionally Tor is a TCP protocol with TLS that can use TFO. However, I haven’t had a chance to verify if, during the initial Tor TLS negotiation, the client’s first request can fit within one packet (containing the clients certificates). If it cannot, then the remaining data must be transmitted after the 3WHS, nullifying the benefits of TFO.

Multipath TCP

Multipath TCP (MTCP) is an effort to support multiple TCP connections in the connection, taking advantage of multiple links for bandwidth and/or availability - all the while being transparent to the application.

A currently draft RFC, draft-barre-mptcp-tfo, seeks to address possible issues and inefficiencies with using MTCP and TFO concurrently. Among others, it provides suggestions for the likely exhausted TCP option space (maximum limit 40 bytes) as both TFO and MTCP requires 12 bytes each, totally 24 bytes, with recent Kernels already using an additional 20 bytes.

Other TCP Protocols

TFO is applicable to any TCP connection that is sensitive to latency, not just HTTP connections. Other protocols such as file servers (SAMBA, CIFS, NFS etc.) and potentially more could benefit from a reduce RTT.

Currently many browsers, such as Chrome/Chromium and Firefox, establish a TCP connection (TCP preconnect) to a server preemptively - before the user requests a resource. The idea is to establish at least one TCP connection before a user requires it, when the connection is required an already established connection can be used. Removing the TCP’s 3WHS for the request.

There are a few minor issues with this is. First, a TCP connection must be established and held open, some load balancers and/or web servers close an option connection that has not made a request after a short timeout, and records this as an error. This causes additional logging and potentially wasted effort investigating the errors. See HAProxy issues and Firefox issue

Another issue with this is that the TCP connection must be re-established constantly, whenever the browser thinks the user is about to connect to the site.

A separate mechanism could be employed using TFO. Whereby the browser preemptively establishes a TCP connection with TFO options allowing the TCP stack to obtain a cookie. Future connections would not require a preconnect if a TFO cookie was successfully obtained. Note however, the cookie may be invalid (server’s key has changed) or the source IP of the client may change, therefore the browser may choose to initiate a preconnect after some period of time.

Note, cookie prefetching would require an API for the application to detect whether a cookie was successfully obtained, to know whether it can close the connection immediately and to not establish a preconnect in the future. No API is currently available in Linux.

History

Feb 17th 2012
- TCP Fast Open initial draft released.
Sept 30th 2012
- Linux 3.6 adds TFO client support. Commits.
Dec 10th 2012
- Linux 3.7 adds TFO server support. Commits.
Jun 30th 2013
- Linux 3.10 stops clearing cached TFO cookies when clearing other TCP metric caches. Commit
Nov 3rd 2013
- Linux 3.12 encrypts server IP along with client IP when generating MAC. Commit.
Jan 19th 2014
- Linux 3.13 enables TFO client support by default by changing the tcp_fastopen sysctl value from 0 to 1. Commit.
- Linux 3.13 randomly generates TFO key only once a socket requests TFO, instead of each boot. Commit.
Aug 3rd 2014
- Linux 3.16 adds IPv6 TFO support. Commit.
Dec 18th 2014
- RFC 7413 is published

Building the Linux Kernel

Fri, 26 Dec 2014 08:46:02 +1030

How to build the Linux Kernel

The following instructions are based on CentOS 7, these should work for most other distributions, however, the initial dependencies will need to be reviewed.

Unless prefixed with sudo, all commands can be executed as a non-root user.

Why

There’s a few reasons why you’d want to build your own kernel, however, most of the point below aren’t significant.

Access recent enhancements not yet available in Linux distributions, in my case, I wanted to continue to run CentOS on my server, but build my kernel myself to access modern features (prebuilt unsupported Kernels are available).
Learning exercise - this was my secondary reason.
Creating a slimmer Kernel for performance (with many modules only being loaded when needed, I’m not sure on this)
Security’s sake (this requires you to be vigilant to always update the Kernel yourself)

Install Dependencies

In this case, we want tools like gcc (for compiling) and ncurses (for menuconfig):

$ sudo yum groupinstall "Development Tools"
$ sudo yum install ncurses-devel -y

Fetch the Kernel source

Go to kernel.org to fetch the kernel version that you’d like, most likely, you’ll just want the latest stable.

$ mkdir linux
$ cd linux/
$ wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.18.1.tar.xz
$ tar xfJ linux-3.18.1.tar.xz
$ cd linux-3.18.1/

Configure the Kernel

To preconfigure my kernel, I’m using the existing configuration options based on CentOS’s current running Kernel. This’ll help me compile everything I need, and just look for the specific options I want or don’t want.

The Kernel also comes with options to build a default config (make defconfig), but I don’t know exactly how this configuration is built. Some options are set in arch/x86/configs/x86_64_defconfig, but not all.

But for the moment, I’ll reuse CentOS’s but you could use the default by running make defconfig instead of the following:

$ cp /boot/config-`uname -r ` .config

If you’re using CentOS’s default from an older Kernel, you’ll want to upgrade the .config file to incorporate the new options. The command make olddefconfig will upgrade the .config file, setting the default values without prompting. Alternatively, you can run make oldconfig which prompts for answers to each configuration option. Only run this if you’re upgrading the .config file (not using a .config file build from make defconfig).

$ make olddefconfig

From here, I use menuconfig, a ncurses based interface to choose Kernel configuration options. There’s also GTK+ (make gconfig), QT (make xconfig), a purely text based option (make config).

$ make menuconfig

Compile the Kernel

This process takes a while, depending on options and CPU age and number of CPUs, less so disk IO speed. Optionally, and likely recommended, is to supply the -j [jobs] flag, this specifies the number of jobs running concurrently - you’ll want to set this number to the number of CPUs in your system.

In my case, because I have 2 CPUs, I would run:

$ make -j 2

Because I wanted to time the events, and run at a higher concurrency rate (long story short, it didn’t perform any better), I used the following command to also include timing information:

$ START_DATE=`date`; time make -j4; echo "Start date: $START_DATE"; echo -n "End date: "; date
real    350m30.902s
user    440m34.103s
sys     224m7.598s
Start date: Wed Dec 24 09:42:43 ACDT 2014
End date:   Wed Dec 24 15:33:14 ACDT 2014

Grab a tea, this’ll take a while if you’re running older hardware (in my case, a Virtual Box guest on a 2008 2.4GHz Core 2 duo).

Install the Kernel

This process is far quicker, but may still take a few minutes depending on your disks. Note, only now do we need to run as a root user.

$ sudo make modules_install
$ sudo make install

Finishing Touches

Your Kernel is not built and installed, it should have also updated GRUB configuration. But in some cases, you may need to run mkinitrd, but it appears make install did more work than I expected.

In my final case, I wanted this new Kernel to be my default, so I set it like so:

$ sudo grub2-set-default 0

Additional Reading

See Linux Kernel in a Nutshell.

Tech on Bradley Falzon

A fork of /x/net/http2 providing Server Push for Go

Overview

Usage

Implementation Details

Implementation Issues

Closing Notes

Thanks

Linux backups using Google's Nearline Storage

Overview

Sign up and initial configuration

Install gsutil

Configure gsutil

Create Backups

Git, Push to Deploy & Hugo

Overview

Steps

Starting Point

Configure Production Remote

Configure Post Receive Hooks

Configure Clients

Push to Production

Additional Tips

Shaving your RTT with TCP Fast Open

TL;DR

Overview

Support

Servers

Clients

Websites

Detailed Process

Sending TFO Request

Receiving a TFO Request

Receiving a TFO Request Response

Sending TFO+Data

Receiving TFO+Data

Receiving a TFO+Data Response

Potential Issues

Duplicate Data & Idempotency

DoS & Amplification Attacks

Middleware Issues

Minor Linux API Caveats

Enabling TFO in the Kernel

Monitoring

Rotating Keys

Use Cases

Mobile Devices

Static Site / CDN

Reducing TLS Latency

DNS Servers using TCP

Reducing Tor Latency

Multipath TCP

Other TCP Protocols

Cookie Prefetching

History

Building the Linux Kernel

How to build the Linux Kernel

Why

Install Dependencies

Fetch the Kernel source

Configure the Kernel

Compile the Kernel

Install the Kernel

Finishing Touches

Additional Reading