DNS-aware Persistent Connections

What exactly is the problem statement?

So, let me try to present a simplistic version of the problem. Let’s assume there is a DNS configuration which points to a set of resources for a domain or a subdomain. The configured domain is used by an application to establish persistent http connections to the resources. The question is,

Is that even a valid problem?

Before I answer valid or not valid, let’s see if this question has been raised elsewhere.

  1. square/okhttp: Again similar issue with some discussions. https://github.com/square/okhttp/issues/3374
  2. akka/akka-http: https://github.com/akka/akka-http/issues/1226
  3. pgbouncer/pgbouncer:A PostgreSQL connection pooler has similar issue reported. https://github.com/pgbouncer/pgbouncer/issues/383
  4. cURL: A good discussion in cURL’s mailing list. https://curl.haxx.se/mail/lib-2017-06/0021.html
  1. There is no clear solution that is being suggested or implemented anywhere.

Let’s demo the issue

A rudimentary golang script to create persistent connections to api.razorpay.com

package mainimport (
"io"
"io/ioutil"
"log"
"net"
"net/http"
"time"
)
func main() {
tr := &http.Transport{
DialContext: (&net.Dialer{
Timeout: 5 * time.Second,
KeepAlive: 30 * time.Second,
DualStack: true,
}).DialContext,
TLSHandshakeTimeout: 5 * time.Second,
IdleConnTimeout: 100 * time.Second,
}
go func(rt http.RoundTripper) {
for {
time.Sleep(1 * time.Second)
c := http.Client{Transport:tr}
resp, err := c.Get("http://api.razorpay.com")
if err != nil {
log.Printf("Failed to do request: %v", err)
continue
}

io.Copy(ioutil.Discard, resp.Body)
resp.Body.Close()
log.Printf("resp status: %v", resp.Status)
}
}(tr)
ch := make(chan struct{})
<-ch
}
2020/08/02 01:17:45 resp status: 200 OK
2020/08/02 01:17:46 resp status: 200 OK
2020/08/02 01:17:48 resp status: 200 OK
2020/08/02 01:17:50 resp status: 200 OK

The solution

The solution to this problem, in my opinion, is simple.

Why is this a responsibilty of the server to terminate connections?

This is because:

  1. The way http connection works is by resolving the DNS IPs before establishing connections to those IPs. DNS never comes into picture once a connection has been established and is kept alive.

Where do load balancers fit into the picture here?

The problem becomes more acute when we have load balancers or reverse proxies in between. A resource server wanting to get out of rotation can terminate existing http connections to it but the load balancer or the reverse proxy would return a 502 Bad Gateway to the client in such a scenario. Although new requests would not land up in this server if its marked unhealthy and the healthcheck removes it from the load balancers set of backend targets, but would the existing connections be pro-actively terminated too in this case? I dont think all load balancers or reverse proxies do that.

What about http2?

With multiplexing in http2, the problem becomes more prominent and this brings a different perspective to the problem. Let’s talk a little about L7 http2 load balancers, envoy for example. By the way, there is a good blog post by envoy on modern load balancers.

Strict DNS based service discovery in envoy

Envoy supports two modes of DNS based service discovery, logical DNS and strict DNS. With both these modes, envoy always does async DNS resolution for zero DNS blocking in the forwarding path, which is indeed a good optimisation. But the difference between strict and logical DNS is that, in strict DNS, envoy drains connections to any host which is not returned in the DNS query, while draining of existing connections does not happen in case of logical DNS.

The 421 (Misdirected Request) status code in http2

A slightly related aspect is 421 (Misdirected Request) status code in http2. Section 9.1.2 of RFC 7540 describes 421 (Misdirected Request) as

But I would still want to handle this at my client.

There is a partial solution available in some languages/frameworks which can help mitigate this if you still need to handle it at the client. The ESTABLISHED connections can be limited by time (as done by akka-http for example) or they can be limited by number of requests (as done by envoy for example)

Summary

  1. DNS aware persistent connections is a common topic raised at multiple places.
  2. There is no standard client-side implementation to handle this.
  3. Envoy’s strict DNS pattern falls closer to a DNS-aware client for persistent connections, but it’s not advisable to be used.
  4. The only available client-side solutions are to use time bound or number of requests bound persistent connections as available in some languages or frameworks.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store