What happens if Keep-Alive is disabled on the client and enabled on the server?

This article is reprinted from the WeChat public account "Lean Coder", the author is an alias with attitude. Please contact the Lean Coder public account to reprint this article.

A web program was recently deployed, and a lot of time_wait TCP connection status appeared on the server, occupying the TCP port. It took several days to investigate.

I have concluded before: HTTP keep-alive is a sliding renewal and reuse of TCP connections at the application layer. If the client and server renew the connection stably, it becomes a true long connection.

Everything about [Http persistent connection]

Is HTTP1.1 Keep-Alive considered a long connection?

Currently, all HTTP network libraries (whether client or server) have HTTP Keep-Alive enabled by default, and negotiate multiplexing connections through the Connection header of Request/Response.

01 Short connections formed by unconventional behavior

I have a project on hand. Due to historical reasons, the client disabled Keep-Alive, and the server enabled Keep-Alive by default. As a result, the negotiation of multiplexing connection failed, and the client would use a new TCP connection for each request, which means it would fall back to a short connection.

The client forcibly disables Keep-Alive

 package main
 import (
 "fmt"
 "io/ioutil"
 "log"
 "net/http"
 "time"
 )

 func main ( ) {
 tr : = http .Transport {
 DisableKeepAlives : true ,
 }
 client : = & http .Client {
 Timeout : 10 * time .Second ,
 Transport : & tr ,
 }
 for {
 requestWithClose ( client )
 time .Sleep ( time .Second * 1 )
 }
 }

 func requestWithClose ( client * http .Client ) {
 resp , err : = client .Get ( "http://10.100.219.9:8081" )
 if err != nil {
 fmt .Printf ( "error occurred while fetching page, error: %s" , err .Error ( ) )
 return
 }
 defer resp .Body .Close ( )
 c , err : = ioutil .ReadAll ( resp .Body )
 if err != nil {
 log .Fatalf ( "Couldn't parse response body. %+v" , err )
 }

 fmt.Println ( string ( c ) )
 }

Keep-Alive is enabled by default on web servers

 package main

 import (
 "fmt"
 "log"
 "net/http"
 )

 // Know the persistent connection used by the client based on RemoteAddr
 func IndexHandler ( w http .ResponseWriter , r * http .Request ) {
 fmt .Println ( "receive a request from:" , r .RemoteAddr , r .Header )
 w .Write ( [ ] byte ( "ok" ) )
 }

 func main ( ) {
 fmt .Printf ( "Starting server at port 8081\n" )
 // net / http enables persistent connections by default
 if err : = http .ListenAndServe ( ":8081" , http .HandlerFunc ( IndexHandler ) ) ; err != nil {
 log .Fatal ( err )
 }
 }

Judging from the server log, it is indeed a short connection.

 receive a request from : 10.22 .34 .48 : 54722 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54724 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54726 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54728 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54731 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54733 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54734 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54738 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54740 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54741 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54743 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54744 map [ Accept - Encoding : [ gzip ] Connection : [ close ] User - Agent : [ Go - http - client / 1.1 ] ]
 receive a request from : 10.22 .34 .48 : 54746 map [ Accept - Encoding : [ gzip ] Connect

02Who is the active disconnector?

I took it for granted that the client was the one that actively disconnected, but reality slapped me in the face.

One day, the time_wait alarm on the server exceeded 300, telling me that the server actively terminated the connection.

Conventional TCP waves 4 times, the active disconnecting party will enter the time_wait state and release the occupied SOCKET after waiting for 2MSL

The following is the TCP connection information captured by tcpdump from the server.

2,3 Red box shows:

The server first initiates a TCP FIN message, and then the client responds with ACK to confirm receipt of the server's closing notification; the client then sends another FIN message to inform that it can now be closed. The server finally sends ACK to confirm receipt and enters the time_wait state, waiting for 2MSL to close the socket.

It is specially pointed out that the red box 1 indicates that both ends of TCP are closed at the same time [1]. At this time, time_wait traces will be left on both the client and the server, and the probability of occurrence is relatively small.

03 No source code, just a string

In this case, the server actively shuts down. Let's look at the source code of golang httpServer.

• http.ListenAndServe(":8081")
• server.ListenAndServe()
• srv.Serve(ln)
• go c.serve(connCtx) uses go coroutine to handle each request

The brief source code of the server connection processing request is as follows:

 func ( c * conn ) serve ( ctx context .Context ) {
 c .remoteAddr = c .rwc .RemoteAddr ( ) .String ( )
 ctx = context .WithValue ( ctx , LocalAddrContextKey , c .rwc .LocalAddr ( ) )
 defer func ( ) {
 if ! c .hijacked ( ) {
 c .close ( ) // When the go coroutine conn processes the request, it actively closes the underlying TCP connection
 c .setState ( c .rwc , StateClosed , runHooks )
 }
 } ( )

 ......
 // HTTP / 1.x from here on .

 ctx , cancelCtx : = context .WithCancel ( ctx )
 c .cancelCtx = cancelCtx
 defer cancelCtx ( )

 c .r = & connReader { conn : c }
 c .bufr = newBufioReader ( c .r )
 c .bufw = newBufioWriterSize ( checkConnErrorWriter { c } , 4 << 10 )

 for {
 w , err : = c .readRequest ( ctx )
 .....
 serverHandler { c .server } .ServeHTTP ( w , w .req )
 w .cancelCtx ( )
 if c .hijacked ( ) {
 return
 }
 w .finishRequest ( )
 if ! w .shouldReuseConnection ( ) {
 if w .requestBodyLimitHit || w .closedRequestBodyEarly ( ) {
 c .closeWriteAndWait ( )
 }
 return
 }
 c .setState ( c .rwc , StateIdle , runHooks )
 c .curReq .Store ( ( * response ) ( nil ) )

 if ! w .conn .server .doKeepAlives ( ) {
 // We 're in shutdown mode. We might' ve replied
 // to the user without "Connection: close" and
 // they might think they can send another
 // request , but such is life with HTTP / 1.1 .
 return
 }

 if d : = c .server .idleTimeout ( ) ; d != 0 {
 c .rwc .SetReadDeadline ( time .Now ( ) .Add ( d ) )
 if _ , err : = c .bufr .Peek ( 4 ) ; err != nil {
 return
 }
 }
 c .rwc .SetReadDeadline ( time .Time { } )
 }
 }

We need attention

①For loop, indicating an attempt to reuse the conn to handle incoming requests

②w.shouldReuseConnection() = false, indicating that the ClientConnection: Close header is read, closeAfterReply=true is set, the for loop is jumped out, the coroutine is about to end, and the defer function is executed before the end, and the connection is closed in the defer function

 c .close ( )
 ......
 // Close the connection.
 func ( c * conn ) close ( ) {
 c .finalFlush ( )
 c .rwc .Close ( )
 }

③If w.shouldReuseConnection() = true, set the connection state to idle, and continue the for loop to reuse the connection and process subsequent requests.

04My Gains

1. TCP 4-wave stereotypes

2. Effect of short connection: The active closing party will generate time_wait status on the machine and need to wait for 2MSL time before closing the SOCKET

3. Source code analysis of golang http keep-alive multiplexing tcp connection

4.tcpdump packet capture posture

Reference Links

[1] TCP double-ended shutdown: https://blog.csdn.net/q1007729991/article/details/69950255