ngx

我们知道针对后端的keepalive是通过nginx.conf配置文件来指定的，例如

 

    upstream resins{         server 61.135.250.217:6800;         keepalive 1024;     }

 

nginx在读取配置文件的时候，就会执行指令相应的函数，查看ngx_http_upstream_keepalive_module源代码，

我们发现keepalive指令对应ngx_http_upstream_keepalive函数

 

     67 static ngx_command_t  ngx_http_upstream_keepalive_commands[] = {      68      69     { ngx_string("keepalive"),      70       NGX_HTTP_UPS_CONF|NGX_CONF_TAKE12,      71       ngx_http_upstream_keepalive, //这里会指明遇到keepalive指令会执行这个函数，这个函数来改变一切      72       0,      73       0,      74       NULL },      75      76       ngx_null_command      77 };

 

读取配置的时候就会执行下面的函数，执行堆栈如下：

 

    #0  ngx_http_upstream_keepalive (cf=0xbffe8ec0, cmd=0x80e8180, conf=0x0)         at /home/wangbin/work/memcached/keepalive/ngx_http_upstream_keepalive_module.c:438     #1  0x0805d892 in ngx_conf_handler (cf=0xbffe8ec0, last=0) at src/core/ngx_conf_file.c:393     #2  0x0805d50b in ngx_conf_parse (cf=0xbffe8ec0, filename=0x0) at src/core/ngx_conf_file.c:243     #3  0x0809449d in ngx_http_upstream (cf=0xbffe8ec0, cmd=0x80e4740, dummy=0x9725bfc) at src/http/ngx_http_upstream.c:3911     #4  0x0805d892 in ngx_conf_handler (cf=0xbffe8ec0, last=1) at src/core/ngx_conf_file.c:393     #5  0x0805d50b in ngx_conf_parse (cf=0xbffe8ec0, filename=0x0) at src/core/ngx_conf_file.c:243     #6  0x080727dc in ngx_http_block (cf=0xbffe8ec0, cmd=0x80e1640, conf=0x97252a8) at src/http/ngx_http.c:241     #7  0x0805d892 in ngx_conf_handler (cf=0xbffe8ec0, last=1) at src/core/ngx_conf_file.c:393     #8  0x0805d50b in ngx_conf_parse (cf=0xbffe8ec0, filename=0x9724c00) at src/core/ngx_conf_file.c:243     #9  0x0805a80f in ngx_init_cycle (old_cycle=0xbffe8f60) at src/core/ngx_cycle.c:263     #10 0x0804a8fd in main (argc=1, argv=0xbffe90e4) at src/core/nginx.c:324

 

 

我们可以看到执行ngx_http_upstream_keepalive的时候，已经改变了upstream的相关内容

 

ngx_http_upstream_keepalive(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)

 

    446     kcf->original_init_upstream = uscf->peer.init_upstream     447                                   ? uscf->peer.init_upstream     448                                   : ngx_http_upstream_init_round_robin;//保留旧的函数指针     449     450     uscf->peer.init_upstream = ngx_http_upstream_init_keepalive;  //设置新的初始化upstream函数指针

改变ngx_http_upstream_srv_conf_t的init_upstream为ngx_http_upstream_init_keepalive，

kcf->original_init_upstream保存了uscf->peer.init_upstream的指针这样就可以截获upstream，让其先到ngx_http_upstream_init_keepalive这儿来

 

 

在ngx_http_upstream_init_keepalive中     先利用原来保存的 kcf->original_init_upstream执行upstream的初始化，然后再执行keepalive方面的初始化     128     if (kcf->original_init_upstream(cf, us) != NGX_OK) {     129         return NGX_ERROR;     130     }

    131     132     kcf->original_init_peer = us->peer.init;//保存原始的init     133                                                                                                                                     134     us->peer.init = ngx_http_upstream_init_keepalive_peer;//设置先到这个函数中来

 

ngx_http_upstream_init_keepalive的执行堆栈如下：

   #0  ngx_http_upstream_init_keepalive (cf=0xbfe493e0, us=0x8656694)         at /home/wangbin/work/memcached/keepalive/ngx_http_upstream_keepalive_module.c:122     #1  0x08090d1c in ngx_http_upstream_init_main_conf (cf=0xbfe493e0, conf=0x864afa0) at src/http/ngx_http_upstream.c:4394     #2  0x0806fadd in ngx_http_block (cf=0xbfe493e0, cmd=0x80ca620, conf=0x864a784) at src/http/ngx_http.c:260     #3  0x0805d023 in ngx_conf_handler (cf=0xbfe493e0, last=1) at src/core/ngx_conf_file.c:393     #4  0x0805cc9f in ngx_conf_parse (cf=0xbfe493e0, filename=0x864a0d0) at src/core/ngx_conf_file.c:243     #5  0x0805a1c7 in ngx_init_cycle (old_cycle=0xbfe49480) at src/core/ngx_cycle.c:263     #6  0x0804a59b in main (argc=1, argv=0xbfe49604) at src/core/nginx.c:327

 

上面的函数执行是为了预先设置好http请求过来时的执行顺序，让其先执行ngx_http_upstream_init_keepalive_peer，而不是原来默认的函数

 

当一个请求过来时，会执行一系列操作，如果需要访问后端，就会执行ngx_http_upstream_init_request，

而ngx_http_upstream_init_request就会执行uscf->peer.init(r, uscf)，上面已经改变了upstream默认的函数指针

，已经指向了ngx_http_upstream_init_keepalive_peer。

 

 

在ngx_http_upstream_init_keepalive_peer函数中     174     if (kcf->original_init_peer(r, us) != NGX_OK) {//先执行原始upstream peer的初始化     175         return NGX_ERROR;     176     }

 

    178     kp->conf = kcf;     179     kp->upstream = r->upstream;//这两句是为了辅助目地设置的         180     kp->data = r->upstream->peer.data;     181     kp->original_get_peer = r->upstream->peer.get;     182     kp->original_free_peer = r->upstream->peer.free; //这上面3句就是保留原始peer的初始化信息     183     184     r->upstream->peer.data = kp;//这句是为了ngx_http_upstream_get_keepalive_peer参数中的void* data,否则就是原始的r->upstream->peer.data     185     r->upstream->peer.get = ngx_http_upstream_get_keepalive_peer;     186     r->upstream->peer.free = ngx_http_upstream_free_keepalive_peer;//进一步截获，让系统先到这儿来

 

在ngx_http_upstream_init_keepalive_peer函数中，改变了peer默认的函数指针，使其先到keepalive模块来

 

真正需要连接的时候，nginx会执行u->peer.get，于是就执行了上面设置的ngx_http_upstream_get_keepalive_peer

 

部分执行堆栈如下：

   #0  ngx_http_upstream_get_keepalive_peer (pc=0x84860dc, data=0x8486400)         at /home/wangbin/work/memcached/keepalive/ngx_http_upstream_keepalive_module.c:260     #1  0x08067cdf in ngx_event_connect_peer (pc=0x84860dc) at src/event/ngx_event_connect.c:24     #2  0x080919ef in ngx_http_upstream_connect (r=0x8481328, u=0x84860d4) at src/http/ngx_http_upstream.c:1084

 

 

在ngx_http_upstream_get_keepalive_peer函数中     230     rc = kp->original_get_peer(pc, kp->data);//先获取原始的信息，比如访问ngx_http_upstream_get_round_robin_peer函数，这里还 没有赋connection                     (gdb) p *pc                     $2 = {connection = 0x0, sockaddr = 0x8491dc4, socklen = 16, name = 0x8493a9c,

tries = 1, get = 0x80cbc73 <ngx_http_upstream_get_keepalive_peer>, free = 0x80cbf04 <ngx_http_upstream_free_keepalive_peer>,                       data = 0x8486400, local = 0x0, rcvbuf = 0, log = 0x8481258, cached = 0, log_error = 1}

    。。。     259             pc->connection = c;//从cache中赋值给pc     260             pc->cached = 1;       这里举例的就是cache中获取connection，赋值给pc->connection

        返回ngx_event_connect_peer 函数时，代码如下：      24     rc = pc->get(pc, pc->data);      25     if (rc != NGX_OK) {      26         return rc;      27     }

     29     s = ngx_socket(pc->sockaddr->sa_family, SOCK_STREAM, 0);

 

      这里需要注意的是如果是第一次访问，cache没有现成的连接，ngx_http_upstream_get_keepalive_peer会返回OK，会执行29行代码及其后面的连接操作；

     如果有现成的连接，ngx_http_upstream_get_keepalive_peer会返回NGX_DONE,所以会跳过后面29行的代码，就不会进入创建连接的过程。

 

     当请求处理完的时候，nginx会执行u->peer.free，也就是先到ngx_http_upstream_free_keepalive_peer函数中来。

 

    ngx_http_upstream_free_keepalive_peer执行堆栈如下：         #0  ngx_http_upstream_free_keepalive_peer (pc=0x84860e8, data=0x848640c, state=0)             at /home/wangbin/work/memcached/keepalive/ngx_http_upstream_keepalive_module.c:368         #1  0x08094fbc in ngx_http_upstream_finalize_request (r=0x84966b0, u=0x84860e0, rc=0) at src/http/ngx_http_upstream.c:2952         #2  0x08094841 in ngx_http_upstream_process_request (r=0x84966b0) at src/http/ngx_http_upstream.c:2679         #3  0x080945ba in ngx_http_upstream_process_upstream (r=0x84966b0, u=0x84860e0) at src/http/ngx_http_upstream.c:2606         #4  0x08093d1c in ngx_http_upstream_send_response (r=0x84966b0, u=0x84860e0) at src/http/ngx_http_upstream.c:2295         #5  0x08092722 in ngx_http_upstream_process_header (r=0x84966b0, u=0x84860e0) at src/http/ngx_http_upstream.c:1592         #6  0x080915ad in ngx_http_upstream_handler (ev=0xb7df30d8) at src/http/ngx_http_upstream.c:898         #7  0x08071c2e in ngx_epoll_process_events (cycle=0x8481b98, timer=56857, flags=1) at src/event/modules/ngx_epoll_module.c:642         #8  0x08064d87 in ngx_process_events_and_timers (cycle=0x8481b98) at src/event/ngx_event.c:245         #9  0x0806ddf0 in ngx_single_process_cycle (cycle=0x8481b98) at src/os/unix/ngx_process_cycle.c:306         #10 0x0804a969 in main (argc=1, argv=0xbfff98f4) at src/core/nginx.c:398

    需要注意的是，在函数ngx_http_upstream_free_keepalive_peer中，都会执行

       return   kp->original_free_peer(pc, kp->data, state);

    这个函数并没有做关闭socket的操作。

 

    真正改变tcp连接不关闭的原因如下：     由于在ngx_http_upstream_free_keepalive_peer函数执行了下面这句     346             pc->connection = NULL;                 (gdb) p pc             $18 = (ngx_peer_connection_t *) 0x84860e8                 导致了nginx没有关闭对后端连接的关闭操作，由于是否关闭连接操作的控制函数在ngx_http_upstream_finalize_request，部分其函数代码如下：             2955        if (u->peer.connection) {         2956         2875         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,         2876                        "close http upstream connection: %d",         2877                        u->peer.connection->fd);         2878         2879         ngx_close_connection(u->peer.connection);         2880     }

    由于pc->connection = NULL，也就是u->peer.connection为null，所以连接保存了下来（保存在 ngx_http_upstream_keepalive_module 的cache中），没有被关闭，供下次调用时继续使用，省去了tcp连接建立和关闭的过程。