Bug 696 - Not uniform distribution queries per threads
Not uniform distribution queries per threads
Status: ASSIGNED
Product: unbound
Classification: Unclassified
Component: server
1.5.1
x86_64 Linux
: P5 major
Assigned To: unbound team
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-13 16:08 CEST by sergey
Modified: 2015-08-20 11:04 CEST (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description sergey 2015-08-13 16:08:58 CEST
thread0.num.queries=34154
thread0.num.cachehits=31171
thread0.num.cachemiss=2983
thread0.num.prefetch=491
thread0.num.recursivereplies=2995
thread0.requestlist.avg=33.1425
thread0.requestlist.max=62
thread0.requestlist.overwritten=0
thread0.requestlist.exceeded=0
thread0.requestlist.current.all=29
thread0.requestlist.current.user=8
thread0.recursion.time.avg=1.195104
thread0.recursion.time.median=0.138591
thread1.num.queries=202689
thread1.num.cachehits=181976
thread1.num.cachemiss=20713
thread1.num.prefetch=2722
thread1.num.recursivereplies=20542
thread1.requestlist.avg=2798.57
thread1.requestlist.max=3335
thread1.requestlist.overwritten=0
thread1.requestlist.exceeded=0
thread1.requestlist.current.all=3168
thread1.requestlist.current.user=1936
thread1.recursion.time.avg=27.656320
thread1.recursion.time.median=14.1968
thread2.num.queries=107482
thread2.num.cachehits=97153
thread2.num.cachemiss=10329
thread2.num.prefetch=1411
thread2.num.recursivereplies=10331
thread2.requestlist.avg=106.557
thread2.requestlist.max=198
thread2.requestlist.overwritten=0
thread2.requestlist.exceeded=0
thread2.requestlist.current.all=120
thread2.requestlist.current.user=54
thread2.recursion.time.avg=1.399058
thread2.recursion.time.median=0.162205
thread3.num.queries=47650
thread3.num.cachehits=43156
thread3.num.cachemiss=4494
thread3.num.prefetch=685
thread3.num.recursivereplies=4488
thread3.requestlist.avg=50.3729
thread3.requestlist.max=90
thread3.requestlist.overwritten=0
thread3.requestlist.exceeded=0
thread3.requestlist.current.all=53
thread3.requestlist.current.user=23
thread3.recursion.time.avg=1.292380
thread3.recursion.time.median=0.137667
thread4.num.queries=23852
thread4.num.cachehits=21760
thread4.num.cachemiss=2092
thread4.num.prefetch=327
thread4.num.recursivereplies=2094
thread4.requestlist.avg=23.1732
thread4.requestlist.max=47
thread4.requestlist.overwritten=0
thread4.requestlist.exceeded=0
thread4.requestlist.current.all=18
thread4.requestlist.current.user=8
thread4.recursion.time.avg=1.074845
thread4.recursion.time.median=0.151066
thread5.num.queries=431125
thread5.num.cachehits=397109
thread5.num.cachemiss=34016
thread5.num.prefetch=5933
thread5.num.recursivereplies=32786
thread5.requestlist.avg=5676.82
thread5.requestlist.max=6215
thread5.requestlist.overwritten=0
thread5.requestlist.exceeded=814
thread5.requestlist.current.all=6202
thread5.requestlist.current.user=4072
thread5.recursion.time.avg=35.249825
thread5.recursion.time.median=26.7133
thread6.num.queries=117296
thread6.num.cachehits=107823
thread6.num.cachemiss=9473
thread6.num.prefetch=1628
thread6.num.recursivereplies=9468
thread6.requestlist.avg=81.9867
thread6.requestlist.max=187
thread6.requestlist.overwritten=0
thread6.requestlist.exceeded=0
thread6.requestlist.current.all=77
thread6.requestlist.current.user=41
thread6.recursion.time.avg=1.039746
thread6.recursion.time.median=0.138262
thread7.num.queries=120733
thread7.num.cachehits=111102
thread7.num.cachemiss=9631
thread7.num.prefetch=1612
thread7.num.recursivereplies=9616
thread7.requestlist.avg=87.528
thread7.requestlist.max=185
thread7.requestlist.overwritten=0
thread7.requestlist.exceeded=0
thread7.requestlist.current.all=86
thread7.requestlist.current.user=41
thread7.recursion.time.avg=1.065807
thread7.recursion.time.median=0.131948
Restarting didn't help. 
I have 16vCPUs.
Comment 1 Wouter Wijngaards 2015-08-13 16:50:54 CEST
Hi Sergey,

The kernel normally distributes queries unevenly, (unless it is running on full capacity).

The option so-reuseport: yes enables a socket option that distributes more evenly, and also improves performance.  I think that is what you want?

Best regards, Wouter
Comment 2 sergey 2015-08-14 08:24:03 CEST
(In reply to Wouter Wijngaards from comment #1)
> Hi Sergey,
> 
> The kernel normally distributes queries unevenly, (unless it is running on
> full capacity).
> 
> The option so-reuseport: yes enables a socket option that distributes more
> evenly, and also improves performance.  I think that is what you want?
> 
> Best regards, Wouter

Thanks I try and write the result
Comment 3 sergey 2015-08-14 08:24:30 CEST
(In reply to Wouter Wijngaards from comment #1)
> Hi Sergey,
> 
> The kernel normally distributes queries unevenly, (unless it is running on
> full capacity).
> 
> The option so-reuseport: yes enables a socket option that distributes more
> evenly, and also improves performance.  I think that is what you want?
> 
> Best regards, Wouter

Thanks I try and write the result
Comment 4 sergey 2015-08-14 08:24:43 CEST
(In reply to Wouter Wijngaards from comment #1)
> Hi Sergey,
> 
> The kernel normally distributes queries unevenly, (unless it is running on
> full capacity).
> 
> The option so-reuseport: yes enables a socket option that distributes more
> evenly, and also improves performance.  I think that is what you want?
> 
> Best regards, Wouter

Thanks I try and write the result
Comment 5 sergey 2015-08-17 09:51:41 CEST
(In reply to sergey from comment #4)
> (In reply to Wouter Wijngaards from comment #1)
> > Hi Sergey,
> > 
> > The kernel normally distributes queries unevenly, (unless it is running on
> > full capacity).
> > 
> > The option so-reuseport: yes enables a socket option that distributes more
> > evenly, and also improves performance.  I think that is what you want?
> > 
> > Best regards, Wouter
> 
> Thanks I try and write the result

I have version: 1.4.20 and when I added so-reuseport: yes
I received this message. If I need to update version to use this features where I can lookup manual?
/etc/unbound/unbound.conf:127: error: unknown keyword 'so-reuseport'
/etc/unbound/unbound.conf:127: error: stray ':'
/etc/unbound/unbound.conf:127: error: unknown keyword 'yes'
Comment 6 Wouter Wijngaards 2015-08-17 10:06:58 CEST
Hi Sergey,

Yes you have to update to get the option (and a lot of bug fixes!).  Or you can continue with the uneven load spread.  The uneven-ness is not harmful (except for you eyeballs looking at the graphs).

The latest version can be downloaded from unbound.net.  There is also documentation there.  The option is documented in the unbound.conf man page.

The SO_REUSEPORT option also needs kernel support.  If you have such an old unbound version, you may also have an old kernel.  Some recent Linux 2.6 kernels have backported the support.  New 3.x kernels also have support for the option.

Best regards, Wouter
Comment 7 sergey 2015-08-17 10:12:46 CEST
(In reply to Wouter Wijngaards from comment #6)
> Hi Sergey,
> 
> Yes you have to update to get the option (and a lot of bug fixes!).  Or you
> can continue with the uneven load spread.  The uneven-ness is not harmful
> (except for you eyeballs looking at the graphs).
> 
> The latest version can be downloaded from unbound.net.  There is also
> documentation there.  The option is documented in the unbound.conf man page.
> 
> The SO_REUSEPORT option also needs kernel support.  If you have such an old
> unbound version, you may also have an old kernel.  Some recent Linux 2.6
> kernels have backported the support.  New 3.x kernels also have support for
> the option.
> 
> Best regards, Wouter

Thanks. With kernel I think all good 3.10.0-229.7.2.el7.x86_64.
Comment 8 sergey 2015-08-20 10:26:40 CEST
(In reply to sergey from comment #7)
> (In reply to Wouter Wijngaards from comment #6)
> > Hi Sergey,
> > 
> > Yes you have to update to get the option (and a lot of bug fixes!).  Or you
> > can continue with the uneven load spread.  The uneven-ness is not harmful
> > (except for you eyeballs looking at the graphs).
> > 
> > The latest version can be downloaded from unbound.net.  There is also
> > documentation there.  The option is documented in the unbound.conf man page.
> > 
> > The SO_REUSEPORT option also needs kernel support.  If you have such an old
> > unbound version, you may also have an old kernel.  Some recent Linux 2.6
> > kernels have backported the support.  New 3.x kernels also have support for
> > the option.
> > 
> > Best regards, Wouter
> 
> Thanks. With kernel I think all good 3.10.0-229.7.2.el7.x86_64.

Hi Wouter Wijngaards.
Now I am install CEntos 7 and unbound 1.5.4 with 8 threads but I can't enable more than 1024 port. I need to use libevent but unfortunately I can't find any good man to do this. Maybe you can help?
Comment 9 Wouter Wijngaards 2015-08-20 10:28:22 CEST
Hi Sergey,

There is documentation
http://unbound.net/documentation/howto_optimise.html

Install libevent-devel, and ./configure --with-libevent

Best regards, Wouter
Comment 10 sergey 2015-08-20 10:43:06 CEST
(In reply to Wouter Wijngaards from comment #9)
> Hi Sergey,
> 
> There is documentation
> http://unbound.net/documentation/howto_optimise.html
> 
> Install libevent-devel, and ./configure --with-libevent
> 
> Best regards, Wouter

I am install unbound from with yum from  repo epel . How I can do enable link it with installed before libevent-devel ? Sorry I have low exp in Linux..
Comment 11 Wouter Wijngaards 2015-08-20 10:45:55 CEST
Hi Sergey,

I thought EPEL linked with libevent for the binaries from its repo.
You can see, unbound -h reports if it is linked with libevent.

You have to recompile to enable libevent, if EPEL does not do it.  Does it use RPMs?  You could get the source-rpm and edit a copy and then create your own rpm perhaps.  Or compile unbound from source; the tarball is on unbound.net.

Best regards, Wouter
Comment 12 sergey 2015-08-20 10:49:32 CEST
(In reply to Wouter Wijngaards from comment #11)
> Hi Sergey,
> 
> I thought EPEL linked with libevent for the binaries from its repo.
> You can see, unbound -h reports if it is linked with libevent.
> 
> You have to recompile to enable libevent, if EPEL does not do it.  Does it
> use RPMs?  You could get the source-rpm and edit a copy and then create your
> own rpm perhaps.  Or compile unbound from source; the tarball is on
> unbound.net.
> 
> Best regards, Wouter

Version 1.5.4
linked libs: mini-event internal (it uses select), OpenSSL 1.0.1e-fips 11 Feb 2013
linked modules: dns64 validator iterator
BSD licensed, see LICENSE in source package for details.

And I set in unbound.conf outgoing-range: 120 and when start unbound received warning: too many file descriptors requested. The builtinmini-event cannot handle more than 1024. Config for less fds or compile with libevent
Aug 20 11:47:57 unbound[4786:0] warning: continuing with less udp ports: 96
Comment 13 Wouter Wijngaards 2015-08-20 10:52:49 CEST
Hi Sergey,

Yeah it does not use libevent.  You have to configure --with-libevent and recompile to do so.  That means compiling yourself, or changing the (source-)rpm somehow.

Best regards, Wouter
Comment 14 sergey 2015-08-20 11:04:08 CEST
(In reply to Wouter Wijngaards from comment #13)
> Hi Sergey,
> 
> Yeah it does not use libevent.  You have to configure --with-libevent and
> recompile to do so.  That means compiling yourself, or changing the
> (source-)rpm somehow.
> 
> Best regards, Wouter

Ok. Thanks.