Bug 664

Summary: [PATCH] libunbound: Python3 related fixes and changes
Product: unbound Reporter: Tomas Hozza <thozza>
Component: serverAssignee: unbound team <unbound-team>
Status: RESOLVED FIXED    
Severity: normal CC: cathya, wouter
Priority: P5    
Version: 1.5.3   
Hardware: All   
OS: Linux   
Attachments: 0001-Use-print_function-also-for-Python2.patch
0002-libunbound-examples-produce-sorted-output.patch
0003-libunbound-Python-libldns-is-not-used-anymore.patch
0004-Resolve-Python-3-incompatibilities-in-libunbound-and.patch

Description Tomas Hozza 2015-04-16 16:08:02 CEST
Hi.

I'm currently working on resolving unbound build issues in Fedora due to Python3. I'm reworking the Fedora package to ship Python2 and also Python3 libunbound and pythonmode bindings.

The python examples in libunbound seem like a good tests for comparing that the Python3 version gives the same results as Python2 version. However in case of the Python2 run, the built-in print statement is used instead of print function, which makes the output different on Python2 and Python3, thus comparing the results using a script hard.

Also results in output of some examples (e.g. mx-lookup.py) were randomly shuffled, thus hard to compare. I wrapped all lists with sorted() to produce predictable output all the time.

I'm attaching a patch to use print_function (available since Python 2.6) also with Python2


example of output before:

[root@unused-4-247 ~]# python2 /usr/share/doc/python-unbound/async-lookup.py
('Call_back:', {'done_flag': False, 'arbitrary': 'object'})
('Result:', ['217.31.205.50'])

[root@unused-4-247 ~]# python3 /usr/share/doc/python-unbound/async-lookup.py
Call_back: {'done_flag': False, 'arbitrary': 'object'}
Result: ['217.31.205.50']


Now:
[root@unused-4-247 ~]# python2 /usr/share/doc/python-unbound/async-lookup.py
Call_back: {'done_flag': False, 'arbitrary': 'object'}
Result: ['217.31.205.50']

[root@unused-4-247 ~]# python3 /usr/share/doc/python-unbound/async-lookup.py
Call_back: {'done_flag': False, 'arbitrary': 'object'}
Result: ['217.31.205.50']


I also have simple script I used for running the examples on Fedora with Python2 and Python3 and comparing the output. It was useful for testing. I can attach it too, if you think it would be useful for others.


Another issue was with unbound.ub_ctx_debugout() function which takes FILE* as parameter. I used the file_py3.i from ldns. Added typemap for "typecheck" due to how the function is declared. I also removed the typemap "freearg" since the FILE* is later used internally by the library for logging. This caused nasty crashes like:

[root@unused-4-247 ~]# python3 /usr/share/doc/python-unbound/dnssec-valid.py
*** Error in `python3': corrupted double-linked list: 0x0000000000fcb1b0 ***
*** Error in `python3': corrupted double-linked list: 0x0000000000fcb1b0 ***


There is still an issue with IDN and encoding when using Python3, but after 3 days fighting the Python3 + SWIG I need some rest from it. :) There is most probably some issue with wrong usage of PyBytes_* PyUnicode_* and PyString_*
Comment 1 Tomas Hozza 2015-04-16 16:08:42 CEST
Created attachment 277 [details]
0001-Use-print_function-also-for-Python2.patch
Comment 2 Tomas Hozza 2015-04-16 16:09:19 CEST
Created attachment 278 [details]
0002-libunbound-examples-produce-sorted-output.patch
Comment 3 Tomas Hozza 2015-04-16 16:09:42 CEST
Created attachment 279 [details]
0003-libunbound-Python-libldns-is-not-used-anymore.patch
Comment 4 Tomas Hozza 2015-04-16 16:09:59 CEST
Created attachment 280 [details]
0004-Resolve-Python-3-incompatibilities-in-libunbound-and.patch
Comment 5 Tomas Hozza 2015-04-16 16:13:01 CEST
BTW, the output of my testing script running all examples (except one using ldns, since we don't build ldns for Python3 in Fedora at this moment).

[root@unused-4-247 ~]# ./test_libunbound.py 
Running... /usr/share/doc/python-unbound/async-lookup.py
OK      async-lookup.py
Running... /usr/share/doc/python-unbound/dns-lookup.py
OK      dns-lookup.py
Running... /usr/share/doc/python-unbound/dnssec-valid.py
OK      dnssec-valid.py
Running... /usr/share/doc/python-unbound/example8-1.py
OK      example8-1.py
Running... /usr/share/doc/python-unbound/idn-lookup.py
ERROR: Python 2 run of 'idn-lookup.py' failed with exit status 1
ERROR: different output of 'idn-lookup.py' for Py2 and Py3

----------------------- PYTHON 2 -------------------------
Result:
      raw data: D9 1F CD 33
      address:217.31.205.51
Result:
      raw data: 00 0A 0C 78 6E 2D 2D 70 6F 74 61 2D 68 36 61 13 78 6E 2D 2D 68 6B 79 72 6B 79 2D 70 74 61 63 37 30 62 63 02 63 7A 00
Traceback (most recent call last):
  File "/usr/share/doc/python-unbound/idn-lookup.py", line 57, in <module>
    print("      priority:%d address:%s" % k)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0161' in position 28: ordinal not in range(128)

----------------------- PYTHON 3 -------------------------

----------------------------------------------------------
FAIL    idn-lookup.py
Running... /usr/share/doc/python-unbound/mx-lookup.py
OK      mx-lookup.py
Running... /usr/share/doc/python-unbound/ns-lookup.py
OK      ns-lookup.py
Running... /usr/share/doc/python-unbound/reverse-lookup.py
OK      reverse-lookup.py
Comment 6 Wouter Wijngaards 2015-04-16 16:42:59 CEST
Hi Tomas,

Thank you for the fixes, I have committed them.

Best regards, Wouter