Bug 664 - [PATCH] libunbound: Python3 related fixes and changes
[PATCH] libunbound: Python3 related fixes and changes
Product: unbound
Classification: Unclassified
Component: server
All Linux
: P5 normal
Assigned To: unbound team
Depends on:
  Show dependency treegraph
Reported: 2015-04-16 16:08 CEST by Tomas Hozza
Modified: 2015-04-16 16:42 CEST (History)
2 users (show)

See Also:

0001-Use-print_function-also-for-Python2.patch (4.54 KB, patch)
2015-04-16 16:08 CEST, Tomas Hozza
Details | Diff
0002-libunbound-examples-produce-sorted-output.patch (7.77 KB, patch)
2015-04-16 16:09 CEST, Tomas Hozza
Details | Diff
0003-libunbound-Python-libldns-is-not-used-anymore.patch (1.62 KB, patch)
2015-04-16 16:09 CEST, Tomas Hozza
Details | Diff
0004-Resolve-Python-3-incompatibilities-in-libunbound-and.patch (12.11 KB, patch)
2015-04-16 16:09 CEST, Tomas Hozza
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Tomas Hozza 2015-04-16 16:08:02 CEST

I'm currently working on resolving unbound build issues in Fedora due to Python3. I'm reworking the Fedora package to ship Python2 and also Python3 libunbound and pythonmode bindings.

The python examples in libunbound seem like a good tests for comparing that the Python3 version gives the same results as Python2 version. However in case of the Python2 run, the built-in print statement is used instead of print function, which makes the output different on Python2 and Python3, thus comparing the results using a script hard.

Also results in output of some examples (e.g. mx-lookup.py) were randomly shuffled, thus hard to compare. I wrapped all lists with sorted() to produce predictable output all the time.

I'm attaching a patch to use print_function (available since Python 2.6) also with Python2

example of output before:

[root@unused-4-247 ~]# python2 /usr/share/doc/python-unbound/async-lookup.py
('Call_back:', {'done_flag': False, 'arbitrary': 'object'})
('Result:', [''])

[root@unused-4-247 ~]# python3 /usr/share/doc/python-unbound/async-lookup.py
Call_back: {'done_flag': False, 'arbitrary': 'object'}
Result: ['']

[root@unused-4-247 ~]# python2 /usr/share/doc/python-unbound/async-lookup.py
Call_back: {'done_flag': False, 'arbitrary': 'object'}
Result: ['']

[root@unused-4-247 ~]# python3 /usr/share/doc/python-unbound/async-lookup.py
Call_back: {'done_flag': False, 'arbitrary': 'object'}
Result: ['']

I also have simple script I used for running the examples on Fedora with Python2 and Python3 and comparing the output. It was useful for testing. I can attach it too, if you think it would be useful for others.

Another issue was with unbound.ub_ctx_debugout() function which takes FILE* as parameter. I used the file_py3.i from ldns. Added typemap for "typecheck" due to how the function is declared. I also removed the typemap "freearg" since the FILE* is later used internally by the library for logging. This caused nasty crashes like:

[root@unused-4-247 ~]# python3 /usr/share/doc/python-unbound/dnssec-valid.py
*** Error in `python3': corrupted double-linked list: 0x0000000000fcb1b0 ***
*** Error in `python3': corrupted double-linked list: 0x0000000000fcb1b0 ***

There is still an issue with IDN and encoding when using Python3, but after 3 days fighting the Python3 + SWIG I need some rest from it. :) There is most probably some issue with wrong usage of PyBytes_* PyUnicode_* and PyString_*
Comment 1 Tomas Hozza 2015-04-16 16:08:42 CEST
Created attachment 277 [details]
Comment 2 Tomas Hozza 2015-04-16 16:09:19 CEST
Created attachment 278 [details]
Comment 3 Tomas Hozza 2015-04-16 16:09:42 CEST
Created attachment 279 [details]
Comment 4 Tomas Hozza 2015-04-16 16:09:59 CEST
Created attachment 280 [details]
Comment 5 Tomas Hozza 2015-04-16 16:13:01 CEST
BTW, the output of my testing script running all examples (except one using ldns, since we don't build ldns for Python3 in Fedora at this moment).

[root@unused-4-247 ~]# ./test_libunbound.py 
Running... /usr/share/doc/python-unbound/async-lookup.py
OK      async-lookup.py
Running... /usr/share/doc/python-unbound/dns-lookup.py
OK      dns-lookup.py
Running... /usr/share/doc/python-unbound/dnssec-valid.py
OK      dnssec-valid.py
Running... /usr/share/doc/python-unbound/example8-1.py
OK      example8-1.py
Running... /usr/share/doc/python-unbound/idn-lookup.py
ERROR: Python 2 run of 'idn-lookup.py' failed with exit status 1
ERROR: different output of 'idn-lookup.py' for Py2 and Py3

----------------------- PYTHON 2 -------------------------
      raw data: D9 1F CD 33
      raw data: 00 0A 0C 78 6E 2D 2D 70 6F 74 61 2D 68 36 61 13 78 6E 2D 2D 68 6B 79 72 6B 79 2D 70 74 61 63 37 30 62 63 02 63 7A 00
Traceback (most recent call last):
  File "/usr/share/doc/python-unbound/idn-lookup.py", line 57, in <module>
    print("      priority:%d address:%s" % k)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0161' in position 28: ordinal not in range(128)

----------------------- PYTHON 3 -------------------------

FAIL    idn-lookup.py
Running... /usr/share/doc/python-unbound/mx-lookup.py
OK      mx-lookup.py
Running... /usr/share/doc/python-unbound/ns-lookup.py
OK      ns-lookup.py
Running... /usr/share/doc/python-unbound/reverse-lookup.py
OK      reverse-lookup.py
Comment 6 Wouter Wijngaards 2015-04-16 16:42:59 CEST
Hi Tomas,

Thank you for the fixes, I have committed them.

Best regards, Wouter