CMake, Boost, and python 2.7 on Tumbleweed

dbuck · February 5, 2020, 4:45pm

Hi,

I’ve been trying to package a game that uses libboost_python with the CMake build system and I cannot get it to find the proper libboost_python 2.7 library (/usr/lib64/libboost_python.so or /usr/lib64/libboost_python-py2_7.so.1.71.0). The game is here:

https://build.opensuse.org/package/show/games/freeorion

It always fails does one of 3 undesirable things:

Doesn’t find any library
Returns /usr/lib64/libboost_python.so.1.71.0
Returns /usr/lib64/libboost_python-py3.so.1.71.0

I’ve attempted to patch the CMakeLists.txt of the game to just use:

find_package(Boost COMPONENTS python REQUIRED)

And I’ve also done a lot with trying to manually set

Boost_PYTHON_LIBRARY

or use hints like:

set(Boost_PYTHON_VERSION_TAG "-py2_7")
set(Boost_PYTHON_VERSION "2.7")

Whatever I do, CMake does not find the proper boost python 2.7 library on tumbleweed. I’ve tried all the python-py27, python27, 2.7, etc. variants with all the options I could find. Does anyone have any suggestions?

Thanks!

P.S. This game takes a very long time to compile and there are other compile errors that I’m working with upstream to patch.

malcolmlewis · February 5, 2020, 4:47pm

Hi there
Yes, python 2 is dropped/deprecated from Tumbleweed Just build for python3

dbuck · February 5, 2020, 5:37pm

This game requires python 2.7 and they are in the process of migrating, but not for their next release. I attempted to use the python3 anyways and I get linker errors like this:

/usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyUnicode_AsUTF8String'

malcolmlewis · February 5, 2020, 6:11pm

dbuck:

This game requires python 2.7 and they are in the process of migrating, but not for their next release. I attempted to use the python3 anyways and I get linker errors like this:
/usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyUnicode_AsUTF8String'

Hi
Add the following to build and see if that helps;


export LC_ALL=en_US.utf8

PyUnicode_AsUTF8String is correct for python > 3.3, are you sure the boost link flag is there (check the CMakeLists.txt)?

dbuck · February 5, 2020, 7:26pm

malcolmlewis:

Hi
Add the following to build and see if that helps;
export LC_ALL=en_US.utf8
PyUnicode_AsUTF8String is correct for python > 3.3, are you sure the boost link flag is there (check the CMakeLists.txt)?

That option did not work. Here is the link command and it’s output. I’m not sure what flag you mean?

 2109s]  77%] Linking CXX executable freeoriond
 2109s] /usr/bin/cmake -E cmake_link_script CMakeFiles/freeoriond.dir/link.txt --verbose=1
 2109s] /usr/bin/ccache /var/lib/build/ccache/bin/c++   -DBOOST_ALLOW_DEPRECATED_HEADERS -O3 -DNDEBUG  -rdynamic CMakeFiles/freeoriond.dir/combat/CombatSystem.cpp.o CMakeFiles/freeoriond.dir/network/ServerNetworking.cpp.o CMakeFiles/freeoriond.dir/python/server/ServerWrapper.cpp.o CMakeFiles/freeoriond.dir/python/server/ServerFramework.cpp.o CMakeFiles/freeoriond.dir/python/CommonFramework.cpp.o CMakeFiles/freeoriond.dir/python/ConfigWrapper.cpp.o CMakeFiles/freeoriond.dir/python/EmpireWrapper.cpp.o CMakeFiles/freeoriond.dir/python/EnumWrapper.cpp.o CMakeFiles/freeoriond.dir/python/LoggingWrapper.cpp.o CMakeFiles/freeoriond.dir/python/UniverseWrapper.cpp.o CMakeFiles/freeoriond.dir/server/dmain.cpp.o CMakeFiles/freeoriond.dir/server/SaveLoad.cpp.o CMakeFiles/freeoriond.dir/server/ServerApp.cpp.o CMakeFiles/freeoriond.dir/server/ServerFSM.cpp.o CMakeFiles/freeoriond.dir/universe/UniverseGenerator.cpp.o CMakeFiles/freeoriond.dir/util/DependencyVersions.cpp.o  -o freeoriond  -Wl,-rpath,/usr/lib64/freeorion libfreeorioncommon.so libfreeorionparse.so -lpython2.7 /usr/lib64/libboost_python-py3.so.1.71.0 /usr/lib64/libboost_log.so.1.71.0 /usr/lib64/libboost_log_setup.so.1.71.0 -lpthread /usr/lib64/libboost_log.so.1.71.0 /usr/lib64/libboost_atomic.so.1.71.0 /usr/lib64/libboost_date_time.so.1.71.0 /usr/lib64/libboost_filesystem.so.1.71.0 /usr/lib64/libboost_iostreams.so.1.71.0 /usr/lib64/libboost_locale.so.1.71.0 /usr/lib64/libboost_chrono.so.1.71.0 /usr/lib64/libboost_regex.so.1.71.0 /usr/lib64/libboost_serialization.so.1.71.0 /usr/lib64/libboost_system.so.1.71.0 /usr/lib64/libboost_thread.so.1.71.0 -lz -lpthread 
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: CMakeFiles/freeoriond.dir/python/server/ServerFramework.cpp.o: in function `PythonServer::InitModules()':
 2110s] ServerFramework.cpp:(.text+0x233a): undefined reference to `boost::python::detail::init_module(char const*, void (*)())'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: CMakeFiles/freeoriond.dir/python/server/ServerFramework.cpp.o: in function `initfreeorion':
 2110s] ServerFramework.cpp:(.text+0xaf): undefined reference to `boost::python::detail::init_module(char const*, void (*)())'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: CMakeFiles/freeoriond.dir/python/CommonFramework.cpp.o: in function `PythonBase::Initialize()':
 2110s] CommonFramework.cpp:(.text+0x603a): undefined reference to `boost::python::detail::init_module(char const*, void (*)())'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: CMakeFiles/freeoriond.dir/python/CommonFramework.cpp.o: in function `initfreeorion_logger':
 2110s] CommonFramework.cpp:(.text+0x6f): undefined reference to `boost::python::detail::init_module(char const*, void (*)())'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyUnicode_AsUTF8String'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyUnicode_FromFormat'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyModule_Create2'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyUnicode_FromString'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `_Py_fopen'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyUnicode_FromStringAndSize'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyBytes_AsString'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyBytes_Size'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyUnicode_AsWideChar'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyUnicode_FromEncodedObject'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyUnicode_AsUTF8'
 2110s] /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/libboost_python-py3.so.1.71.0: undefined reference to `PyUnicode_InternFromString'
 2110s] collect2: error: ld returned 1 exit status
 2110s] make[2]: *** [CMakeFiles/freeoriond.dir/build.make:327: freeoriond] Error 1
 2110s] make[2]: Leaving directory '/home/abuild/rpmbuild/BUILD/src-tarball/build'
 2110s] make[1]: *** [CMakeFiles/Makefile2:688: CMakeFiles/freeoriond.dir/all] Error 2

malcolmlewis · February 5, 2020, 9:20pm

Hi
I see in the c++ line -lpython2.7 that should be python3 but it may also need -lboost_python3 after -lpthread.

tsu2 · February 5, 2020, 11:29pm

Can’t python 2.7 be provided by configuring a repository?
Of course, if python 2 has been deprecated in TW, that would mean that you’d likely have to package up any Python 2 dependencies since none would be provided by the installed system.

BTW -
As of today, it’s not gone yet… but of course you don’t build something that’ll work only today and not tomorrow…

https://software.opensuse.org/package/python?search_term="python"

TSU

malcolmlewis · February 5, 2020, 11:42pm

Hi
We are discussing packages in the development project to head off to Tumbleweed, going forward, must be built with python3, they will then work with the current Leap releases, no build, no get into Tumbleweed, no get into Leap going forward…

There is zero reason to not fix and build for python3.

dbuck · February 5, 2020, 11:59pm

I got python3 to link properly by disallowing python2 development files and altering the CMakeLists.txt to explicitly search for python3. However, the game does fail to work properly because of python2 syntax in game files.

I’ve been following development of the python port for this game and it will not be ready by release. This game only releases about once per year so it would be broken for a while. Is there no way to force python2 support on tumbleweed?

malcolmlewis · February 6, 2020, 12:37am

Hi
Not that I’m aware of…

What are the errors seen?

dbuck · February 6, 2020, 1:03am

At a minimum the standard syntax ones (e.g. print ""), but the game uses python in AI and many other math-intensive scenarios and I have no idea if a drop-in replacement of python3 would break other things (i.e. integer division).

malcolmlewis · February 6, 2020, 1:28am

Hi
I would disable the build Tumbleweed builds until it python3 ready…

If you look at DimStar’s Factory branch and switch to obs_scm service (it’s what I use these days) can pull the latest commits in a flash, plus switching to obscpio is a build saver…

dyfet · February 6, 2020, 5:21pm

I could imagine someone setting up a purely community repo that offered python2 to solve this on obs (not me though ;), perhaps rebuilding pre-existing python2 leap packages under tumbleweed. But I happen to fully agree that python2 should be excluded from official repos going forward.

malcolmlewis · February 6, 2020, 5:35pm

Hi
I don’t think that would be a good idea, no upstream support for any security issues? I would not want to use it Anyway, going forward as Tumbleweed does, I still have 40 python2 packages on my system of the 2314 python2 packages available…


awk "BEGIN {print `zypper -q --no-refresh se -t package python2 | wc -l` - 3}"
2314
awk "BEGIN {print `zypper -q --no-refresh se -i -t package python2 | wc -l` - 3}"
40

dyfet · February 6, 2020, 7:22pm

malcolmlewis:

Hi
I don’t think that would be a good idea, no upstream support for any security issues? I would not want to use it Anyway, going forward as Tumbleweed does, I still have 40 python2 packages on my system of the 2314 python2 packages available…
awk "BEGIN {print `zypper -q --no-refresh se -t package python2 | wc -l` - 3}"
2314
awk "BEGIN {print `zypper -q --no-refresh se -i -t package python2 | wc -l` - 3}"
40

I agree there that I would not want to use it either. But there are clearly enterprise users who still have python2 in active deployment, it still comes as a “required” base on IBM (Redhat 7) Linux, and IBM has bought the joy of supporting that themselves for perhaps another decade there, heck they still have in active support RedHat 6 that came with Python 2.6. Those future patches could possibly be brought over if some user really wanted to support it in their own repo in obs I suppose. Certainly they do have the freedom to do so. I know I wouldn’t be using it either, though.

malcolmlewis · February 6, 2020, 7:29pm

Hi
Getting a bit OT For SLE 15 SP1 it’s a separate module and not a requirement…

dbuck · February 7, 2020, 4:08am

Well, it’s settled - one of the game developers told me to just wait until next game release as current python3 patches couldn’t be backported anyways. This package was removed from Factory because of failing Tumbleweed build, does Factory require Tumbleweed to build?

malcolmlewis · February 7, 2020, 4:23am

Hi
In a development repo, yes for it to move to factory.