Python scripts fail on 12.3

English Programming/Scripting
#1

Hi All,

I’m going slightly barmy trying to figure this out, having been round several Python fora and elsewhere. I am discussing this with the vendors, but I thought I’d try here and have another sting to my bow.

I have a Ruckus Wireless system, and one of the means of providing access to it is to run a web-based portal provided by Ruckus that interfaces with the controller, and delivers a dynamic PSK to each user (each user gets a different PSK for our SSID) based on credentials the user supplies.

These comprise a handful of html pages and three python scripts. Simples.

Because our University provides Single Sign-On via the Shibboleth system, I figured I could drop the Ruckus portal software in a Shibboleth-protected directory on the web server, and pass the username and password to the wireless controller. I’ve configured the wireless controller to talk to a Radius server in our local network, which will just answer “YES” to valid usernames sent to it. Basically, a User will authenticate to Unversity SSO via Shibboleth, be passed to the local Wifi portal, which will send the username and a password to the Radius server via the wifi controller. If the username exists locally (in our AD) then the user is granted access to the wifi.

The problem is that although a user can successfully authenticate to SSO via Shibboleth, the python scripts totally fail - even when providing a username and password manually (I haven’t got round to feeding them data from the shibboleth session yet).

The supremely irritating thing is that in my lengthy discussions with Ruckus (gong on since November) they’ve got it to work on their Linux test boxes, but they’re using CentOS 6.

I’m having to use something I can install Shibboleth on and which runs Python 2.7 - so that means OpenSuse 12.3 or Centos 6. I’m testing Centos build today, but it is somewhat different (understatement) to the installations I’m used to (SLES 11 and OpenSuse12/13). I can’t test it with 13.1 as Shibboleth does not work on that system yet - the Apache module fails to load as there are dependencies that can’t be provided.

The error is as follows:

MOD_PYTHON ERROR

ProcessId: 15322
Interpreter: ‘gatekeeper.new.ox.ac.uk’

ServerName: ‘gatekeeper.new.ox.ac.uk’
DocumentRoot: ‘/srv/www/htdocs’

URI: ‘/cgi-bin/hotspot_login.py’
Location: None
Directory: ‘/srv/www/cgi-bin/’
Filename: ‘/srv/www/cgi-bin/hotspot_login.py’
PathInfo: ‘’

Phase: ‘PythonHandler’
Handler: ‘mod_python.cgihandler’

Traceback (most recent call last):

File “/usr/lib64/python2.7/site-packages/mod_python/importer.py”, line 1537, in HandlerDispatch
default=default_handler, arg=req, silent=hlist.silent)

File “/usr/lib64/python2.7/site-packages/mod_python/importer.py”, line 1229, in _process_target
result = _execute_target(config, req, object, arg)

File “/usr/lib64/python2.7/site-packages/mod_python/importer.py”, line 1128, in _execute_target
result = object(arg)

File “/usr/lib64/python2.7/site-packages/mod_python/cgihandler.py”, line 96, in handler
imp.load_module(module_name, fd, path, desc)

File “/srv/www/cgi-bin/hotspot_login.py”, line 22, in <module>
import xmlcommon

ImportError: No module named xmlcommon

xmlcommon.py DOES exist in the same directory as hotspot_login.py (the python script that kicks the process off), and has the same permissions.

If I start the Python interpreter from the command line, I can import that module - but it fails to do so when called from another python script in the same directory.

Any pointers? I’m all out of ideas here.

Cheers,
James

#2

On 2014-01-28, jhd10374 <jhd10374@no-mx.forums.opensuse.org> wrote:
>
> Hi All,
>
> I’m going slightly barmy trying to figure this out, having been round
> several Python fora and elsewhere. I am discussing this with the
> vendors, but I thought I’d try here and have another sting to my bow.

It might well prove to be a `sting’!

I have a Ruckus Wireless system, and one of the means of providing
access to it is to run a web-based portal provided by Ruckus that
interfaces with the controller, and delivers a dynamic PSK to each user
(each user gets a different PSK for our SSID) based on credentials the
user supplies.

The supremely irritating thing is that in my lengthy discussions with
Ruckus (gong on since November) they’ve got it to work on their Linux
test boxes, but they’re using CentOS 6.

Most University Departments (including mine) who dabble in Linux know very little about distributions outside their
Red-Hat based comfort zone. Sometimes it’s just best to add another distro on your hard drive and conform along the
paths of least resistence. If you can’t bear that, consider installing Centos inside KVM inside openSUSE.

I’m having to use something I can install Shibboleth on and which runs
Python 2.7 - so that means OpenSuse 12.3 or Centos 6. I’m testing
Centos build today, but it is somewhat different (understatement) to the
installations I’m used to (SLES 11 and OpenSuse12/13).

Build it? Why don’t you just binary install it?

I can’t test it
with 13.1 as Shibboleth does not work on that system yet - the Apache
module fails to load as there are dependencies that can’t be provided.

The error is as follows:

If you want anyone other than NNTP users confined to console use to view the output, please surround it with code tags
iconified by octothorpes.

File “/srv/www/cgi-bin/hotspot_login.py”, line 22, in
import xmlcommon

ImportError: No module named xmlcommon

xmlcommon.py DOES exist in the same directory as hotspot_login.py (the
python script that kicks the process off), and has the same permissions.

If I start the Python interpreter from the command line, I can import
that module - but it fails to do so when called from another python
script in the same directory.

Have you tried adding the directory containing xmlcommon.py to your python path inside ~/.bashrc? e.g.


export PYTHONPATH=.:/home/user/path_to_xmlcommon/:PYTHONPATH

> Any pointers? I’m all out of ideas here.

I don’t know anything about web-based coding (I don’t even know html) and so I strongly suspect my suggestion won’t
work.

#3

On Tue 28 Jan 2014 03:36:01 PM CST, jhd10374 wrote:

If I start the Python interpreter from the command line, I can import
that module - but it fails to do so when called from another python
script in the same directory.

Any pointers? I’m all out of ideas here.

Cheers,
James

Hi
So the beginning of the script calls all the imports os sys etc?

You might have to add some debug to the scripts so it tells you where
it’s looking when it trys to import the module.

–
Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
openSUSE 13.1 (Bottle) (x86_64) GNOME 3.10.2 Kernel 3.11.6-4-desktop
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks!

#4

And please, next time you post computer text here in a post, use CODE tags around it (I see you tried some non-standard font to make things more clear, but the CODE tags are much better). You get the CODE tags by clicking on the # buttton in the toolbar of the post editor.

#5

Not specifically in .bashrc, but I have the following in /etc/profile.d/python.sh:


export PYTHONPATH=/srv/www/cgi-bin

Since the code is run as a cgi script by Apache, I wasn’t convinced putting it in a user’s .bashrc would be useful. However, I’ll give it a whirl.

Cheers,
James

#6

Yes: the script starts:

#!/usr/bin/python


from subprocess import Popen
from xml.dom import minidom
import cgi     
import cStringIO
import os
import pycurl  
import re
import socket
import logging
import urllib
import urllib2
from urlparse import parse_qs, urlsplit, urlunsplit
from urllib import urlencode
import hashlib
import sys
import datetime




# Custom modules - these must be placed in the same directory as this file


import xmlcommon
import hspotcommon

J

#7

jhd10374 wrote:
> flymail;2620096 Wrote:
>> On 2014-01-28, jhd10374 <jhd10374@no-mx.forums.opensuse.org> wrote:
>> Have you tried adding the directory containing xmlcommon.py to your
>> python path inside ~/.bashrc? e.g.
>>
> Code:
> --------------------
> > >
> > export PYTHONPATH=.:/home/user/path_to_xmlcommon/:PYTHONPATH
> >
> --------------------
>>
>
> Not specifically in .bashrc, but I have the following in
> /etc/profile.d/python.sh:
>
>
> Code:
> --------------------
>
> export PYTHONPATH=/srv/www/cgi-bin
>
> --------------------

I’m noeither a python guru nor a shell expert, but AFAIK profile is only
used for login shells whilst bashrc is used for all. Since a CGI script
is not a login environment, the path needs to be set in bashrc, or
explicitly in the script. Normally it isn’t usual to include the current
directory (.) in system paths, because it increases the security risk.

#8

Ok, that makes sense. Since a ~/.bashrc is an individual user’s additions to bashrc, I assume I am looking to set this somewhere in the /etc directory?

Aha - In there I can see only /etc/bash.bashrc which warns me against making changes to it and to use bash.bashrc.local

Testing…

Cheers,
James

#9

When I understand the explanation above correctly then an Apache process, which is normaly owned by user wwwrun, starts the CGI program, which in this case is a python script, and that will thus also run with owner wwwrun.

When this assumption is correct, it is useless to add anything to .bashrc of another user than wwwrun. Also, no bash is involved at all, thus changing e.g. /etc/bash.bashrc.local is also useless.

The above because I get the idea that seraching goes along a path that is not very promising. Just my two cents.

#10

hcvv wrote:
> When I understand the explanation above correctly then an Apache
> process, which is normaly owned by user wwwrun, starts the CGI program,
> which in this case is a python script, and that will thus also run with
> owner wwwrun.
>
> When this assumption is correct, it is useless to add anything to
> .bashrc of another user than wwwrun. Also, no bash is involved at all,
> thus changing e.g. /etc/bash.bashrc.local is also useless.

Very true! So either change the library path in the code itself or make
use of Apache’s SetEnv directive

#11

What is the current working directory of the process in which that
script runs? If it is not the same as the path where your custom modules
are located it will of course not find them.
The easiest solution is to change the working directory to the
appropriate path. There are several possibilities to do that, you could
even do it in your python script before the import statements for the
custom modules.

–
PC: oS 13.1 x86_64 | i7-2600@3.40GHz | 16GB | KDE 4.11 | GTX 650 Ti
ThinkPad E320: oS 13.1 x86_64 | i3@2.30GHz | 8GB | KDE 4.11 | HD 3000
HTPC: oS 13.1 x86_64 | Celeron@1.8GHz | 2GB | Gnome 3.10 | HD 2500

#12

Am 28.01.2014 18:18, schrieb Martin Helm:
> What is the current working directory of the process in which that
> script runs? If it is not the same as the path where your custom modules
> are located it will of course not find them.
Sorry that was of course nonsense, python looks into the path where the
script is located not the cwd.

Check your python path and add


import sys
print sys.path

at the beginning of your script to see what is in it.

–
PC: oS 13.1 x86_64 | i7-2600@3.40GHz | 16GB | KDE 4.11 | GTX 650 Ti
ThinkPad E320: oS 13.1 x86_64 | i3@2.30GHz | 8GB | KDE 4.11 | HD 3000
HTPC: oS 13.1 x86_64 | Celeron@1.8GHz | 2GB | Gnome 3.10 | HD 2500

#13

Ok, I edited hotspot_login.py to add those lines.

The file is now being offered as a download when called from the web page that uses it. Don’t think this is related (it has happened before), so to check sys.path I did the following:


gatekeeper:/srv/www/cgi-bin # python
Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.path
'', '/usr/lib/python2.7/site-packages/setuptools-2.0.1-py2.7.egg', '/srv/www/cgi-bin', '/srv/www/cgi-bin/PYTHONPATH', '/usr/lib/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/local/lib64/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages']
>>>
  • /srv/www/cgi-bin is present, but then so is /srv/www/cgi-bin/PYTHONPATH which means I broke something somewhere.

Thanks,
J

#14

I have the following in my Apache default-server.conf - does it not equate?



<Directory "/srv/www/cgi-bin">
 AllowOverride None
 Options +ExecCGI
 Options Indexes FollowSymLinks ExecCGI 
 AddHandler cgi-script .cgi .py .pyc
 Order allow,deny
 Allow from all
 Require all granted 
 AddHandler mod_python .py .pyc
 PythonHandler mod_python.cgihandler
 PythonDebug on
 PythonPath "sys.path+'/srv/www/cgi-bin/']"
 </Directory>
  • Particularly the PythonPath directive?

Cheers,
James

#15

jhd10374 wrote:
> djh-novell;2620128 Wrote:
>> hcvv wrote:
>>> When this assumption is correct, it is useless to add anything to
>>> .bashrc of another user than wwwrun. Also, no bash is involved at all,
>>> thus changing e.g. /etc/bash.bashrc.local is also useless.
>> Very true! So either change the library path in the code itself or make
>> use of Apache’s SetEnv directive
>
> I have the following in my Apache default-server.conf - does it not
> equate?

I’m afraid I don’t know. As I said, I’m not a Python guru.

#16

On 2014-01-29, jhd10374 <jhd10374@no-mx.forums.opensuse.org> wrote:
> Code:
> --------------------
>
> gatekeeper:/srv/www/cgi-bin # python
> Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2
> Type “help”, “copyright”, “credits” or “license” for more information.
> >>> import sys
> >>> print sys.path
> ‘’, ‘/usr/lib/python2.7/site-packages/setuptools-2.0.1-py2.7.egg’, ‘/srv/www/cgi-bin’, ‘/srv/www/cgi-bin/PYTHONPATH’, ‘/usr/lib/python27.zip’, ‘/usr/lib64/python2.7’, ‘/usr/lib64/python2.7/plat-linux2’, ‘/usr/lib64/python2.7/lib-tk’, ‘/usr/lib64/python2.7/lib-old’, ‘/usr/lib64/python2.7/lib-dynload’, ‘/usr/lib64/python2.7/site-packages’, ‘/usr/local/lib64/python2.7/site-packages’, ‘/usr/local/lib/python2.7/site-packages’, ‘/usr/lib64/python2.7/site-packages/gtk-2.0’, ‘/usr/lib/python2.7/site-packages’]
> >>>
>
> --------------------
>
>
> - /srv/www/cgi-bin is present, but then so is
> /srv/www/cgi-bin/PYTHONPATH which means I broke something somewhere.

It might worth checking to see if this is an issue with Python 3.3 or above. I wonder if this is something to do with
some issues in Python 2.7, where Python doesn’t properly treat the directory paths as containing packages without at
least an empty init.py file. You may find the following documentation helpful:

http://docs.python.org/2/tutorial/modules.html#packages

#17

Worth a try, so I put Python3 on - but the module won’t import under python3’s command line. Doesn’t surprise me, as Ruckus were quite insistent on python 2.7.x

Cheers,
James

#18

jhd10374 wrote:
>>> gatekeeper:/srv/www/cgi-bin # python
>>> Python 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC] on linux2
>>> Type “help”, “copyright”, “credits” or “license” for more
>> information.
>>> >>> import sys
>>> >>> print sys.path
>>> ‘’, ‘/usr/lib/python2.7/site-packages/setuptools-2.0.1-py2.7.egg’,
>> ‘/srv/www/cgi-bin’, ‘/srv/www/cgi-bin/PYTHONPATH’,
>> ‘/usr/lib/python27.zip’, ‘/usr/lib64/python2.7’,
>> ‘/usr/lib64/python2.7/plat-linux2’, ‘/usr/lib64/python2.7/lib-tk’,
>> ‘/usr/lib64/python2.7/lib-old’, ‘/usr/lib64/python2.7/lib-dynload’,
>> ‘/usr/lib64/python2.7/site-packages’,
>> ‘/usr/local/lib64/python2.7/site-packages’,
>> ‘/usr/local/lib/python2.7/site-packages’,
>> ‘/usr/lib64/python2.7/site-packages/gtk-2.0’,
>> ‘/usr/lib/python2.7/site-packages’]

What does Python do with an empty string at the start of sys.path?

#19

On 2014-01-29, Dave Howorth <djh-novell@no-mx.forums.opensuse.org> wrote:
> What does Python do with an empty string at the start of sys.path?

It tells Python to look first for packages in the current directory.

#20

On 2014-01-29, jhd10374 <jhd10374@no-mx.forums.opensuse.org> wrote:
> Worth a try, so I put Python3 on - but the module won’t import under
> python3’s command line. Doesn’t surprise me, as Ruckus were quite
> insistent on python 2.7.x

In which case try this (may need sudoing):


sh-4.2 touch /srv/www/cgi-bin/__init__.py

And try running again inside Python 2.7.3.