I was looking for a way to update all packages installed using pip and found this. For some time I have been using it and although it used to throw some error messages the programs seem to work fine. But today I noticed that one of the installed programs (streamlink) stopped to work:
Then I ran as root:
Quite a few errors as you can see.
How can I fix this mess please? (my Leap 42.3 system is up to date, zypper-wise)
Don’t have a real solution, but AFAICS the only culprit is urllib3, but the weird thing is that 1.23 meets the >=1.21.1
Where ( from which repo ) did you install the python* packages? If not from the distro, did you do a vendor change on that repo ?
I tried to remove urlllib3 using ‘pip uninstall’, then installed python3-urllib3 using zypper:
# zypper in python3-urllib3
Loading repository data...
Reading installed packages...
Resolving package dependencies...
The following NEW package is going to be installed:
python3-urllib3
1 new package to install.
Overall download size: 138.8 KiB. Already cached: 0 B. After the operation, additional 530.2 KiB will be used.
Continue? [y/n/...? shows all options] (y): y
Retrieving package python3-urllib3-1.19.1-1.1.noarch (1/1), 138.8 KiB (530.2 KiB unpacked)
Retrieving: python3-urllib3-1.19.1-1.1.noarch.rpm ........................................................[done]
Checking for file conflicts: .............................................................................[done]
(1/1) Installing: python3-urllib3-1.19.1-1.1.noarch ......................................................[done]
#
Then I ran again the command which updates all the pip packages and I got the same error. Also at the end of it ‘pip list’ shows again urllib 1.23.
Where ( from which repo ) did you install the python* packages? If not from the distro, did you do a vendor change on that repo ?
I have no idea how pip installs things to be honest. How can I check? As for zypper the list is:
# zypper lr --uri
Repository priorities are without effect. All enabled repositories share the same priority.
# | Alias | Name | Enabled | GPG Check | Refresh | URI
---+-------------------------------------+-------------------------+---------+-----------+---------+--------------------------------------------------------------------------------------
1 | *Src-Non-OSS | *Src (Non-OSS) | No | ---- | ---- | http://download.opensuse.org/source/distribution/leap/42.3/repo/non-oss/
2 | KDE:Extra | KDE:Extra | Yes | (r ) Yes | No | http://download.opensuse.org/repositories/KDE:/Extra/openSUSE_Leap_42.3/
3 | RawTherapee | RawTherapee | Yes | (r ) Yes | No | http://download.opensuse.org/repositories/home:/rawtherapee/openSUSE_Leap_42.3/
4 | download.nvidia.com-leap | nVidia Graphics Drivers | Yes | (r ) Yes | Yes | http://download.nvidia.com/opensuse/leap/42.3
5 | download.opensuse.org-non-oss | *Main (Non-OSS) | Yes | (r ) Yes | Yes | http://download.opensuse.org/distribution/leap/42.3/repo/non-oss/
6 | download.opensuse.org-non-oss_1 | *Update (Non-OSS) | Yes | (r ) Yes | Yes | http://download.opensuse.org/update/leap/42.3/non-oss/
7 | download.opensuse.org-oss | *Main (OSS) | Yes | (r ) Yes | Yes | http://download.opensuse.org/distribution/leap/42.3/repo/oss/
8 | download.opensuse.org-oss_1 | *Update (OSS) | Yes | (r ) Yes | Yes | http://download.opensuse.org/update/leap/42.3/oss/
9 | ftp.gwdg.de-suse | Packman | Yes | (r ) Yes | Yes | http://packman.inode.at/suse/openSUSE_Leap_42.3/
10 | http-download.opensuse.org-ab86e53a | server:mail | Yes | (r ) Yes | Yes | http://download.opensuse.org/repositories/server:/mail/openSUSE_Leap_42.3/
11 | repo-debug | *Debug (OSS) | Yes | (r ) Yes | Yes | http://download.opensuse.org/debug/distribution/leap/42.3/repo/oss/
12 | repo-debug-non-oss | *Debug (Non-OSS) | No | ---- | ---- | http://download.opensuse.org/debug/distribution/leap/42.3/repo/non-oss/
13 | repo-debug-update | *Debug-Update (OSS) | No | ---- | ---- | http://download.opensuse.org/debug/update/leap/42.3/oss/
14 | repo-debug-update-non-oss | *Debug-Update (Non-OSS) | No | ---- | ---- | http://download.opensuse.org/debug/update/leap/42.3/non-oss/
15 | repo-source | *Src (OSS) | No | ---- | ---- | http://download.opensuse.org/source/distribution/leap/42.3/repo/oss/
16 | server:php:extensions | server:php:extensions | Yes | (r ) Yes | No | http://download.opensuse.org/repositories/server:/php:/extensions/openSUSE_Leap_42.3/
I just found:
# pip show urllib3
Name: urllib3
Version: 1.23
Summary: HTTP library with thread-safe connection pooling, file post, and more.
Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: andrey.petrov@shazow.net
License: MIT
Location: /usr/lib/python3.4/site-packages
Requires:
Required-by: requests
I hope this provides the feedback you need.
Tried to update pip itself (worked) and then to update all pip packages (no errors this time). Strangely - now the list of packages is much much longer:
And streamlink still throws the same errors.
Then I ran again the command updating all packages - new errors appeared. Final result - streamlink still doesn’t work, now even ‘pip list’ is broken (as well as the pip command itself):
I think I found a way to clean up the mess.
Force reinstalled python(3)-pip from yast,
updated it through ‘pip install --upgrade pip’
removed all pip packages which could be removed, then installed only those which I need.
Lesson learned: Obviously one needs to be careful with commands which update things.
On 2018-06-07, heyjoe <heyjoe@no-mx.forums.microfocus.com> wrote:
> Lesson learned: Obviously one needs to be careful with commands which
> update things.
Glad things are working for you. However, as a long-time Python programmer I’m going to suggest a different solution.
Never PIP-install anything as root. While PIP installations can be convenient just to get the job done, they can wreak
havoc with GNU/Linux package managers especially when it comes to upgrading. If you can’t install the necessary package
with YaST/zypper and you need to pip install, run this as user:
user $ pip install --user <package>
This will install the package in the user’s ~/.local/ directory. Also consider virtualenv (although I don’t use it).
@flymail - thanks for the info.
The reason I am installing as root is because I was running into issues previously when trying as normal user and I was told that ‘pip install’ must be run as root.
Now you say the opposite. Perhaps both methods have pros and cons. As a non-python programmer I can only think of: doing less things as root makes sense. However I don’t like packages to be installed in ~/.local because I try to keep my daily backups of my actual data in $HOME and .local is backed up too. I don’t want to backup software which I have not written myself (i.e. downloadable scritps/binaries by others). Can you recommend a method for installing all the pip packages in a particular directory, for example ~/pip? Then I can set a rule for not backing that up. I was reading through ‘pip help install’ but can’t figure the right option for it.
I noticed that now torbrowser-launcher doesn’t work. I tried forcefully updating it through yast but that changed nothing:
~]: torbrowser-launcher
Traceback (most recent call last):
File "/usr/bin/torbrowser-launcher", line 29, in <module>
import torbrowser_launcher
File "/usr/lib/python2.7/site-packages/torbrowser_launcher/__init__.py", line 33, in <module>
from .common import Common, SHARE
File "/usr/lib/python2.7/site-packages/torbrowser_launcher/common.py", line 55, in <module>
from twisted.internet import gtk2reactor
File "/usr/lib64/python2.7/site-packages/twisted/internet/gtk2reactor.py", line 23, in <module>
from twisted.internet import _glibbase
File "/usr/lib64/python2.7/site-packages/twisted/internet/_glibbase.py", line 20, in <module>
from twisted.internet import base, posixbase, selectreactor
File "/usr/lib64/python2.7/site-packages/twisted/internet/posixbase.py", line 18, in <module>
from twisted.internet import error, udp, tcp
File "/usr/lib64/python2.7/site-packages/twisted/internet/tcp.py", line 28, in <module>
from twisted.internet._newtls import (
File "/usr/lib64/python2.7/site-packages/twisted/internet/_newtls.py", line 21, in <module>
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
File "/usr/lib64/python2.7/site-packages/twisted/protocols/tls.py", line 41, in <module>
from OpenSSL.SSL import Error, ZeroReturnError, WantReadError
File "/usr/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import rand, crypto, SSL
File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 124, in <module>
SSL_ST_INIT = _lib.SSL_ST_INIT
AttributeError: 'module' object has no attribute 'SSL_ST_INIT'
Any idea how to fix that?
On 2018-06-07, heyjoe <heyjoe@no-mx.forums.microfocus.com> wrote:
> @flymail - thanks for the info.
Pleasure.
> The reason I am installing as root is because I was running into issues
> previously when trying as normal user and I was ‘told’
> (http://tinyurl.com/yctcbay8) that ‘pip install’ must be run as root.
… which kinda puts me in a difficult situation because Malcolm Lewis knows a lot more than me. I have used PIP on many
different GNU/Linux systems and know from experience that PIP installation as root has lead to many troubles. Perhaps
openSUSE is more resistant to these troubles for reasons I am unaware.
> However I don’t like packages to be installed in ~/.local
> because I try to keep my daily backups of my actual data in $HOME and
> .local is backed up too. I don’t want to backup software which I have
> not written myself (i.e. downloadable scritps/binaries by others).
I think this is laudable, but bear in mind all your $HOME/.directories are also being backed up which include scripts,
temporary files, and configuration files which you did not write yourself. I think the best solution to this approach is
to change your back up system so it backs-up everything except $HOME/.* .
> Can you recommend a method for installing all the pip packages in a
> particular directory, for example ~/pip? Then I can set a rule for not
> backing that up. I was reading through ‘pip help install’ but can’t
> figure the right option for it.
I would probably recommend a different approach entirely if you want to avoid installing packages to $HOME along the
following lines:
$ su -
# mkdir -p /opt/<user>/site-packages
# chown -hR <user> /opt/<user>/site-packages
# exit
$ mkdir -p ~/.local/lib/python3.4/
$ cd ~/.local/lib/python3.4/
$ ln -s /opt/<user>/site-packages
$ ls -l
$ pip install --user <package>
$ ls -lrth site-packages
$ python3
>>> import <package>
>>> exit()
$ cd ~/
… so in effect symlink to /opt/<user>/site-packages from $HOME.
But I would still prefer the approach I’ve suggested above of backing up everything apart from $HOME/.* …
Thanks flymail.
I use custom made regex expressions in my Bacula rules which allows me to avoid what doesn’t need backup (caches etc). But $HOME/.* in general needs backup because it contains lots of configs etc. So all I really need is one more rule, considering a custom directory for pip packages.
Thanks also for the suggested alternative approach.
Questions:
On 2018-06-08, heyjoe <heyjoe@no-mx.forums.microfocus.com> wrote:
> - What is t he reason to suggest /opt in particular? What purpose does
> this system directory serve?
According to the FHS, the /opt/ directory is for optional software packages. In practice, this means it is convenient
place to put 3rd part applications that are not part of the default OS or managed by the OS package manager.
> - Would it be equivalent to simply exclude from backup ~/.local/lib
> (which I currently don’t have at all) or ~/.local/lib/python*?
Yes.
> - Any idea how to fix torbrowser-launcher? (I suppose it needs some pip
> packages but i don’t know how to find out which)
Sorry, no - I don’t know that package.
Thanks for the info and reference. So far I have always thought that I should use /usr/local for this. I suppose /opt is better.
torbrowser-launcher is part of the Leap distro. I was rather wondering how do find out which particular pip packages it needs (which I guess is the reason for which it complains).
On 2018-06-08, heyjoe <heyjoe@no-mx.forums.microfocus.com> wrote:
> Thanks for the info and reference. So far I have always thought that I
> should use /usr/local for this. I suppose /opt is better.
The conventions of deciding between /usr/local/ and /opt/ are confusing and inconsistent. Many 3rd party binary
installers decide to install into one or the other with little control from root. However in your case, you are looking
to install packages at the user-level rather than at root-level. Since /opt/ and /usr/ can in principle be separate
partitions with different security policies, I’d much prefer to install user-level packages in /opt/ than /usr/local/.
However, by far the safest approach is to install user-level packages in $HOME/.local/ and this is my normal practice.
It’s more than worth doing this whatever changes this would entail to your backup system.
> torbrowser-launcher is part of the Leap distro. I was rather wondering
> how do find out which particular pip packages it needs (which I guess is
> the reason for which it complains).
I’d check the GitHub/source for the requirements. If they are not properly listed, then I’d avoid this package like the
plague.
Thanks for the added explanations.
I’d check the GitHub/source for the requirements. If they are not properly listed, then I’d avoid this package like the
plague.
Checked the GitHub url given in YaST. I see dependencies listed only in build instructions. This is very strange because even with the same ‘pip list’ output which I had at the beginning of this thread (when torbrowser-launcher worked fine) now it does not. Weird.
After seeing that virt-manager stopped working too I filed 2 bug reports which were closed .
But I have just found a fix for the errors which:
All errors are gone now. Before doing that the output of ‘pip list’ was:
Package Version
---------------- ---------
apparmor 2.10.3
avideo 2017.9.27
bottle 0.12.8
certifi 2018.4.16
chardet 3.0.4
cupshelpers 1.0
idna 2.6
iso-639 0.4.5
iso3166 0.8
isodate 0.6.0
LibAppArmor 2.10.3
numpy 1.9.3
pip 10.0.1
pycryptodome 3.6.1
pycups 1.9.72
pycurl 7.19.5.1
pygobject 3.20.1
pysmbc 1.0.15.4
PySocks 1.6.8
PyYAML 3.12
requests 2.18.4
setuptools 39.2.0
six 1.11.0
speedtest-cli 2.0.2
streamlink 0.13.0
urllib3 1.22
websocket-client 0.48.0
After the fix the list is:
apparmor (2.10.3)
avideo (2017.9.27)
bottle (0.12.8)
certifi (2018.4.16)
chardet (3.0.4)
coverage (4.3.4)
cupshelpers (1.0)
idna (2.6)
iso-639 (0.4.5)
iso3166 (0.8)
isodate (0.6.0)
LibAppArmor (2.10.3)
nose (1.3.7)
numpy (1.9.3)
Pillow (2.9.0)
pip (7.1.2)
py (1.4.31)
pycryptodome (3.6.1)
pycups (1.9.72)
pycurl (7.19.5.1)
pygobject (3.20.1)
pysmbc (1.0.15.4)
PySocks (1.6.8)
pyxdg (0.25)
PyYAML (3.12)
requests (2.18.4)
setuptools (39.2.0)
simplejson (3.8.2)
six (1.11.0)
speedtest-cli (2.0.2)
streamlink (0.13.0)
urllib3 (1.22)
websocket-client (0.48.0)
And everything works as expected.
This implies it is possible to uninstall some necessary pip packages (dependencies) which have been installed previously through yast. Yet for others it is not possible (as explained earlier, because pip was saying they were part of distutils).
In other words - to avoid breaking the system indeed it seems more appropriate not to run pip commands as root regardless of the previous advice given in another thread.
On 2018-06-11, heyjoe <heyjoe@no-mx.forums.microfocus.com> wrote:
> This implies it is possible to uninstall some necessary pip packages
> (dependencies) which have been installed previously through yast. Yet
> for others it is not possible (as explained earlier, because pip was
> saying they were part of distutils).
Packages installed by YaST should be uninstalled by YaST. PIP-installed site-packages can be just deleted when they are
not required
>
> In other words - to avoid breaking the system indeed it seems more
> appropriate not to run pip commands as root regardless of the previous
> advice given in another thread.
My general advice for installing Python packages is as follows:
There are some Python packages that challenge this model (e.g. TensorFlow) and in some cases can never be installed
in many configurations. In these cases, it is usually a good idea to install a separate GNU/Linux distribution in
parallel just to handle these packages without the pain of having to restore Karma between packages.
Sure. The problem is that it is possible to delete packages installed by YaST through ‘pip uninstall’ and that seems to have created my problem. IOW pip can break system integrity. Not sure if this should be considered a bug.
My general advice for installing Python packages is as follows:
Thanks for sharing. So far I have been doing exactly this except that 3. was done as root.
On 2018-06-11, heyjoe <heyjoe@no-mx.forums.microfocus.com> wrote:
> Thanks for sharing. So far I have been doing exactly this except that 3.
> was done as root.
Perhaps one of the most useful tricks is upgrading pip without interfering with the YaST’s package-managed version
(which is usually out of date):
$ pip3 install --user --upgrade pip
Got your reply in an email notification but strangely your post is not here:
Perhaps one of the most useful tricks is upgrading pip without interfering with the YaST’s package-managed version
(which is usually out of date):Code:
$ pip3 install --user --upgrade pip
I did that already and pip got updated to 10.0.1 but strangely when I run ‘pip -V’ it still shows me 7.1.2. The only workaround I have found is ‘~/.local/bin/pip list’. Still this is strange because ~/.local/bin/ is in my $PATH (at its end). Should I try putting it in the beginning? Any danger from doing so? (potentially a local user script being called instead a system one?)