I am using *Python 3.13* in a virtual environment under *Ubuntu Linux 24.04*
..
The version of Python was compiled from source code and installed with make altinstall. I attempted to use *pip* to install the *Sphinx* package into
the virtual environment using the command *pip install sphinx* in the
virtual environment*.* I expected that *sphinx* would be installed in the *site-packages* directory in the virtual environment. Instead, it was
installed into the site-packages directory in */home/jonathan/.locals/lib/python3.13/site-packages* even though I did not specify *--user* to the *pip install* command. Is this expected behavior? I wanted Sphinx to be installed in the virtual environment so that it would
be accessible to all users of the virtual environment.


--
Jonathan Gossage

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 14/04/25 11:10, Jonathan Gossage via Python-list wrote:

I am using *Python 3.13* in a virtual environment under *Ubuntu Linux 24.04* .
The version of Python was compiled from source code and installed with make altinstall. I attempted to use *pip* to install the *Sphinx* package into
the virtual environment using the command *pip install sphinx* in the
virtual environment*.* I expected that *sphinx* would be installed in the *site-packages* directory in the virtual environment. Instead, it was installed into the site-packages directory in */home/jonathan/.locals/lib/python3.13/site-packages* even though I did not specify *--user* to the *pip install* command. Is this expected behavior? I wanted Sphinx to be installed in the virtual environment so that it would
be accessible to all users of the virtual environment.

Which command did you type?
Which type of virtual-environment, and was it activated at the time?

--
Regards,
=dn


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: DWM (3:633/280.2@fidonet)

Please include the group in your response; don't just send it to me.

On 4/14/2025 5:09 AM, Jonathan Gossage wrote:

The virtual environment was owned by the user running pip. It was not
owned by root. Does pip not support virtual environments that are owned
by a non-root user and have a multi-user secondary group? The actual
command was *pip install mypy flake8 sphinx*. The other packages were
also installed into the user .local tree and work properly for the user doing the installation, but not for other members of the group that have access to the virtual environment.

Pip doesn't know about the environment it runs in. It seems to me that
you didn't active the venv before you installed using pip. So nothing
would have gotten installed into the venv. So where is the venv that you
set up? I usually put them into ~/venv. For example, a venv named "gf4"
is at ~/venv/gf4.

To activate a venv, you have to source its activate script, which is in
the venv. First you have to mark it as executable. Then you source it -

source ~/venv/gf4/bin/activate

Now when you run python (or more likely, python3), it will find the
venv's directories before it will find the system's or user's. You know
the activation has been successful because the prompt changes to show
you. The activation applies to the terminal session in which you
activated the venv.

On Sun, Apr 13, 2025 at 10:11 PM Thomas Passin <list1@tompassin.net <mailto:list1@tompassin.net>> wrote:

On 4/13/2025 7:10 PM, Jonathan Gossage via Python-list wrote:
> I am using *Python 3.13* in a virtual environment under *Ubuntu
Linux 24.04*
> .
> The version of Python was compiled from source code and installed
with make
> altinstall. I attempted to use *pip* to install the *Sphinx*
package into
> the virtual environment using the command *pip install sphinx* in the
> virtual environment*.* I expected that *sphinx* would be
installed in the
> *site-packages* directory in the virtual environment. Instead, it was
> installed into the site-packages directory in
> */home/jonathan/.locals/lib/python3.13/site-packages* even though
I did not
> specify *--user* to the *pip install* command. Is this expected
behavior? I
> wanted Sphinx to be installed in the virtual environment so that
it would
> be accessible to all users of the virtual environment.

If you ran the command as a user, then pip would not have root
privileges and could not install into the system directory. It would
fall back to installing into the user's tree. It usually prints a
message to that effect. That's standard behavior if you don't have the
venv activated.

If you want to install something into a virtual environment, you
have to
activate the environment before installing the something.

A complication could occur if the system's Python version is the
same as
the one you built. You might inadvertently run the system's binary of
python 3.13 instead of your own. I'm not familiar with the make
altinstall command but I doubt that it changes the ordinary rules for
invoking python and using a venv.

--
Jonathan Gossage

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

Thomas Passin <list1@tompassin.net> writes:
[...]

To activate a venv, you have to source its activate script, which is
in the venv. First you have to mark it as executable. Then you source
it -

source ~/venv/gf4/bin/activate

[...]

No, you don't have to (and probably shouldn't) mark the script as
executable.

Making a script executable (chmod +x) is required before *executing* it,
but when you *source* a script (using "source" or "."), your current
shell reads it and evaluates its content.

Making the active script executable introdues the risk that you'll
accidentally execute it rather than sourcing it. If you do that, it
will probably set up the environment in a new shell process which then immediately terminates.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: None to speak of (3:633/280.2@fidonet)

rbowman <bowman@montana.com> writes:

On Mon, 14 Apr 2025 09:55:09 -0400, Thomas Passin wrote:

Pip doesn't know about the environment it runs in. It seems to me that
you didn't active the venv before you installed using pip. So nothing
would have gotten installed into the venv. So where is the venv that you
set up? I usually put them into ~/venv. For example, a venv named "gf4"
is at ~/venv/gf4.

Are you sure about that?

Sure about what?

activate has

VIRTUAL_ENV="/home/rbowman/work/python/weather"
export VIRTUAL_ENV

[...]

The activate script is created when you create the venv (using something
like `python3 -m venv /path/to/new/venv`), and it's customized with the location of the venv.

If Thomas creates a venv in ~/venv/gf4, that's what will appear in his
venv's active script (with the ~ expanded to the path of his home
directory).

(I'm relatively new at this. Please let me know if I've gotten any of
the details wrong.)

[...]

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: None to speak of (3:633/280.2@fidonet)

On 4/14/2025 6:20 PM, Keith Thompson via Python-list wrote:

Thomas Passin <list1@tompassin.net> writes:
[...]

To activate a venv, you have to source its activate script, which is
in the venv. First you have to mark it as executable. Then you source
it -

source ~/venv/gf4/bin/activate

[...]

No, you don't have to (and probably shouldn't) mark the script as
executable.

Making a script executable (chmod +x) is required before *executing* it,
but when you *source* a script (using "source" or "."), your current
shell reads it and evaluates its content.

Making the active script executable introdues the risk that you'll accidentally execute it rather than sourcing it. If you do that, it
will probably set up the environment in a new shell process which then immediately terminates.

You are right, my bad. When I went to check on what the venv prompt
looked like after activation, I hadn't run my Linux VM for too long and
forgot that the activate script needs to be sourced - in Windows it just
gets run as any other script. I noticed it wasn't marked executable and blindly "fixed" that. Then of course I remembered the script has to be sourced - and forgot to undo the execute flag.


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 4/14/2025 6:20 PM, Keith Thompson via Python-list wrote:

Thomas Passin <list1@tompassin.net> writes:
[...]

To activate a venv, you have to source its activate script, which is
in the venv. First you have to mark it as executable. Then you source
it -

source ~/venv/gf4/bin/activate

[...]

No, you don't have to (and probably shouldn't) mark the script as
executable.

Making a script executable (chmod +x) is required before *executing* it,
but when you *source* a script (using "source" or "."), your current
shell reads it and evaluates its content.

Making the active script executable introdues the risk that you'll accidentally execute it rather than sourcing it. If you do that, it
will probably set up the environment in a new shell process which then immediately terminates.

You are right, my bad. When I went to check on what the venv prompt
looked like after activation, I hadn't run my Linux VM for too long and
forgot that the activate script needs to be sourced - in Windows it just
gets run as any other script. I noticed it wasn't marked executable and blindly "fixed" that. Then of course I remembered the script has to be sourced - and forgot to undo the execute flag.


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

Am Mo., 14. Apr. 2025 um 01:14 Uhr schrieb Jonathan Gossage via Python-list <python-list@python.org>:

I am using *Python 3.13* in a virtual environment under *Ubuntu Linux
24.04*
.
[...]

Instead, it was
installed into the site-packages directory in */home/jonathan/.locals/lib/python3.13/site-packages* even though I did not specify *--user* to the *pip install* command. Is this expected behavior? I wanted Sphinx to be installed in the virtual environment so that it would
be accessible to all users of the virtual environment.

Hi Jonathan,

Many people put emphasis on that you need to *activate* a virtualenv before using it, but no-one so far stressed the fact that you got Sphinx installed
to ~/jonathan/.local/lib/python3.13/site-packages *without using *--user.

Do you have, by any chance, one of the following pip-related configuration files present?

- /etc/pip.conf
- ~/.config/pip/pip.conf
- ~/.pip/pip.conf (lagacy)

I took this list from https://pip.pypa.io/en/stable/topics/configuration/.
*If* there is any such file present, you might look out for a line:

user = true

but I am not entirely certain about this.

Looking forward to your reply!

Best,
Friedrich

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 4/15/25 10:43, Friedrich Romstedt via Python-list wrote:

Many people put emphasis on that you need to *activate* a virtualenv before using it, but no-one so far stressed the fact that you got Sphinx installed to ~/jonathan/.local/lib/python3.13/site-packages *without using *--user.

To be clear: you do not have to activate a virtualenv to use *Python*
from it. If you just call the python by the path it's in, it figures everything out (and sets some variables you can query vi sysconfig if
you have reason to actually need those details (look for installation schemes).

What activating gives you is some path fiddling, and some prompt
fiddling (although the prompt fiddling keeps saying it's deprecated).
The latter is a visual clue; the former means you can also find *other* commands installed in the virtualenv - including pip.

/path/to/virtualenv//bin/python -m pip install ... will work whether
you activated or not.

pip install ... finds the first command in your PATH named pip, which
may or may not be the one in the virtualenv, *unless* you've activated
it, because that's the only way the virtualenv bin directory ends up
early in your path.

Meanwhile - the install to ~/.local : some distros just default to that behavior whether you ask for it or not, because they consider it unsafe
to install in the system location. Some have chosen to yell at you even
if you try a "user install" with the system Python. Up to them what they
do...



--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

Various responses have been provided but the OP has not yet replied
on-list (as verified by Archive). Is this an error with the
list-processor or have some posts been sent to one person only (using
Reply instead of ReplyList)?

There are always others who would like to learn from list-discussions -
but will hearing only half of some of the conversation help them?


On 14/04/25 11:33, dn via Python-list wrote:

On 14/04/25 11:10, Jonathan Gossage via Python-list wrote:

I am using *Python 3.13* in a virtual environment under *Ubuntu Linux
24.04*
.
The version of Python was compiled from source code and installed with
make
altinstall. I attempted to use *pip* to install the *Sphinx* package into
the virtual environment using the command *pip install sphinx* in the
virtual environment*.* I expected that *sphinx* would be installed in the
*site-packages* directory in the virtual environment. Instead, it was
installed into the site-packages directory in
*/home/jonathan/.locals/lib/python3.13/site-packages* even though I
did not
specify *--user* to the *pip install* command. Is this expected
behavior? I
wanted Sphinx to be installed in the virtual environment so that it would
be accessible to all users of the virtual environment.

Which command did you type?
Which type of virtual-environment, and was it activated at the time?

--
Regards,
=dn


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: DWM (3:633/280.2@fidonet)

On 4/15/2025 12:43 PM, Friedrich Romstedt via Python-list wrote:

Am Mo., 14. Apr. 2025 um 01:14 Uhr schrieb Jonathan Gossage via Python-list <python-list@python.org>:

I am using *Python 3.13* in a virtual environment under *Ubuntu Linux
24.04*
.
[...]

Instead, it was
installed into the site-packages directory in
*/home/jonathan/.locals/lib/python3.13/site-packages* even though I did not >> specify *--user* to the *pip install* command. Is this expected behavior? I >> wanted Sphinx to be installed in the virtual environment so that it would
be accessible to all users of the virtual environment.

Hi Jonathan,

Many people put emphasis on that you need to *activate* a virtualenv before using it, but no-one so far stressed the fact that you got Sphinx installed to ~/jonathan/.local/lib/python3.13/site-packages *without using *--user.

On Linux, at least, it's standard for pip to install into the user's site-packages location if it's not invoked with admin privileges - even without --user. Pip will emit a message saying so. Well, that used to be
true but nowadays Pip wants you to use the --break-system-packages flag
if you want to insist on installing into the system's Python install,
even if it's going to go into --user. I'm not sure if the restriction
will be in place given that the OP built his own Python version.

Do you have, by any chance, one of the following pip-related configuration files present?

- /etc/pip.conf
- ~/.config/pip/pip.conf
- ~/.pip/pip.conf (lagacy)

I took this list from https://pip.pypa.io/en/stable/topics/configuration/. *If* there is any such file present, you might look out for a line:

user = true

but I am not entirely certain about this.

Looking forward to your reply!

Best,
Friedrich

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 4/15/2025 3:41 PM, dn via Python-list wrote:

Various responses have been provided but the OP has not yet replied on-
list (as verified by Archive). Is this an error with the list-processor
or have some posts been sent to one person only (using Reply instead of ReplyList)?

There are always others who would like to learn from list-discussions -
but will hearing only half of some of the conversation help them?

The OP sent some replies to me and I asked them to include the list.

On 14/04/25 11:33, dn via Python-list wrote:

On 14/04/25 11:10, Jonathan Gossage via Python-list wrote:

I am using *Python 3.13* in a virtual environment under *Ubuntu Linux
24.04*
.
The version of Python was compiled from source code and installed
with make
altinstall. I attempted to use *pip* to install the *Sphinx* package
into
the virtual environment using the command *pip install sphinx* in the
virtual environment*.* I expected that *sphinx* would be installed in
the
*site-packages* directory in the virtual environment. Instead, it was
installed into the site-packages directory in
*/home/jonathan/.locals/lib/python3.13/site-packages* even though I
did not
specify *--user* to the *pip install* command. Is this expected
behavior? I
wanted Sphinx to be installed in the virtual environment so that it
would
be accessible to all users of the virtual environment.

Which command did you type?
Which type of virtual-environment, and was it activated at the time?

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 2025-04-15, Thomas Passin via Python-list <python-list@python.org> wrote:

On Linux, at least, it's standard for pip to install into the user's site-packages location if it's not invoked with admin privileges - even without --user. Pip will emit a message saying so. Well, that used to be true but nowadays Pip wants you to use the --break-system-packages flag
if you want to insist on installing into the system's Python install,
even if it's going to go into --user.

I've always been a little baffled by that message when installing with
--user. How can that possibly break system stuff?

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 4/15/25 16:07, Grant Edwards via Python-list wrote:

On 2025-04-15, Thomas Passin via Python-list <python-list@python.org> wrote:

On Linux, at least, it's standard for pip to install into the user's
site-packages location if it's not invoked with admin privileges - even
without --user. Pip will emit a message saying so. Well, that used to be
true but nowadays Pip wants you to use the --break-system-packages flag
if you want to insist on installing into the system's Python install,
even if it's going to go into --user.

I've always been a little baffled by that message when installing with --user. How can that possibly break system stuff?

Your user install dir is in your python path, so when you go to run an installed Python program which imports other packages, it might pick up
the version you have in your user space rather than the system one it
was tested with. It's about a whole curated Python environment that the distro spends time validating - a different version picked up elsewhere *might* be fine. Or it might not. I think they've had enough "it's not"
bug reports to have gotten quite prickly about the subject.


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 2025-04-16, Mats Wichmann via Python-list <python-list@python.org> wrote:

On 4/15/25 16:07, Grant Edwards via Python-list wrote:

On 2025-04-15, Thomas Passin via Python-list <python-list@python.org> wrote: >>

On Linux, at least, it's standard for pip to install into the user's
site-packages location if it's not invoked with admin privileges - even
without --user. Pip will emit a message saying so. Well, that used to be >>> true but nowadays Pip wants you to use the --break-system-packages flag
if you want to insist on installing into the system's Python install,
even if it's going to go into --user.

I've always been a little baffled by that message when installing with
--user. How can that possibly break system stuff?

Your user install dir is in your python path, so when you go to run an installed Python program which imports other packages, it might pick up
the version you have in your user space rather than the system one it
was tested with.

Yes, I understand that. But that's breaking stuff for the user not for
the system. Of course installing stuff for the user can break stuff
for the user. I don't need a warning to tell me that.

Also... when installing stuff with pip --user, it is always a package
that is not installed for the system (usually not even available for
the system). How can that "break system packages"?

--
Grant


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 4/15/2025 5:38 PM, rbowman via Python-list wrote:

On Tue, 15 Apr 2025 14:12:19 -0400, Thomas Passin wrote:

On Linux, at least, it's standard for pip to install into the user's
site-packages location if it's not invoked with admin privileges - even
without --user. Pip will emit a message saying so. Well, that used to be
true but nowadays Pip wants you to use the --break-system-packages flag
if you want to insist on installing into the system's Python install,
even if it's going to go into --user. I'm not sure if the restriction
will be in place given that the OP built his own Python version.

Is that pip or a distro's version of pip? On Fedora I get the message
about defaulting to user. On Ubuntu I get a message to use a venv or if I really want a global install to use something like 'pip install python3- black'. Ubuntu's is pip 24.2, Fedor's is 24.3.1 but neither of them show '--break-system-packages' in --help.

The behavior is specifed in

https://packaging.python.org/en/latest/specifications/externally-managed-environments/#externally-managed-environments

Exactly how pip works and what messages it emits are specified by this document, and the details depend on how the distro's packagers configure
it. For example, here is a bit of the spec:

"Before a Python-specific package installer (that is, a tool such as pip
- not an external tool such as apt) installs a package into a certain
Python context, it should make the following checks by default:

Is it running outside of a virtual environment? It can determine
this by whether sys.prefix == sys.base_prefix.

Is there an EXTERNALLY-MANAGED file in the directory identified by sysconfig.get_path("stdlib", sysconfig.get_default_scheme())?

If both of these conditions are true, the installer should exit with an
error message indicating that package installation into this Python interpreter’s directory are disabled outside of a virtual environment.

The installer should have a way for the user to override these rules,
such as a command-line flag --break-system-packages. This option should
not be enabled by default and should carry some connotation that its use
is risky."


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

Op 15/04/2025 om 20:31 schreef Mats Wichmann via Python-list:

To be clear: you do not have to activate a virtualenv to use *Python*
from it. If you just call the python by the path it's in, it figures everything out (and sets some variables you can query vi sysconfig if
you have reason to actually need those details (look for installation schemes).

What activating gives you is some path fiddling, and some prompt
fiddling (although the prompt fiddling keeps saying it's deprecated).
The latter is a visual clue; the former means you can also find
*other* commands installed in the virtualenv - including pip.

/path/to/virtualenv//bin/python -m pip install ...ÿÿ will work whether
you activated or not.

pip install ...ÿ finds the first command in your PATH named pip, which
may or may not be the one in the virtualenv, *unless* you've activated
it, because that's the only way the virtualenv bin directory ends up
early in your path.

And the pip command that is found and run will use it's own settings
regarding where to install packages, even if you activated a virtualenv.
For example, you can't use /usr/bin/pip to install something in a
virtualenv. To install something in a virtualenv, you need to use the
pip in that virtualenv (either by first activating that venv, or by
running something like venv/bin/pip, or venv). (Of course to do that pip
needs to be installed in that venv. That might or might not be the case depending on how the venv was created.)

I kinda get the feeling that something like that is going on here.

--
"There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be
replaced by something even more bizarre and inexplicable.
There is another theory which states that this has already happened."
-- Douglas Adams, The Restaurant at the End of the Universe


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 4/17/2025 4:58 AM, Roel Schroeven via Python-list wrote:

Op 15/04/2025 om 20:31 schreef Mats Wichmann via Python-list:

To be clear: you do not have to activate a virtualenv to use *Python*
from it. If you just call the python by the path it's in, it figures
everything out (and sets some variables you can query vi sysconfig if
you have reason to actually need those details (look for installation
schemes).

What activating gives you is some path fiddling, and some prompt
fiddling (although the prompt fiddling keeps saying it's deprecated).
The latter is a visual clue; the former means you can also find
*other* commands installed in the virtualenv - including pip.

/path/to/virtualenv//bin/python -m pip install ...ÿÿ will work whether
you activated or not.

pip install ...ÿ finds the first command in your PATH named pip, which
may or may not be the one in the virtualenv, *unless* you've activated
it, because that's the only way the virtualenv bin directory ends up
early in your path.

Or you can cd to the venv directory and it will be first on the pythonpath.

And the pip command that is found and run will use it's own settings regarding where to install packages, even if you activated a virtualenv.
For example, you can't use /usr/bin/pip to install something in a virtualenv.

pip detects if you are running in a venv and changes its behavior to match.

To install something in a virtualenv, you need to use the

pip in that virtualenv (either by first activating that venv, or by
running something like venv/bin/pip, or venv). (Of course to do that pip needs to be installed in that venv. That might or might not be the case depending on how the venv was created.)

I kinda get the feeling that something like that is going on here.

IMHO pip should always be run as a module, e.g., python3 -m pip. This
way you always get the same pip and environment as the python executable
that is being used.


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

Also... when installing stuff with pip --user, it is always a package
that is not installed for the system (usually not even available for
the system). How can that "break system packages"?

pip installs dependencies. Dependencies may disagree on the version
with the system packages.

This is a difference between eg. how conda works and pip. Conda is an
actual package manager: it ensures that all packages in a particular environment agree on version requirements. pip will break your
environment in subsequent installs because it doesn't keep track of
what was installed before.

On top of this, pip may, in general, cause any amount of damage to
your system regardless of where or how you install it because by
default it's allowed to build wheels from source packages. The build
may run whatever code, including formatting hard drives, mining
bitcoin etc. The reason it doesn't happen very often is that package maintainers kind of trust each other to be nice. There aren't really
any safeguards to prevent malicious actors from doing this, but you
would have to want to install their package for some reason.

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 2025-04-17, Left Right via Python-list <python-list@python.org> wrote:

Also... when installing stuff with pip --user, it is always a package
that is not installed for the system (usually not even available for
the system). How can that "break system packages"?

pip installs dependencies. Dependencies may disagree on the version
with the system packages.

Good point. I don't generally allow pip to install dependencies, so I
forgot about that source of conflict. But as far as I can see, it
still can only break stuff for the user, not for the system. The
system is never going to look in my ~/.local/lib/python* directories.

--
Grant


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 4/17/25 15:15, Grant Edwards via Python-list wrote:

On 2025-04-17, Left Right via Python-list <python-list@python.org> wrote:

Also... when installing stuff with pip --user, it is always a package
that is not installed for the system (usually not even available for
the system). How can that "break system packages"?

pip installs dependencies. Dependencies may disagree on the version
with the system packages.

Good point. I don't generally allow pip to install dependencies, so I
forgot about that source of conflict. But as far as I can see, it
still can only break stuff for the user, not for the system. The
system is never going to look in my ~/.local/lib/python* directories.

Sure, and it's "your fault" (sic) if you break yourself. You and I can
maybe accept that. But it's usually not obvious that you're done
anything to break stuff - you just asked to install something, according
to the instructions you read on the project's website, which it's
reasonable (?) to expect reflects best practices. Usually a simple "pip install".

Say you've been using "MLwiz" installed via a system package. It has
two dozen dependencies, some of which have further deps. And it works
great. Then you install "Chatbridge" via pip, and it brings with it its
own dependencies. It's not really useful to refuse to let it bring its
own deps with it since there may be version-specific requirements on its
deps, which that project has reasons for. So now you've got a few pkgs
in the system with different versions in your .local, and you can't run
MLwiz any longer, because a .local pkg (picked up first) conflicts with
the requirements of MLwiz.

There's just not a really great answer to this.

"Use a virtualenv". Fine, but that's still daunting to many, and not necessarily as well described as it could be, and the choice of the word "virtual" in the name makes it sound more complex than it actually is. Projects can certainly do better at describing installation scenarios -
at least if they have more than a trivial number of deps, where the
conflict chances go up rapidly.



--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the
user's .local packages in the first place. That should only be for
things the user has installed "for this user".

--
Greg

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

--vrzy7qk5nyvbbp5o
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

=20
Seems to me a system-installed application shouldn't be looking in the
user's .local packages in the first place. That should only be for things
the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say
that a system-installed Python shouldn't look into ~/.local, then you've
just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed scripts. This isn't as easy as checking whether the path starts with
/usr/bin or whether it belongs to root. Tying into the system's package
manager doesn't look appealing to me (not to mention that it might be unacceptably slow).

hjp

--=20
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

--vrzy7qk5nyvbbp5o
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmgCcgIACgkQ8g5IURL+ KF1crxAAohZV8Zn9PAMD+EmDnHFVqmp1CeDDLoMYcedcxWZhsDW5lBVJKr59c8yv jz6bUVZJdOvpeXOXz52GkUIV2mghqo2sI7TLYYNVd4y62mVHCglPytx1mCxYF1Aa zQrdbcJuPCYdS1Q5aENc2h+/KYcW+pii+dP1LaViWg+xb8Du09BYsY09Psr1ogRl r25/6IOPBXdl0GJXhjE3Er8DJCDlbaxgvMGK/5S3kceK3gvFIDWyp/ZctHmbZDdM kPsbeW5el3qMFfUs7Y+4Osw36kMi5/BcY9Ef0qNy0vfn0Mj4qmQUR0BlsLicByha 59obkuNEgHLTphLhlzPT0bPGaVISp81I2GCHUsueVM05JXPqyU0NegIi5QdLnKhS q9OpMtEQBYKvS3g+cH3ddxzrFY/FQkj5V3xsKQdnXCiWNl+3mo7hAmpWnrt8wf9T 4X+vuzYJY2hJslGiQlHiwO6UI66ObB+aNEhJHxLMJr8HNOUj1cXoH99Oc0UHltyN eyVFTL4zRvVi1mXnoZJGiCA1jL9Ssdm7JOT+cnf0d6ZMapEHqQ2W8gIs/CQIVLTV PHsgiuuPY9LS0XiOG90OVVCb5EDBt8hfhVBnozjQE5o+W1EJyNTncaePIpnXkBeo mxog3RB9/FCihbvRZJinTQNKgHvpfamDZHk/vMh01MtTJqiA/sU=
=MF8q
-----END PGP SIGNATURE-----

--vrzy7qk5nyvbbp5o--

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On Fri, 18 Apr 2025 at 16:50, Peter J. Holzer via Python-list <python-list@python.org> wrote:

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the user's .local packages in the first place. That should only be for things the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say
that a system-installed Python shouldn't look into ~/.local, then you've
just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed scripts. This isn't as easy as checking whether the path starts with
/usr/bin or whether it belongs to root. Tying into the system's package manager doesn't look appealing to me (not to mention that it might be unacceptably slow).

Couldn't the system-installed scripts have a shebang like:

#!/usr/bin/python3 -s

--
Oscar

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 4/18/2025 11:38 AM, Peter J. Holzer via Python-list wrote:

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the
user's .local packages in the first place. That should only be for things
the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say
that a system-installed Python shouldn't look into ~/.local, then you've
just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed scripts. This isn't as easy as checking whether the path starts with
/usr/bin or whether it belongs to root. Tying into the system's package manager doesn't look appealing to me (not to mention that it might be unacceptably slow).

Let's try a specific example. Let's say that PyQt6 v6.8.3 is installed
in the system site directory. The OS uses PyQt6 for some system
purposes. Now the user comes along and forces an install of PyQt6 v6.9.0
in the user site directory. 6.9.0 has a bug that would crash one of the system's application but not the user's programs. (This is not a
far-fetched scenario).

When the system launches its application the PYTHONPATH will start with
system site directories; local user site directories will be on the
PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3
because that will come first on the path. No crash here.

If the user has a program that actually does require the use of v6.9.0,
he's going to have to make sure that the user's local site directories
come first on the path. One way to do that is to set the PYTHONPATH to
point to the user's location.

In what scenario is a system application going to load a wrong version
of a dependency from a user's site location? The only one I can think of
is for the user, with the help of sudo, or by editing some
system-enabled script, were to change the global PYTHONPATH. That seems
a stretch.


--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

--ur5g4cjgqzkwwx4c
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2025-04-18 17:11:33 +0100, Oscar Benjamin via Python-list wrote:

On Fri, 18 Apr 2025 at 16:50, Peter J. Holzer via Python-list <python-list@python.org> wrote:

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the user's .local packages in the first place. That should only be for th=

ings

the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say that a system-installed Python shouldn't look into ~/.local, then you've just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed scripts. This isn't as easy as checking whether the path starts with /usr/bin or whether it belongs to root. Tying into the system's package manager doesn't look appealing to me (not to mention that it might be unacceptably slow).

=20
Couldn't the system-installed scripts have a shebang like:
=20
#!/usr/bin/python3 -s

Yes, that should work. At least I can't think of any downsides at the
moment.

hjp

--=20
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

--ur5g4cjgqzkwwx4c
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmgDYREACgkQ8g5IURL+ KF0z/xAAgIhbMqE4Ompkwiexp2e7jrFbcKcAoWUehTQc8QQpfynNxK1AGQ66XWXc 5bIjOOCZbp4XYpLfpIfd+cPlGYmaANTnRcBdWLNwrjF//uYx6HuvtBOqKZKGGq1e 5aQbh0+3gq4Ade4JKCQXWh1YQ8vF0ux66vSbRKNa7PAe9hKcrKaBe0pMrmstNfk8 caUE1oWdxvGvgx0waH1NOwhaJiiwhItTAAvo9xxcbgzBhbIizogwuYAzCJ+xkiP+ R4IPcwLz0lReTJSSnXprUDrh17mi4wv3aNXbdpxLB82Efm7ZxQd14/lBC4kSqlYL yS5NOujbj8Po/y+nzi9XGQHJVn4P9Naom8zAcqBalbInj4qu1Xtvw0MEMrLvvQL2 eoS26KYCXbDR+0jxV/kGhPqn2TTr/HbYvQoFxPZiH/tfd8v0xXQQarZ1Cd/Jm+Ho YPMvCR403/uSvb1ilvgBujlQNTaVkadI0aUkWBNlg3F4evRdstF2IcsA2Lnbu3Ns pXjsGKBQDCq02v4DLNIr0uge7Ia9NcnyAhYdRvEMv6pjcy5RsyepR3UAcfiNnjR/ KW1ysFKe1CDkDFxnAb6Ei6202ADA+DtQZIx9Qre3RBFMR7Vuh0SjlyYtYQMvuHDl Zzooid9mr4EPpjYuGTq9kxS8I+wMTPLrjwI+xcR+3OsbhZe4BjM=
=CIdO
-----END PGP SIGNATURE-----

--ur5g4cjgqzkwwx4c--

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

--vp5o7ne6zbzmsmpv
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2025-04-18 13:08:36 -0400, Thomas Passin via Python-list wrote:

On 4/18/2025 11:38 AM, Peter J. Holzer via Python-list wrote:

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

=20
Seems to me a system-installed application shouldn't be looking in the user's .local packages in the first place. That should only be for th=

ings

the user has installed "for this user".

=20
It's not the application that looks into .local, it's Python. If you say that a system-installed Python shouldn't look into ~/.local, then you've just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed scripts. This isn't as easy as checking whether the path starts with /usr/bin or whether it belongs to root. Tying into the system's package manager doesn't look appealing to me (not to mention that it might be unacceptably slow).

=20
Let's try a specific example. Let's say that PyQt6 v6.8.3 is installed in
the system site directory. The OS uses PyQt6 for some system purposes. Now the user comes along and forces an install of PyQt6 v6.9.0 in the user si=

te

directory. 6.9.0 has a bug that would crash one of the system's applicati=

on

but not the user's programs. (This is not a far-fetched scenario).

Agreed. Also since PyQt6 is a GUI framework, I'm going to assume that
those applications are supposed to be invoked by the user on their
desktop, not by a cronjob controlled by the system.

When the system launches its application the PYTHONPATH will start with system site directories; local user site directories will be on the PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3 because that will come first on the path. No crash here.
=20
If the user has a program that actually does require the use of v6.9.0, h=

e's

going to have to make sure that the user's local site directories come fi=

rst

on the path. One way to do that is to set the PYTHONPATH to point to the user's location.

This is IMHO not practical. The user would have to set PYTHONPATH for
some programs, but not for others. You can't do this with .bashrc (or
similar), the user would have to write a wrapper script for each of
their programs which depend on something in ~/.local. Possible of course
but cumbersome.

I like Oscar's suggestion that Python scripts provided by the
distribution include -s in the shebang much better.

Or - what I tend to do - simply use a virtual environment for each
script that needs a package that the system doesn't provide. But of
course that's basically "disable/don't use .local" and all those venvs
take space and need to be maintained.

In what scenario is a system application going to load a wrong version of=

a

dependency from a user's site location?

When the user sets PYTHONPATH and then accidentally invokes the
system-supplied application without resetting it first.

The only one I can think of is for the user, with the help of sudo, or
by editing some system-enabled script, were to change the global
PYTHONPATH. That seems a stretch.

No, there doesn't have to be a global (in the sense that it applies to
all users) PYTHONPATH for that to happen. You just need a PYTHONPATH
that is set for all processes of that user - which the user can
certainly set without sudo (usually by editing .bashrc or maybe using
their desktop environment's settings dialog).

hjp

--=20
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

--vp5o7ne6zbzmsmpv
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmgDZSgACgkQ8g5IURL+ KF2Fgw/+JSjASdQ79OXceKZHRToWr38cbCb9O5Q74xaoIjbOjrBK/Rgz3OFsSFxU NmwxxXvRiPkOa1KnvawDXmj/iedEqx1hIFJ+Nc66t7lyrWAtu7MSW/gmxXawGbL+ Y40/jMN6US0Ra9ZeYt1Jv/nU7qqgdT/p3mtDnQXlG3LLOZZEHhu//pS09tUHD7a/ y2B88/QzCe2WqnNEGnId+LUIzocEGTGKKWss7MVguKAw86ZxqA6tdv5W7GNTkHfb l77rJcaiJx/tAH29lbheo34WfHW3BpdACxBwgqLQJZmqqrLb4rY36mh540r2yGpU bVfdTWbRvFUkJVcRdvdDpKagShSLA5a2nxU9D8BVlSz8rbAOofHzKkDN+ZiC3OWb f6Xgz/eBVuxWDENOTobJOCiI8pCiPay7qlT9v9UVmKYotY7Rj3PysSUfyvhIPqSf +rarT1puLYAnorj+S54KkVBFKsU21wOMKGqg0YONn5rXRzbo1H6HIGcgwdz/7uNn hYZp5y0G0L+lIacueemMXQrhvfcu9OSSMqzakLMfzT1IAY5BtiDD7k07jD0ZFIAy cFJPuq3mLYVZi40xOlkuC5kPb72cVOMgIN1BXgZWhMqVDrSClqnXa2ASZu4WxWpr fTtM8+HFRzgmtN5ZK7Naa47yzff0xqlmZXDgbVDs/5kwfRE1Oes=
=jwSV
-----END PGP SIGNATURE-----

--vp5o7ne6zbzmsmpv--

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

Peter J. Holzer wrote:

On 2025-04-18 13:08:36 -0400, Thomas Passin via Python-list wrote:

....

When the system launches its application the PYTHONPATH will start with
system site directories; local user site directories will be on the
PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3
because that will come first on the path. No crash here.

If the user has a program that actually does require the use of v6.9.0, he's >> going to have to make sure that the user's local site directories come first >> on the path. One way to do that is to set the PYTHONPATH to point to the
user's location.

This is IMHO not practical. The user would have to set PYTHONPATH for
some programs, but not for others. You can't do this with .bashrc (or similar), the user would have to write a wrapper script for each of
their programs which depend on something in ~/.local. Possible of course
but cumbersome.

currently in my .bashrc i have it set up to look for which
directory the terminal is in and then runs the activate script
for that environment. i like to keep my desktops/projects
open with various numbers of terminals available so this way
they are all ready to go when the system boots up.

I like Oscar's suggestion that Python scripts provided by the
distribution include -s in the shebang much better.

Or - what I tend to do - simply use a virtual environment for each
script that needs a package that the system doesn't provide. But of
course that's basically "disable/don't use .local" and all those venvs
take space and need to be maintained.

i like that they do not change until i specificly ask them to
be changed.


....

The only one I can think of is for the user, with the help of sudo, or
by editing some system-enabled script, were to change the global
PYTHONPATH. That seems a stretch.

No, there doesn't have to be a global (in the sense that it applies to
all users) PYTHONPATH for that to happen. You just need a PYTHONPATH
that is set for all processes of that user - which the user can
certainly set without sudo (usually by editing .bashrc or maybe using
their desktop environment's settings dialog).

yes, that is part of what .bashrc is for, making sure your
environment variables are set how you'd like them.


songbird

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: the little wild kingdom (3:633/280.2@fidonet)

On 4/19/2025 4:56 AM, Peter J. Holzer via Python-list wrote:

On 2025-04-18 13:08:36 -0400, Thomas Passin via Python-list wrote:

On 4/18/2025 11:38 AM, Peter J. Holzer via Python-list wrote:

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the >>>> user's .local packages in the first place. That should only be for things >>>> the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say >>> that a system-installed Python shouldn't look into ~/.local, then you've >>> just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed
scripts. This isn't as easy as checking whether the path starts with
/usr/bin or whether it belongs to root. Tying into the system's package
manager doesn't look appealing to me (not to mention that it might be
unacceptably slow).

Let's try a specific example. Let's say that PyQt6 v6.8.3 is installed in
the system site directory. The OS uses PyQt6 for some system purposes. Now >> the user comes along and forces an install of PyQt6 v6.9.0 in the user site >> directory. 6.9.0 has a bug that would crash one of the system's application >> but not the user's programs. (This is not a far-fetched scenario).

Agreed. Also since PyQt6 is a GUI framework, I'm going to assume that
those applications are supposed to be invoked by the user on their
desktop, not by a cronjob controlled by the system.

When the system launches its application the PYTHONPATH will start with
system site directories; local user site directories will be on the
PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3
because that will come first on the path. No crash here.

If the user has a program that actually does require the use of v6.9.0, he's >> going to have to make sure that the user's local site directories come first >> on the path. One way to do that is to set the PYTHONPATH to point to the
user's location.

This is IMHO not practical. The user would have to set PYTHONPATH for
some programs, but not for others. You can't do this with .bashrc (or similar), the user would have to write a wrapper script for each of
their programs which depend on something in ~/.local. Possible of course
but cumbersome.

I like Oscar's suggestion that Python scripts provided by the
distribution include -s in the shebang much better.

Or - what I tend to do - simply use a virtual environment for each
script that needs a package that the system doesn't provide. But of
course that's basically "disable/don't use .local" and all those venvs
take space and need to be maintained.

My problem with venvs, especially if I have more than one, is that I eventually forget what they were for and what is different about each
one. If there's only one and it's used for all non-system work, that's
OK but beyond that and they tend to suffer knowledge rot.

In what scenario is a system application going to load a wrong version of a >> dependency from a user's site location?

When the user sets PYTHONPATH and then accidentally invokes the system-supplied application without resetting it first.

The only one I can think of is for the user, with the help of sudo, or
by editing some system-enabled script, were to change the global
PYTHONPATH. That seems a stretch.

No, there doesn't have to be a global (in the sense that it applies to
all users) PYTHONPATH for that to happen. You just need a PYTHONPATH
that is set for all processes of that user - which the user can
certainly set without sudo (usually by editing .bashrc or maybe using
their desktop environment's settings dialog).

Yes, I agree and I wasn't thinking locally enough.

--- MBSE BBS v1.1.1 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)