tmp is on a different filesystem: 'OSError: [Errno 18] Invalid cross-device link'

[vbabiy]

diff --git a/pip.py b/pip.py
--- a/pip.py
+++ b/pip.py
@@ -4714,7 +4714,7 @@
new_dir = os.path.dirname(new_path)
logger.info('Removing file or directory %s' % full_path)
self._moved_paths.append((prefix, path))

•                os.renames(full_path, new_path)
  

•                shutil.move(full_path, new_path)
           for pth in self.pth.values():
               pth.remove()
           logger.notify('Successfully uninstalled %s' % self.dist.project_name)
  

---

• Bitbucket: https://bitbucket.org/ianb/pip/issue/27
• Originally Reported By: Anonymous
• Originally Created At: 2009-11-13 14:18:19

[vbabiy]

diff --git a/pip.py b/pip.py

--- a/pip.py

+++ b/pip.py

@@ -4714,7 +4714,7 @@

                     new_dir = os.path.dirname(new_path)

                     logger.info('Removing file or directory %s' %

full_path)

                     self._moved_paths.append((prefix, path))

-                    os.renames(full_path, new_path)

+                    shutil.move(full_path, new_path)

                 for pth in self.pth.values():

                     pth.remove()

                 logger.notify('Successfully uninstalled %s' %

self.dist.project_name)

---

Original Comment By: Anonymous

[vbabiy]

Hmm, one problem is that we depend on the auto-creation of directories that
os.renames does for us, and shutil.move doesn't do that.

I'm also concerned about the loss of file metadata (permissions etc) that it
seems would happen if shutil.move had to resort to a copy (based on the
warning in shutil docs: http://docs.python.org/library/shutil.html).

We could try to create our own temp directory somewhere close to the
currently-installed location rather than in /tmp, but that seems like it might
have its own issues.

Not quite sure what to do here, input welcome.

---

Original Comment By: Carl Meyer

[vbabiy]

Further discussion: http://groups.google.com/group/python-
virtualenv/browse_thread/thread/18f5c08bd599288b

virtualenv/browse_thread/thread/18f5c08bd599288b

---

Original Comment By: Carl Meyer

[vbabiy]

There has been some discussion on list of this issue:

How portable does this need to be in terms of python versions?

shutil.copytree from python 2.5 handles copy then copystat which should
preserve permissions, etc

[1][1]

[2][2]

[3][3]

[4][4]

[5][5]

[6][6]



try:

   os.rename(src, dst)

except os.error, e:

   if e.errno == errno.EXDEV:

       shutil.copytree(src, dst, symlinks=True)

       shutil.rmtree(src)

---

Original Comment By: Paul Nasrat

[vbabiy]

I've worked around this by taking advantage of the fact that the tempfile
module will look at the environment, so I can

export TMPDIR=pwd/tmp

to use a local tmp directory, and things go fine.

---

Original Comment By: Anonymous

[vbabiy]

Oops, sorry that anonymous, was me.

---

Original Comment By: cdent

[vbabiy]

Issue #35 was marked as a duplicate of this issue.

device

---

Original Comment By: Carl Meyer

[vbabiy]

Ouch. #35 brings up the case of something called AFS where apparently
"every directory is presented to the system as a different device." And here I
was about ready to fix this by putting the save_dir in site-packages if
necessary; even that wouldn't help with AFS.

Patch on #35 is at http://lass.lfod.us/~will/pip-renames.diff --
uses shutil.move, so it still has the problem of potentially losing file
metadata. (Also, Paul Nasrat, shutil.copytree doesn't fully solve that problem
even if it uses copystat; file ownership information is still lost).

I may end up doing both: try putting the save_dir in site-packages to allow
use of os.renames on sane filesystems, and if even site-packages is still on a
different filesystem, resort to shutil.copytree.

device

---

Original Comment By: Carl Meyer

[vbabiy]

use shutil.move for rollback-save (fixes #27)

If TMPDIR is on a different filesystem, this has the potential to lose file
ownership info. In general that shouldn't be an issue; seems to be the best we
can do.

→ [Author: Carl Meyer

Age: 16 months ago

use shutil.move for rollback-save (fixes 27) If TMPDIR is on a different
filesystem, this has the potential to lose file ownership info. In general
that shouldn't be an issue; seems to be the best we can
do.">r398:4ae24807d08b]2

oserror

---

Original Comment By: Carl Meyer

[vbabiy]

Issue #52 was marked as a duplicate of this issue.

---

Original Comment By: Carl Meyer

This issue still occurs with pip 8.1.2 and python 3.5.2

See the following output from an attempt to install pgcli:

(pgcli) rdahlgren@athas:~/venvs $ pip --version
pip 8.1.2 from /home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages (python 3.5)
(pgcli) rdahlgren@athas:~/venvs $ python --version
Python 3.5.2
(pgcli) rdahlgren@athas:~/venvs $ pip install pgcli
Collecting pgcli
Collecting configobj>=5.0.6 (from pgcli)
Collecting setproctitle>=1.1.9 (from pgcli)
Collecting prompt-toolkit<1.1.0,>=1.0.0 (from pgcli)
  Using cached prompt_toolkit-1.0.5-py3-none-any.whl
Collecting psycopg2>=2.5.4 (from pgcli)
Collecting sqlparse==0.1.19 (from pgcli)
Requirement already satisfied (use --upgrade to upgrade): wcwidth>=0.1.6 in ./pgcli/lib/python3.5/site-packages (from pgcli)
Collecting pgspecial>=1.5.0 (from pgcli)
Requirement already satisfied (use --upgrade to upgrade): click>=4.1 in ./pgcli/lib/python3.5/site-packages (from pgcli)
Requirement already satisfied (use --upgrade to upgrade): Pygments>=2.0 in ./pgcli/lib/python3.5/site-packages (from pgcli)
Collecting humanize>=0.5.1 (from pgcli)
Requirement already satisfied (use --upgrade to upgrade): six in ./pgcli/lib/python3.5/site-packages (from configobj>=5.0.6->pgcli)
Installing collected packages: configobj, setproctitle, prompt-toolkit, psycopg2, sqlparse, pgspecial, humanize, pgcli
  Found existing installation: prompt-toolkit 0.46
    Uninstalling prompt-toolkit-0.46:
Exception:
Traceback (most recent call last):
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/shutil.py", line 538, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/prompt_toolkit-0.46.dist-info/DESCRIPTION.rst' -> '/tmp/pip-7zlbeq9d-uninstall/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/prompt_toolkit-0.46.dist-info/DESCRIPTION.rst'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/req/req_set.py", line 736, in install
    requirement.uninstall(auto_confirm=True)
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/req/req_install.py", line 742, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/shutil.py", line 553, in move
    os.unlink(src)
PermissionError: [Errno 13] Permission denied: '/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/prompt_toolkit-0.46.dist-info/DESCRIPTION.rst'

My /tmp is a separate filesystem and flagged noexec, as is the long standing best practice for /tmp:

(pgcli) rdahlgren@athas:~/venvs $ cat /etc/fstab | grep '/tmp'
UUID=REDACTED /tmp          btrfs       rw,relatime,space_cache,noexec  0 0

[WhyNotHugo]

This is still an issue on pip 9.0.1. Why is this closed?