From mboxrd@z Thu Jan 1 00:00:00 1970
Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org)
by finch.gentoo.org with esmtp (Exim 4.60)
(envelope-from <gentoo-commits+bounces-396856-garchives=archives.gentoo.org@lists.gentoo.org>)
id 1RKFfg-0004l6-4T
for garchives@archives.gentoo.org; Sat, 29 Oct 2011 20:38:28 +0000
Received: from pigeon.gentoo.org (localhost [127.0.0.1])
by pigeon.gentoo.org (Postfix) with SMTP id 79E5E21C028;
Sat, 29 Oct 2011 20:38:20 +0000 (UTC)
Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183])
by pigeon.gentoo.org (Postfix) with ESMTP id 3400221C028
for <gentoo-commits@lists.gentoo.org>; Sat, 29 Oct 2011 20:38:20 +0000 (UTC)
Received: from pelican.gentoo.org (unknown [66.219.59.40])
(using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by smtp.gentoo.org (Postfix) with ESMTPS id 6E8441B4011
for <gentoo-commits@lists.gentoo.org>; Sat, 29 Oct 2011 20:38:19 +0000 (UTC)
Received: from localhost.localdomain (localhost [127.0.0.1])
by pelican.gentoo.org (Postfix) with ESMTP id CFEC880042
for <gentoo-commits@lists.gentoo.org>; Sat, 29 Oct 2011 20:38:18 +0000 (UTC)
From: "Zac Medico" <zmedico@gentoo.org>
To: gentoo-commits@lists.gentoo.org
Content-type: text/plain; charset=UTF-8
Reply-To: gentoo-dev@lists.gentoo.org, "Zac Medico" <zmedico@gentoo.org>
Message-ID: <0e120da008c9d0d41c9372c81145c6e153028a6d.zmedico@gentoo>
Subject: [gentoo-commits] proj/portage:master commit in: bin/
X-VCS-Repository: proj/portage
X-VCS-Files: bin/egencache
X-VCS-Directories: bin/
X-VCS-Committer: zmedico
X-VCS-Committer-Name: Zac Medico
X-VCS-Revision: 0e120da008c9d0d41c9372c81145c6e153028a6d
Date: Sat, 29 Oct 2011 20:38:18 +0000 (UTC)
Precedence: bulk
List-Post: <mailto:gentoo-commits@lists.gentoo.org>
List-Help: <mailto:gentoo-commits+help@lists.gentoo.org>
List-Unsubscribe: <mailto:gentoo-commits+unsubscribe@lists.gentoo.org>
List-Subscribe: <mailto:gentoo-commits+subscribe@lists.gentoo.org>
List-Id: Gentoo Linux mail <gentoo-commits.gentoo.org>
X-BeenThere: gentoo-commits@lists.gentoo.org
Content-Transfer-Encoding: quoted-printable
X-Archives-Salt:
X-Archives-Hash: fb58ead545c6fd8974ebb105c17a9308
commit: 0e120da008c9d0d41c9372c81145c6e153028a6d
Author: Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Oct 29 20:36:23 2011 +0000
Commit: Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Oct 29 20:36:23 2011 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=3Dproj/portage.git;a=
=3Dcommit;h=3D0e120da0
egencache: avoid redundant md5-dict writes
The pms cache already does this automatically, since __setitem__ calls
are used to detect stat collisions in order to solve bug #139134.
---
bin/egencache | 58 ++++++++++++++++++++++++++++++++++++++++++++-------=
-----
1 files changed, 45 insertions(+), 13 deletions(-)
diff --git a/bin/egencache b/bin/egencache
index 22ce8ec..33839aa 100755
--- a/bin/egencache
+++ b/bin/egencache
@@ -199,7 +199,12 @@ def parse_args(args):
class GenCache(object):
def __init__(self, portdb, cp_iter=3DNone, max_jobs=3DNone, max_load=3D=
None,
rsync=3DFalse):
+ # The caller must set portdb.porttrees in order to constrain
+ # findname, cp_list, and cpv_list to the desired tree.
+ tree =3D portdb.porttrees[0]
self._portdb =3D portdb
+ self._eclass_db =3D portdb._repo_info[tree].eclass_db
+ self._auxdbkeys =3D portage.auxdbkeys
# We can globally cleanse stale cache only if we
# iterate over every single cp.
self._global_cleanse =3D cp_iter is None
@@ -214,22 +219,25 @@ class GenCache(object):
consumer=3Dself._metadata_callback,
max_jobs=3Dmax_jobs, max_load=3Dmax_load)
self.returncode =3D os.EX_OK
- conf =3D portdb.repositories.get_repo_for_location(portdb.porttrees[0]=
)
+ conf =3D portdb.repositories.get_repo_for_location(tree)
self._trg_caches =3D tuple(conf.iter_pregenerated_caches(
- portage.auxdbkeys[:], force=3DTrue, readonly=3DFalse))
+ self._auxdbkeys, force=3DTrue, readonly=3DFalse))
if not self._trg_caches:
raise Exception("cache formats '%s' aren't supported" %
(" ".join(conf.cache_formats),))
- if rsync:
- from portage.cache.metadata import database as pms_database
- for trg_cache in self._trg_caches:
- if isinstance(trg_cache, pms_database):
- trg_cache.raise_stat_collision =3D True
- # Make _metadata_callback write this cache first, in case
- # it raises a StatCollision and triggers mtime
- # modification.
- self._trg_caches =3D tuple([trg_cache] +
- [x for x in self._trg_caches if x is not trg_cache])
+
+ self._avoid_redundant_write =3D set()
+ from portage.cache.metadata import database as pms_database
+ for trg_cache in self._trg_caches:
+ if not isinstance(trg_cache, pms_database):
+ self._avoid_redundant_write.add(id(trg_cache))
+ elif rsync:
+ trg_cache.raise_stat_collision =3D True
+ # Make _metadata_callback write this cache first, in case
+ # it raises a StatCollision and triggers mtime
+ # modification.
+ self._trg_caches =3D tuple([trg_cache] +
+ [x for x in self._trg_caches if x is not trg_cache])
=20
self._existing_nodes =3D set()
=20
@@ -244,6 +252,27 @@ class GenCache(object):
cpv, repo_path, metadata, ebuild_hash)
=20
def _write_cache(self, trg_cache, cpv, repo_path, metadata, ebuild_hash=
):
+
+ if id(trg_cache) in self._avoid_redundant_write:
+ # This cache does not avoid redundant writes automatically,
+ # so check for an identical existing entry before writing.
+ # This prevents unecessary disk writes and can also prevent
+ # unecessary rsync transfers.
+ try:
+ dest =3D trg_cache[cpv]
+ except (KeyError, CacheError):
+ pass
+ else:
+ if trg_cache.validate_entry(dest,
+ ebuild_hash, self._eclass_db):
+ identical =3D True
+ for k in self._auxdbkeys:
+ if dest.get(k, '') !=3D metadata.get(k, ''):
+ identical =3D False
+ break
+ if identical:
+ return
+
try:
chf =3D trg_cache.validation_chf
metadata['_%s_' % chf] =3D getattr(ebuild_hash, chf)
@@ -256,7 +285,10 @@ class GenCache(object):
# exception from _setitem() if they detect this type of stat
# collision. These exceptions are handled by bumping the
# mtime on the ebuild (and the corresponding cache entry).
- # See bug #139134.
+ # This type of cache must not be included in the above
+ # _avoid_redundant_write set, since __setitem__ must be
+ # called in order to detect the StatCollision (redundant
+ # writes will be avoided internally). See bug #139134.
max_mtime =3D sc.mtime
for ec, ec_hash in metadata['_eclasses_'].items():
if max_mtime < ec_hash.mtime: