From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id E6FFE138350 for ; Fri, 17 Apr 2020 19:52:32 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 623A8E08D0; Fri, 17 Apr 2020 19:52:31 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 36734E08D0 for ; Fri, 17 Apr 2020 19:52:31 +0000 (UTC) Received: from oystercatcher.gentoo.org (oystercatcher.gentoo.org [148.251.78.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id D75FE34F263 for ; Fri, 17 Apr 2020 19:52:29 +0000 (UTC) Received: from localhost.localdomain (localhost [IPv6:::1]) by oystercatcher.gentoo.org (Postfix) with ESMTP id 12EF31E8 for ; Fri, 17 Apr 2020 19:52:27 +0000 (UTC) From: "Matt Turner" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Matt Turner" Message-ID: <1587153092.bb21b8615e64cb31fa9aa9d533ef328dc1374e45.mattst88@gentoo> Subject: [gentoo-commits] proj/catalyst:master commit in: doc/, catalyst/base/, catalyst/ X-VCS-Repository: proj/catalyst X-VCS-Files: catalyst/base/genbase.py catalyst/support.py doc/HOWTO.txt X-VCS-Directories: catalyst/ catalyst/base/ doc/ X-VCS-Committer: mattst88 X-VCS-Committer-Name: Matt Turner X-VCS-Revision: bb21b8615e64cb31fa9aa9d533ef328dc1374e45 X-VCS-Branch: master Date: Fri, 17 Apr 2020 19:52:27 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply X-Archives-Salt: 7426358a-6e19-441f-a373-6187e4096bd0 X-Archives-Hash: af620e9b9c846634e1009563e0a404c5 commit: bb21b8615e64cb31fa9aa9d533ef328dc1374e45 Author: Matt Turner gentoo org> AuthorDate: Fri Apr 17 18:03:02 2020 +0000 Commit: Matt Turner gentoo org> CommitDate: Fri Apr 17 19:51:32 2020 +0000 URL: https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=bb21b861 catalyst: gzip the .CONTENTS file Other algorithms give better compression ratios, but the difference is not meaningful for a 2MiB text file. In my testing bzip2 gave a better compression ratio of 15:1 vs gzip's 11:1, but that ends up being only a size difference of 50KiB (136 vs 187) which is only an additional 2.5% savings from the uncompressed input. Choose gzip because transparent decompression is widely supported by web servers and clients. Closes: https://bugs.gentoo.org/630284 Signed-off-by: Matt Turner gentoo.org> catalyst/base/genbase.py | 5 +++-- catalyst/support.py | 2 +- doc/HOWTO.txt | 6 +++--- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/catalyst/base/genbase.py b/catalyst/base/genbase.py index eb09a4e0..3db20f84 100644 --- a/catalyst/base/genbase.py +++ b/catalyst/base/genbase.py @@ -2,6 +2,7 @@ import hashlib import io import os +import gzip class GenBase(): """ @@ -28,7 +29,7 @@ class GenBase(): def gen_contents_file(self, path): c = self.settings['contents_map'] - with io.open(path + '.CONTENTS', 'w', encoding='utf-8') as file: + with gzip.open(path + '.CONTENTS.gz', 'w', encoding='utf-8') as file: file.write(c.contents(path, '', verbose=self.settings['VERBOSE'])) def gen_digest_file(self, path): @@ -36,6 +37,6 @@ class GenBase(): return with io.open(path + '.DIGESTS', 'w', encoding='utf-8') as file: - for f in [path, path + '.CONTENTS']: + for f in [path, path + '.CONTENTS.gz']: for i in self.settings['digests']: file.write(self.generate_hash(f, name=i)) diff --git a/catalyst/support.py b/catalyst/support.py index 654b23aa..988a81f5 100644 --- a/catalyst/support.py +++ b/catalyst/support.py @@ -65,7 +65,7 @@ def file_check(filepath, extensions=None, strict=True): files = glob.glob("%s.*" % filepath) # remove any false positive files files = [x for x in files if not x.endswith( - ".CONTENTS") and not x.endswith(".DIGESTS")] + ".CONTENTS") and not x.endswith(".CONTENTS.gz") and not x.endswith(".DIGESTS")] if len(files) == 1: return files[0] if len(files) > 1 and strict: diff --git a/doc/HOWTO.txt b/doc/HOWTO.txt index 960b5761..7b759121 100644 --- a/doc/HOWTO.txt +++ b/doc/HOWTO.txt @@ -22,7 +22,7 @@ Create a snapshot of your current Portage tree (you may want to # catalyst --snapshot 20130131 # ls /var/tmp/catalyst/snapshots/ portage-20130131.tar.bz2 - portage-20130131.tar.bz2.CONTENTS + portage-20130131.tar.bz2.CONTENTS.gz portage-20130131.tar.bz2.DIGESTS where the storage location is relative to the default @@ -44,7 +44,7 @@ For example, Grab the tarball and put it where catalyst will find it: # wget http://…/stage3-amd64-20121213.tar.bz2 - # wget http://…/stage3-amd64-20121213.tar.bz2.CONTENTS + # wget http://…/stage3-amd64-20121213.tar.bz2.CONTENTS.gz # wget http://…/stage3-amd64-20121213.tar.bz2.DIGESTS.asc # sha512sum -c stage3-amd64-20121213.tar.bz2.DIGESTS.asc # gpg --verify stage3-amd64-20121213.tar.bz2.DIGESTS.asc @@ -89,7 +89,7 @@ which will build the target and install something like: # ls /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.* /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2 - /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2.CONTENTS + /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2.CONTENTS.gz /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2.DIGESTS The name is an expansion of