From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 467DC138A87 for ; Mon, 23 Feb 2015 20:59:22 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id E2ED9E08B8; Mon, 23 Feb 2015 20:59:21 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 600BCE08A8 for ; Mon, 23 Feb 2015 20:59:21 +0000 (UTC) Received: from oystercatcher.gentoo.org (oystercatcher.gentoo.org [148.251.78.52]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 78C8A340913 for ; Mon, 23 Feb 2015 20:59:20 +0000 (UTC) Received: from localhost.localdomain (localhost [127.0.0.1]) by oystercatcher.gentoo.org (Postfix) with ESMTP id 2A4E812674 for ; Mon, 23 Feb 2015 20:59:19 +0000 (UTC) From: "Alex Legler" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Alex Legler" Message-ID: <1424724981.bbc8b2a61544af2859276cf8badb53ca3b1244c5.a3li@gentoo> Subject: [gentoo-commits] proj/ag:master commit in: lib/ X-VCS-Repository: proj/ag X-VCS-Files: lib/utils.rb X-VCS-Directories: lib/ X-VCS-Committer: a3li X-VCS-Committer-Name: Alex Legler X-VCS-Revision: bbc8b2a61544af2859276cf8badb53ca3b1244c5 X-VCS-Branch: master Date: Mon, 23 Feb 2015 20:59:19 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Archives-Salt: e5a80351-bb8e-4f65-a587-a45769628e65 X-Archives-Hash: 7b627d67c39731d1d5bc3b7d6fdcca2f commit: bbc8b2a61544af2859276cf8badb53ca3b1244c5 Author: Alex Legler a3li li> AuthorDate: Mon Feb 23 20:56:21 2015 +0000 Commit: Alex Legler gentoo org> CommitDate: Mon Feb 23 20:56:21 2015 +0000 URL: http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=bbc8b2a6 Test (hopefully) faster encoding fix method --- lib/utils.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/utils.rb b/lib/utils.rb index aaa04c7..fc4427e 100644 --- a/lib/utils.rb +++ b/lib/utils.rb @@ -2,8 +2,15 @@ require 'charlock_holmes' module Ag module Utils + module_function - def fix_encoding(str, fail_hard = false) + def fix_encoding(str) + s = str.encode('UTF-8', 'UTF-8', invalid: :replace, replace: '') + s = s.unpack('C*').pack('U*') unless s.valid_encoding? + s + end + + def fix_encoding_old(str, fail_hard = false) detection = CharlockHolmes::EncodingDetector.detect(str) CharlockHolmes::Converter.convert(str, detection[:encoding], 'UTF-8') rescue => e