From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id AD90A1381F3 for ; Tue, 6 Aug 2013 16:03:11 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 2E241E0B4F; Tue, 6 Aug 2013 16:03:06 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 76657E0B4F for ; Tue, 6 Aug 2013 16:03:00 +0000 (UTC) Received: from hornbill.gentoo.org (hornbill.gentoo.org [94.100.119.163]) (using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 46B6633E9B5 for ; Tue, 6 Aug 2013 16:02:59 +0000 (UTC) Received: from localhost.localdomain (localhost [127.0.0.1]) by hornbill.gentoo.org (Postfix) with ESMTP id D5F2DE5463 for ; Tue, 6 Aug 2013 16:02:57 +0000 (UTC) From: "André Erdmann" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "André Erdmann" Message-ID: <1375804931.86b204c29cd4db9d7a404fa9f2738dd3be638f5c.dywi@gentoo> Subject: [gentoo-commits] proj/R_overlay:master commit in: doc/html/ X-VCS-Repository: proj/R_overlay X-VCS-Files: doc/html/usage.html X-VCS-Directories: doc/html/ X-VCS-Committer: dywi X-VCS-Committer-Name: André Erdmann X-VCS-Revision: 86b204c29cd4db9d7a404fa9f2738dd3be638f5c X-VCS-Branch: master Date: Tue, 6 Aug 2013 16:02:57 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Archives-Salt: ef95eb90-777b-454b-a258-abac69589d44 X-Archives-Hash: c32d036952e647a3bdeaccaf93eb5f53 commit: 86b204c29cd4db9d7a404fa9f2738dd3be638f5c Author: André Erdmann mailerd de> AuthorDate: Tue Aug 6 16:02:11 2013 +0000 Commit: André Erdmann mailerd de> CommitDate: Tue Aug 6 16:02:11 2013 +0000 URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=86b204c2 update doc/html/usage.html (generated file) --- doc/html/usage.html | 589 +++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 419 insertions(+), 170 deletions(-) diff --git a/doc/html/usage.html b/doc/html/usage.html index 16fbbe9..39d7e63 100644 --- a/doc/html/usage.html +++ b/doc/html/usage.html @@ -418,58 +418,59 @@ ul.auto-toc {
  • 10.5.2   file logging
  • -
  • 10.6   options for debugging, manual dependency rule creation and testing
  • +
  • 10.6   license map options
  • +
  • 10.7   options for debugging, manual dependency rule creation and testing
  • -
  • 11   Other config files
      -
    • 11.1   USE_EXPAND flag rename file
    • -
    • 11.2   Field Definition Config
        -
      • 11.2.1   Example: The default field definition file
      • +
      • 11   Other config files
      • -
      • 12   Roverlay Console
          -
        • 12.1   Dependency Resolution Console
        • +
        • 12   Roverlay Console
        • -
        • 13   Roverlay Interface
            -
          • 13.1   DepRes Interface
          • -
          • 13.2   Remote Interface
          • +
          • 13   Roverlay Interface
          • -
          • 14   Implementation Overview
              -
            • 14.1   PackageInfo
            • -
            • 14.2   Repository Management
                -
              • 14.2.1   Repository
                  -
                • 14.2.1.1   Adding new repository types
                • +
                • 14   Implementation Overview
                    +
                  • 14.1   PackageInfo
                  • +
                  • 14.2   Repository Management
                  • -
                  • 14.3   Overlay
                      -
                    • 14.3.1   Metadata Creation
                    • -
                    • 14.3.2   Manifest Creation
                    • +
                    • 14.3   Overlay
                    • -
                    • 14.4   Ebuild Creation
                        -
                      • 14.4.1   Ebuild Variables
                      • +
                      • 14.4   Ebuild Creation
                      • -
                      • 14.5   Overlay Creation
                          -
                        • 14.5.1   Selfdep Validation
                        • +
                        • 14.5   Overlay Creation
                        • -
                        • 14.6   Dependency Resolution
                            -
                          • 14.6.1   Dependency types @@ -976,21 +977,19 @@ most cases, but fine-tuning can be done via a wrapper around /bin/sh that runs a shell or shell script in roverlay's hook environment.

                            roverlay-related scripts can use it to automatically inherit roverlay's -config and $FUNCTIONS file:

                            -
                            +config and get access to its $FUNCTIONS file:

                            +
                             #!/usr/bin/roverlay-sh
                             
                            -# reset DEBUG, VERBOSE, QUIET
                            -DEBUG=n; QUIET=n; VERBOSE=y
                            -
                             # load the functions file (optional)
                             . "${FUNCTIONS?}" || exit
                             
                             # script body
                             true
                             
                            -

                            <<TODO: maybe there's a better place for the details>>

                            +
                            roverlay-mkconfig
                            +
                            a script that creates a config file for roverlay
                            name_is_todo--roverlay_creation_helper
                            <<TODO>> Safely runs overlay creation <<and $$afterwards>>. @@ -1341,7 +1340,9 @@ Options with whitespace are not supported.
                          • This is your best bet if the remote is a repository but does not offer rsync access. Basic digest verification is supported (MD5). The remote has to have a package list file, typically named PACKAGES, -with a special syntax (debian control file syntax).

                            +with a special syntax (debian control file syntax). Syncing is retried up to +3 times, for example if a connection timeout occurs or a remote file +disappears after reading the package list file.

                            A package list example, excerpt from omegahat's PACKAGES file (as of Aug 2012):

                            @@ -1453,7 +1454,6 @@ reduces its size considerably.

                            The additions directory is a directory with overlay-like structure that contains extra files for overlay creation. Currently, ebuild patches and ebuild files are supported.

                            -

                            To give an idea of how this directory could

                            6.1   Patching ebuilds

                            Patches can apply to a specific version or to all versions of a @@ -1496,13 +1496,13 @@ for collecting dependencies.

                            Foreign ebuilds can be imported into the overlay by simple putting them into the additions directory.

                            The naming convention is similar to ebuild patches and identical to the -portage tree:

                            -
                            -${CATEGORY}/${PN}/${PF}.ebuild
                            -
                            -

                            Ebuilds imported that way can not be overwritten by generated ebuilds and +portage tree, ${CATEGORY}/${PN}/${PF}.ebuild.

                            +

                            Ebuilds imported that way cannot be overwritten by generated ebuilds and benefit from most overlay creation features, e.g. Manifest file creation. However, they cannot be used for metadata creation.

                            +

                            The ${CATEGORY}/${PN}/metadata.xml file will be imported if it exists and +if ${PN} in the overlay (a) has no metadata.xml file or (b) has no +generated ebuilds (only imports).

                            @@ -1781,7 +1781,9 @@ colong char ::

                            start with #. There are a few exceptions to that, the #deptype and #! NOPARSE keywords. Comments inside rule blocks are not allowed and will be read as normal dependency strings.
                            -
                            Selfdep
                            + +
                            +
                            Selfdep

                            This is a classification for dependencies on packages which are also part of the overlay being created. Typically, selfdeps refer to other R packages, but there may be a few exceptions.

                            @@ -2907,6 +2909,13 @@ This will pass '--bwlimit=<value>' to all rsync commands.

                            Defaults to <not set>, which disables bandwidth limitation.

                            +
                            +
                            PORTDIR
                            +

                            Path to the portage tree. This option is recommended, but not always +required.

                            +

                            Defaults to "/usr/portage".

                            +
                            +

                            10.2   overlay options

                            @@ -3342,8 +3351,41 @@ files will be kept.

                            +
                            +

                            10.6   license map options

                            +
                            +
                            CREATE_LICENSE_FILE
                            +
                            Alias to CREATE_LICENSES_FILE.
                            +
                            +
                            +
                            CREATE_LICENSES_FILE
                            +

                            Create the CACHEDIR/license file after scanning PORTDIR/licenses for +licenses. This file can serve as fallback if PORTDIR is not available +or if USE_PORTAGE_LICENSES is set to no.

                            +

                            Defaults to yes.

                            +
                            +
                            +
                            +
                            LICENSE_MAP
                            +

                            Path to the license map file, which is used to translate license strings +into valid licenses (accepted by portage). Its syntax is similar to +dependency rules.

                            +

                            This option is not required, but recommended if the +field definition config file contains a field with the isLicense flag.

                            +

                            Defaults to <not set>.

                            +
                            +
                            +
                            +
                            USE_PORTAGE_LICENSES
                            +

                            A bool that controls whether PORTDIR/licenses should be scanned for +licenses. As fallback, or if this option is set to no, the +CACHEDIR/license file is read.

                            +

                            Defaults to yes.

                            +
                            +
                            +
                            -

                            10.6   options for debugging, manual dependency rule creation and testing

                            +

                            10.7   options for debugging, manual dependency rule creation and testing

                            DESCRIPTION_DIR

                            A directory where all description data read from an R package will be @@ -3528,178 +3570,262 @@ such a field is found.

                            12   Roverlay Console

                            -

                            <<section is TODO>>

                            -

                            <<links to depres console chapter may need to be fixed>>

                            -

                            <<basic commands, table>>

                            -

                            <<note regarding python -OO and missing help texts>>

                            +

                            roverlay provides an interactive console for accessing certain subsystems, +e.g. dependency resolution. Its features include tab completion for filesystem +paths and a command history that supports navigation with the arrow keys.

                            +

                            The console also implements a subset of typical tools like cat, which +behave similar but not identical to their counterparts.

                            +

                            See the following sections for how to run specific consoles.

                            +

                            The following table lists all basic commands, which are available in all +roverlay consoles. Some commands have more detailed help messages, which are +printed when running <cmd> --help.

                            + + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
                            basic console commands (subsystem-independent)
                            commandextended +--helpdescription
                            exit, quit, q, qqnoexit
                            help, ?noprint help message (list known commands)
                            help cmdnoshow command-specific help message
                            aliasnoshow command aliases
                            unaliasnounset command aliases (not implemented)
                            catyesread files and print them (supports compressed +files)
                            cdno

                            change working directory

                            +

                            Actually, this does not change the working +directory. It simply sets the prefix which is used +when dealing with relative filesystem paths.

                            +
                            chrootno

                            enter/leave/show command chroot

                            +

                            A command chroot prefixes all input with a +specific command.

                            +
                            declarenoshow variables
                            echonoprint text (supports python string formatting)
                            execnoswitch to another subsystem (not implemented)
                            historynoshow command history
                            lsnoprint directory content
                            pwdnoprint current working directory
                            setnoset variables
                            unsetnounset variables
                            +
                            +

                            Note

                            +

                            Running the console with python -OO removes most of the help messages.

                            +

                            12.1   Dependency Resolution Console

                            -
                            -

                            Warning

                            -

                            This section is out-of-date.

                            -

                            As previously stated, the DepRes Console is only meant for testing. It is an interactive console with the following features:

                            • resolve dependencies
                            • create new dependency rules
                            • -
                            • create dependency rules for each R package found in a directory
                            • load rules from files
                            • save rules to a file

                            Rules are managed in a set. These so-called rule pools are organized in -a first-in-first-out data structure that allows -to create or remove them easily at runtime.

                            -

                            Running roverlay depres_console will print a welcome message that -lists all available commands and a few usage hints.

                            -

                            For reference, these commands are currently available:

                            -

                            <<TODO: rewrite/update command table, it's out-of-date>>

                            +a first-in-first-out data structure that allows to create or remove +them easily at runtime.

                            +

                            Running roverlay depres_console prints a short welcome message and starts +the console.

                            +

                            For reference, these commands are available (in addition to the basic ones):

                            +--+++ + - - - - - - - - - - - - - + + + + + + - - + + + - - + + + + + + + - + + + + + + + + + + - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                            depres commands
                            commandextended +--help description
                            help, -h, ?lists all commands
                            help <cmd>, -h <cmd>prints a command-specific help message
                            load <file|dir>, -l <file|dir>loads a rule file or a directory with rule files -into a new rule pool
                            load_conf, -lcloads the rule files listed in the config file -into a new rule pool
                            add_rule <rule> -+ <rule>
                            add_pool, <<nocreates a new, empty rule pool
                            add_rule, +no creates a new rule and adds it to the topmost, i.e. latest rule pool. This command uses Rule File Syntax. Multi line rules are supported.
                            add_pool, -<<creates a new rule pool
                            load, lyesload rule files
                            unwind_pool, ->>removes the topmost rule pool and all of its -rules
                            load_conf, lcnoload configured rule files
                            print_pool, print, +pyesprint one or more rule pools
                            resolve <dep>, -?? <dep>
                            resolve, ??no tries to resolve the given dependency string and prints the result
                            unwind_pool, >>yesremoves the topmost rule pool and all of its +rules
                            write, wyeswrites rules to a file
                            !enter the resolve command chroot -all input will be prefixed with "resolve"noenter the resolve command chroot, which prefixes +all input with resolve
                            !!no leave any command chroot
                            print, pprints the rules of the topmost rule pool
                            print --all|-aprints the rules of all rule pools
                            print <id> [<id>..]prints the rules of a specific rule pool
                            write <file>, -w <file>writes the rules of the topmost rule pool into -<file>. See write --help for advanced usage.
                            cd <dir>changes the working directory -This is a virtual command. <<TODO:EXPLAIN>>
                            set VAR=VALUEset variables
                            unset VARunset variables
                            declareshow declared variables
                            aliasshow command aliases
                            unaliasreserved for future usage
                            exit, qq, qexits the DepRes Console
                            -
                            -

                            Note

                            -

                            Some commands also offer more detailed help via <command> --help.

                            -
                            -

                            <<TODO>>

                            -

                            <<Example Session is out-of-date>>

                            Example Session:
                            -== depres console ==
                            -Run 'help' to list all known commands
                            -More specifically, 'help <cmd>' prints a help message for the given
                            -command, and 'help --list' lists all help topics available
                            -Use 'load_conf' or 'lc' to load the configured rule files
                            -
                            -commands: load, unwind, set, help, >>, load_conf, <<, cd, mkhelp,
                            -resolve, lc, add_pool, addrule, h, +, l, li, write, p, r, ?, w, print,
                            -sd, unset, scandir
                            -exit-commands: q, qq, exit
                            +[roverlay depres_console]
                            +
                            +== dependency resolution console (r2) ==
                            +Run 'help' to list all known commands.
                            +More specifically, 'help <cmd>' prints a help message for
                            +the given command, and 'help --list' lists all help topics.
                            +Use 'load_conf' or 'lc' to load the configured rule files.
                            +
                            +cmd % help
                            +
                            +Documented commands (type help <topic>):
                            +========================================
                            +EOF       cat      echo  history    print       qq       unalias
                            +add_pool  cd       exec  load       print_pool  quit     unset
                            +add_rule  chroot   exit  load_conf  pwd         resolve  unwind_pool
                            +alias     declare  help  ls         q           set      write
                            +
                            +cmd % <tab><tab>
                            +EOF          chroot       history      pwd          unalias
                            +add_pool     declare      load         q            unset
                            +add_rule     echo         load_conf    qq           unwind_pool
                            +alias        exec         ls           quit         write
                            +cat          exit         print        resolve
                            +cd           help         print_pool   set
                             
                             cmd % + ~dev-lang/R :: R language
                            -new rules:
                            +
                            +cmd % print --help
                            +usage: print_pool [-h] [--all] [<id> [<id> ...]]
                            +
                            +positional arguments:
                            +  <id>        print specific pools (by id)
                            +
                            +optional arguments:
                            +  -h, --help  show this help message and exit
                            +  --all, -a   print all pools
                            +
                            +cmd % print -a
                             ~dev-lang/R :: R language
                            ---- ---
                            -command succeeded.
                             
                            -cmd % ? R language
                            -Trying to resolve ('R language',).
                            -Resolved as: ('dev-lang/R',)
                            +cmd % !
                            +
                            +(resolve) % R language
                            +'R language' has been resolved as dev-lang/R.
                             
                            -cmd % ? R language [ 2.15 ]
                            -Trying to resolve ('R language [ 2.15 ]',).
                            -Resolved as: ('>=dev-lang/R-2.15',)
                            +(resolve) % R language [ 2.15 ]
                            +'R language [ 2.15 ]' has been resolved as >=dev-lang/R-2.15.
                             
                            -cmd % ? R
                            -Trying to resolve ('R',).
                            -Channel returned None. At least one dep could not be resolved.
                            +(resolve) % R
                            +'R' could not be resolved.
                            +
                            +(resolve) % !!
                             
                             cmd % p
                             ~dev-lang/R :: R language
                             
                             cmd % >>
                            -Pool removed from resolver.
                            +pool has been removed.
                            +
                            +cmd % >>
                            +resolver has no pools.
                             
                             cmd % p
                             
                            -cmd % ? R language
                            -Trying to resolve ('R language',).
                            -Channel returned None. At least one dep could not be resolved.
                             
                            -cmd % exit
                            +cmd % ?? R language
                            +'R language' could not be resolved.
                            +
                            +cmd % set PS1=#!
                            +
                            +#! exit
                             
                            @@ -3708,7 +3834,7 @@ cmd % exit

                            13   Roverlay Interface

                            Roverlay provides an API for accessing its functionality independently of -R overlay creation. Only dependency resolution is available, currently.

                            +R overlay creation.

                            Note, however, that a minimal config file may still be required for accessing roverlay interfaces.

                            The table below lists all interfaces and where to find them:

                            @@ -3775,7 +3901,16 @@ for usage examples.

                            13.2   Remote Interface

                            -

                            <<TODO; this interface isn't mature enough yet (it will likely change in future)>>

                            +

                            The RemoteInterface is experimental/incomplete and currently offers the +following functionality:

                            +
                              +
                            • set sync mode to online/offline
                            • +
                            • sync/nosync
                            • +
                            • load repo config
                            • +
                            • list repos
                            • +
                            • list repo packages
                            • +
                            +

                            Refer to in-code documentation for details.

                            @@ -4002,12 +4137,12 @@ would require variable interpolation, e.g. for ${PN

                            14.4   Ebuild Creation

                            Ebuild creation is the process centered around one PackageInfo instance p that tries to create an ebuild for it.

                            -

                            It does the following steps:

                            +

                            An EbuildCreationJob does the following steps:

                            1. Read the DESCRIPTION file of p R package tarball and stores the data in an associative array ('DESCRIPTION field' -> 'data')
                            2. Call dependency resolution
                            3. -
                            4. If dependency resolution was successful and any selfdeps found +
                            5. If dependency resolution was successful and any selfdeps found (dependencies on other packages): pause ebuild creation for p until it has been verified whether these dependencies are satisfiable. This is necessary because dependency resolution does not know whether a @@ -4045,9 +4180,121 @@ for a PackageInfo object and inform the OverlayCreation about afterwards. Overlay creation keeps going if an ebuild cannot be created, instead the event is logged. Running more than one OverlayWorker in parallel is possible.

                              +

                              The following pseudo-code illustrates how overlay creation basically works:

                              +
                              +ACCEPT_PACKAGES:
                              +
                              +for each received PackageInfo <p>
                              +
                              +   create an EbuildCreationJob for p and add it to the work queue
                              +
                              +end for
                              +
                              +
                              +
                              +CREATE_OVERLAY:
                              +
                              +while work_queue is not empty
                              +
                              +   work_queue_next <= empty
                              +
                              +
                              +   in parallel with N OverlayWorkers (>=0 threads):
                              +
                              +      for each EbuildCreationJob ebuild_job from the work_queue
                              +
                              +         run/resume ebuild_job (as described in Ebuild Creation)
                              +
                              +         if ebuild_job is paused
                              +
                              +            add ebuild_job to work_queue_next
                              +
                              +         end if
                              +
                              +      end for
                              +
                              +   end in parallel
                              +
                              +
                              +   if work_queue_next is not empty
                              +
                              +      <run selfdep validation>
                              +
                              +      work_queue <= work_queue_next
                              +
                              +   end if
                              +
                              +end while
                              +

                              14.5.1   Selfdep Validation

                              -

                              <<TODO: specify algo here>>

                              +

                              EbuildCreationJobs are processed in no specific order and possibly +concurrently. This leads to the problem that dependency resolution cannot +know whether a successfully resolved selfdep is actually satisfiable.

                              +

                              For example, if a package A depends on another package B and B is +uncreatable (ebuild creation will fail for B), then dependency resolution +still resolves B (e.g. as sci-R/B). A has a dangling selfdep now.

                              +

                              Selfdep validation is the process of identifying dangling selfdeps and +removing packages with such dependencies or simply dropping +the dependencies, depending on their dependency type.

                              +

                              It is an algorithm with 3 phases:

                              +
                                +
                              1. prepare: create a graph containing all selfdeps
                                  +
                                • collect: find ebuild creation jobs with selfdeps (direct selfdeps)
                                • +
                                • link: expand the selfdep graph, find selfdep-selfdep dependencies +(indirect selfdeps)
                                • +
                                +
                              2. +
                              3. reduce: find dangling selfdeps and mark them as invalid
                                  +
                                • a selfdep is dangling iff the overlay contains no suitable +PackageInfo with valid selfdeps
                                • +
                                • repeat this step until no more dangling selfdeps found
                                • +
                                +
                              4. +
                              5. balance: find ebuild creation jobs with invalid selfdeps (inversed collect)
                                  +
                                • drop optional dependencies
                                • +
                                • let ebuild creation fail if a mandatory selfdep is not valid
                                • +
                                +
                              6. +
                              +

                              The actual implementation in roverlay is spread across several modules:

                              + + ++++ + + + + + + + + + + + + + + + + + + + + + + +
                              modules/packages involved in selfdep validation
                              module/packagephase(s) / description
                              roverlay.overlay.creatorall (controls selfdep validation), +especially collect and +reduction loop
                              roverlay.ebuild.creation +roverlay.ebuild.depresprepare (collect), +balance (drop dependencies, +ebuild creation failure)
                              roverlay.overlay.root +roverlay.overlay.category +roverlay.overlay.pkgdirprepare (link), +balance (remove packages)
                              roverlay.depres.depresultall (selfdep data object)
                              roverlay.packageinfoall (package data object),
                            @@ -4073,7 +4320,7 @@ all required dependencies. No ebuild can be created in that case.

                            Details about dependency resolution like how channels work can be found in the following subsections.

                            -

                            14.6.1   Dependency types

                            +

                            14.6.1   Dependency types

                            Every dependency string has a dependency type that declares how a dependency should be resolved. It has one or more of these properties:

                            @@ -4282,13 +4529,15 @@ while <dependencies queued for resolution> else if <a rule pool accepts depenv's type and resolves depenv> + resolved <= True else if <depenv's type contains TRY_OTHER> - if <a rule pool supports TRY_OTHER and does not accept depenv's type and resolves depenv> resolved <= True + + end if end if end if @@ -4315,7 +4564,7 @@ becomes "loop until resolver closes".