From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path:
-
-
-
-
-
-
-
+
-
-
-
-
-
@@ -976,21 +977,19 @@ most cases, but fine-tuning can be done via a wrapper around /bin/sh that runs a shell or shell script in roverlay's
hook environment.
+
roverlay-related scripts can use it to automatically inherit roverlay's -config and $FUNCTIONS file:
-+config and get access to its $FUNCTIONS file: +#!/usr/bin/roverlay-sh -# reset DEBUG, VERBOSE, QUIET -DEBUG=n; QUIET=n; VERBOSE=y - # load the functions file (optional) . "${FUNCTIONS?}" || exit # script body true-<<TODO: maybe there's a better place for the details>>
+
This is your best bet if the remote is a repository but does not offer rsync access. Basic digest verification is supported (MD5). The remote has to have a package list file, typically named PACKAGES, -with a special syntax (debian control file syntax).
+with a special syntax (debian control file syntax). Syncing is retried up to +3 times, for example if a connection timeout occurs or a remote file +disappears after reading the package list file.A package list example, excerpt from omegahat's PACKAGES file (as of Aug 2012):
@@ -1453,7 +1454,6 @@ reduces its size considerably.The additions directory is a directory with overlay-like structure that contains extra files for overlay creation. Currently, ebuild patches and ebuild files are supported.
-To give an idea of how this directory could
6.1 Patching ebuilds
Patches can apply to a specific version or to all versions of a @@ -1496,13 +1496,13 @@ for collecting dependencies.
Foreign ebuilds can be imported into the overlay by simple putting them into the additions directory.
The naming convention is similar to ebuild patches and identical to the -portage tree:
--${CATEGORY}/${PN}/${PF}.ebuild --Ebuilds imported that way can not be overwritten by generated ebuilds and +portage tree, ${CATEGORY}/${PN}/${PF}.ebuild.
+Ebuilds imported that way cannot be overwritten by generated ebuilds and benefit from most overlay creation features, e.g. Manifest file creation. However, they cannot be used for metadata creation.
+The ${CATEGORY}/${PN}/metadata.xml file will be imported if it exists and +if ${PN} in the overlay (a) has no metadata.xml file or (b) has no +generated ebuilds (only imports).
@@ -1781,7 +1781,9 @@ colong char ::start with #. There are a few exceptions to that, the #deptype and #! NOPARSE keywords. Comments inside rule blocks are not allowed and will be read as normal dependency strings. -Selfdep + ++
+- Selfdep
This is a classification for dependencies on packages which are also part of the overlay being created. Typically, selfdeps refer to other R packages, but there may be a few exceptions.
@@ -2907,6 +2909,13 @@ This will pass '--bwlimit=<value>' to all rsync commands.Defaults to <not set>, which disables bandwidth limitation.
+
- PORTDIR
+- +
Path to the portage tree. This option is recommended, but not always +required.
+Defaults to "/usr/portage".
++10.2 overlay options
@@ -3342,8 +3351,41 @@ files will be kept.+10.6 license map options
++
+- CREATE_LICENSE_FILE
+- Alias to CREATE_LICENSES_FILE.
++
+- CREATE_LICENSES_FILE
+- +
Create the CACHEDIR/license file after scanning PORTDIR/licenses for +licenses. This file can serve as fallback if PORTDIR is not available +or if USE_PORTAGE_LICENSES is set to no.
+Defaults to yes.
++
+ +- LICENSE_MAP
+- +
Path to the license map file, which is used to translate license strings +into valid licenses (accepted by portage). Its syntax is similar to +dependency rules.
+This option is not required, but recommended if the +field definition config file contains a field with the isLicense flag.
+Defaults to <not set>.
+-10.6 options for debugging, manual dependency rule creation and testing
+10.7 options for debugging, manual dependency rule creation and testing
- DESCRIPTION_DIR
A directory where all description data read from an R package will be @@ -3528,178 +3570,262 @@ such a field is found.
12 Roverlay Console
-<<section is TODO>>
-<<links to depres console chapter may need to be fixed>>
-<<basic commands, table>>
-<<note regarding python -OO and missing help texts>>
+roverlay provides an interactive console for accessing certain subsystems, +e.g. dependency resolution. Its features include tab completion for filesystem +paths and a command history that supports navigation with the arrow keys.
+The console also implements a subset of typical tools like cat, which +behave similar but not identical to their counterparts.
+See the following sections for how to run specific consoles.
+The following table lists all basic commands, which are available in all +roverlay consoles. Some commands have more detailed help messages, which are +printed when running <cmd> --help.
++
+basic console commands (subsystem-independent) ++ + ++ + + + + + command +extended +--help +description ++ exit, quit, q, qq +no +exit ++ help, ? +no +print help message (list known commands) ++ help cmd +no +show command-specific help message ++ alias +no +show command aliases ++ unalias +no +unset command aliases (not implemented) ++ cat +yes +read files and print them (supports compressed +files) ++ cd +no ++ change working directory
+Actually, this does not change the working +directory. It simply sets the prefix which is used +when dealing with relative filesystem paths.
++ chroot +no ++ enter/leave/show command chroot
+A command chroot prefixes all input with a +specific command.
++ declare +no +show variables ++ echo +no +print text (supports python string formatting) ++ exec +no +switch to another subsystem (not implemented) ++ history +no +show command history ++ ls +no +print directory content ++ pwd +no +print current working directory ++ set +no +set variables ++ + unset +no +unset variables ++Note
+Running the console with python -OO removes most of the help messages.
+12.1 Dependency Resolution Console
--Warning
-This section is out-of-date.
-As previously stated, the DepRes Console is only meant for testing. It is an interactive console with the following features:
- resolve dependencies
- create new dependency rules
-- create dependency rules for each R package found in a directory
- load rules from files
- save rules to a file
Rules are managed in a set. These so-called rule pools are organized in -a first-in-first-out data structure that allows -to create or remove them easily at runtime.
-Running roverlay depres_console will print a welcome message that -lists all available commands and a few usage hints.
-For reference, these commands are currently available:
-<<TODO: rewrite/update command table, it's out-of-date>>
+a first-in-first-out data structure that allows to create or remove +them easily at runtime. +Running roverlay depres_console prints a short welcome message and starts +the console.
+For reference, these commands are available (in addition to the basic ones):
+
-depres commands - - + + + - command +extended +--help description - help, -h, ? -lists all commands -- help <cmd>, -h <cmd> -prints a command-specific help message -- load <file|dir>, -l <file|dir> -loads a rule file or a directory with rule files -into a new rule pool -- load_conf, -lc -loads the rule files listed in the config file -into a new rule pool -add_rule <rule> -+ <rule> ++ add_pool, << +no +creates a new, empty rule pool +- add_rule, + +no creates a new rule and adds it to the topmost, i.e. latest rule pool. This command uses Rule File Syntax. Multi line rules are supported. add_pool, -<< -creates a new rule pool +- load, l +yes +load rule files unwind_pool, ->> -removes the topmost rule pool and all of its -rules ++ load_conf, lc +no +load configured rule files +- print_pool, print, +p +yes +print one or more rule pools resolve <dep>, -?? <dep> ++ resolve, ?? +no tries to resolve the given dependency string and prints the result + unwind_pool, >> +yes +removes the topmost rule pool and all of its +rules +write, w +yes +writes rules to a file +! -enter the resolve command chroot -all input will be prefixed with "resolve" +no +enter the resolve command chroot, which prefixes +all input with resolve - !! +no leave any command chroot - print, p -prints the rules of the topmost rule pool -- print --all|-a -prints the rules of all rule pools -- print <id> [<id>..] -prints the rules of a specific rule pool -- write <file>, -w <file> -writes the rules of the topmost rule pool into -<file>. See write --help for advanced usage. -- cd <dir> -changes the working directory -This is a virtual command. <<TODO:EXPLAIN>> -- set VAR=VALUE -set variables -- unset VAR -unset variables -- declare -show declared variables -- alias -show command aliases -- unalias -reserved for future usage -exit, qq, q -exits the DepRes Console ---Note
-Some commands also offer more detailed help via <command> --help.
-<<TODO>>
-<<Example Session is out-of-date>>
@@ -3708,7 +3834,7 @@ cmd % exit
- Example Session:
-== depres console == -Run 'help' to list all known commands -More specifically, 'help <cmd>' prints a help message for the given -command, and 'help --list' lists all help topics available -Use 'load_conf' or 'lc' to load the configured rule files - -commands: load, unwind, set, help, >>, load_conf, <<, cd, mkhelp, -resolve, lc, add_pool, addrule, h, +, l, li, write, p, r, ?, w, print, -sd, unset, scandir -exit-commands: q, qq, exit +[roverlay depres_console] + +== dependency resolution console (r2) == +Run 'help' to list all known commands. +More specifically, 'help <cmd>' prints a help message for +the given command, and 'help --list' lists all help topics. +Use 'load_conf' or 'lc' to load the configured rule files. + +cmd % help + +Documented commands (type help <topic>): +======================================== +EOF cat echo history print qq unalias +add_pool cd exec load print_pool quit unset +add_rule chroot exit load_conf pwd resolve unwind_pool +alias declare help ls q set write + +cmd % <tab><tab> +EOF chroot history pwd unalias +add_pool declare load q unset +add_rule echo load_conf qq unwind_pool +alias exec ls quit write +cat exit print resolve +cd help print_pool set cmd % + ~dev-lang/R :: R language -new rules: + +cmd % print --help +usage: print_pool [-h] [--all] [<id> [<id> ...]] + +positional arguments: + <id> print specific pools (by id) + +optional arguments: + -h, --help show this help message and exit + --all, -a print all pools + +cmd % print -a ~dev-lang/R :: R language ---- --- -command succeeded. -cmd % ? R language -Trying to resolve ('R language',). -Resolved as: ('dev-lang/R',) +cmd % ! + +(resolve) % R language +'R language' has been resolved as dev-lang/R. -cmd % ? R language [ 2.15 ] -Trying to resolve ('R language [ 2.15 ]',). -Resolved as: ('>=dev-lang/R-2.15',) +(resolve) % R language [ 2.15 ] +'R language [ 2.15 ]' has been resolved as >=dev-lang/R-2.15. -cmd % ? R -Trying to resolve ('R',). -Channel returned None. At least one dep could not be resolved. +(resolve) % R +'R' could not be resolved. + +(resolve) % !! cmd % p ~dev-lang/R :: R language cmd % >> -Pool removed from resolver. +pool has been removed. + +cmd % >> +resolver has no pools. cmd % p -cmd % ? R language -Trying to resolve ('R language',). -Channel returned None. At least one dep could not be resolved. -cmd % exit +cmd % ?? R language +'R language' could not be resolved. + +cmd % set PS1=#! + +#! exit13 Roverlay Interface
Roverlay provides an API for accessing its functionality independently of -R overlay creation. Only dependency resolution is available, currently.
+R overlay creation.Note, however, that a minimal config file may still be required for accessing roverlay interfaces.
The table below lists all interfaces and where to find them:
@@ -3775,7 +3901,16 @@ for usage examples.13.2 Remote Interface
-<<TODO; this interface isn't mature enough yet (it will likely change in future)>>
+The RemoteInterface is experimental/incomplete and currently offers the +following functionality:
++
+- set sync mode to online/offline
+- sync/nosync
+- load repo config
+- list repos
+- list repo packages
+Refer to in-code documentation for details.
@@ -4002,12 +4137,12 @@ would require variable interpolation, e.g. for ${PN14.4 Ebuild Creation
Ebuild creation is the process centered around one PackageInfo instance p that tries to create an ebuild for it.
-It does the following steps:
+An EbuildCreationJob does the following steps:
- Read the DESCRIPTION file of p R package tarball and stores the data in an associative array ('DESCRIPTION field' -> 'data')
- Call dependency resolution
-- If dependency resolution was successful and any selfdeps found +
- If dependency resolution was successful and any selfdeps found (dependencies on other packages): pause ebuild creation for p until it has been verified whether these dependencies are satisfiable. This is necessary because dependency resolution does not know whether a @@ -4045,9 +4180,121 @@ for a PackageInfo object and inform the OverlayCreation about afterwards. Overlay creation keeps going if an ebuild cannot be created, instead the event is logged. Running more than one OverlayWorker in parallel is possible. +
The following pseudo-code illustrates how overlay creation basically works:
++ACCEPT_PACKAGES: + +for each received PackageInfo <p> + + create an EbuildCreationJob for p and add it to the work queue + +end for + + + +CREATE_OVERLAY: + +while work_queue is not empty + + work_queue_next <= empty + + + in parallel with N OverlayWorkers (>=0 threads): + + for each EbuildCreationJob ebuild_job from the work_queue + + run/resume ebuild_job (as described in Ebuild Creation) + + if ebuild_job is paused + + add ebuild_job to work_queue_next + + end if + + end for + + end in parallel + + + if work_queue_next is not empty + + <run selfdep validation> + + work_queue <= work_queue_next + + end if + +end while +14.5.1 Selfdep Validation
-<<TODO: specify algo here>>
+EbuildCreationJobs are processed in no specific order and possibly +concurrently. This leads to the problem that dependency resolution cannot +know whether a successfully resolved selfdep is actually satisfiable.
+For example, if a package A depends on another package B and B is +uncreatable (ebuild creation will fail for B), then dependency resolution +still resolves B (e.g. as sci-R/B). A has a dangling selfdep now.
+Selfdep validation is the process of identifying dangling selfdeps and +removing packages with such dependencies or simply dropping +the dependencies, depending on their dependency type.
+It is an algorithm with 3 phases:
++
+- prepare: create a graph containing all selfdeps
++
+- collect: find ebuild creation jobs with selfdeps (direct selfdeps)
+- link: expand the selfdep graph, find selfdep-selfdep dependencies +(indirect selfdeps)
+- reduce: find dangling selfdeps and mark them as invalid
++
+- a selfdep is dangling iff the overlay contains no suitable +PackageInfo with valid selfdeps
+- repeat this step until no more dangling selfdeps found
+- balance: find ebuild creation jobs with invalid selfdeps (inversed collect)
++
+- drop optional dependencies
+- let ebuild creation fail if a mandatory selfdep is not valid
+The actual implementation in roverlay is spread across several modules:
++
modules/packages involved in selfdep validation ++ + ++ + + + + module/package +phase(s) / description ++ roverlay.overlay.creator +all (controls selfdep validation), +especially collect and +reduction loop ++ roverlay.ebuild.creation +roverlay.ebuild.depres +prepare (collect), +balance (drop dependencies, +ebuild creation failure) ++ roverlay.overlay.root +roverlay.overlay.category +roverlay.overlay.pkgdir +prepare (link), +balance (remove packages) ++ roverlay.depres.depresult +all (selfdep data object) ++ + roverlay.packageinfo +all (package data object), +@@ -4073,7 +4320,7 @@ all required dependencies. No ebuild can be created in that case.Details about dependency resolution like how channels work can be found in the following subsections.
-14.6.1 Dependency types
+14.6.1 Dependency types
Every dependency string has a dependency type that declares how a dependency should be resolved. It has one or more of these properties:
@@ -4282,13 +4529,15 @@ while <dependencies queued for resolution> else if <a rule pool accepts depenv's type and resolves depenv> + resolved <= True else if <depenv's type contains TRY_OTHER> - if <a rule pool supports TRY_OTHER and does not accept depenv's type and resolves depenv> resolved <= True + + end if end if end if @@ -4315,7 +4564,7 @@ becomes "loop until resolver closes".