source: BLFS/libs/func_dependencies@ f1edc8e

ablfs-more legacy trunk
Last change on this file since f1edc8e was f1edc8e, checked in by Pierre Labastie <pierre@…>, 7 years ago

Because we do not want to depend on <pack>groupxx if it creates a circular

dependency, it may happen that nothing depends on <pack>groupxx. Detect and
fix this by appending <pack>groupsxx to root.dep

  • Property mode set to 100644
File size: 19.5 KB
RevLine 
[3c10176]1#!/bin/bash
2#
3# $Id$
4#
5
[2140f22]6#-----------------------------------------------------------------------------#
7# This is a set of (recursive) functions for manipulating a dependency graph. #
8# We use algorithms and definitions from chapter 4 (mainly section 4.2) of #
9# https://algs4.cs.princeton.edu/. The graph we manipulate is the directed #
10# graph of the dependencies: nodes are packages in the BLFS book. A node A is #
11# connected to a node B if package A depends on B. A topological order (rev- #
12# erted) is exactly what we want for a build order. But a topological order #
13# only exists if the graph is acyclic. We'll therefore have to remove cycles. #
14# There are a number of other features we want to consider: #
15# - edges are weighted according to the dependency requirement: #
16# 1 for required #
17# 2 for recommended #
18# 3 for optional #
19# 4 for external #
20# We should consider only edges with weight lower or equal to that #
21# specified by the user, but see below. #
22# - we do not want to build the whole book. The user specifies a set of #
23# packages, and we have to consider only nodes reachable from this set #
24# using edges of weight not exceeding the specified weight. #
25# - when doing the topological sort, we want to consider all the edges and #
26# not only those not exceeding the specified weight: If a package A in the #
27# reachable subgraph depends optionally on another package B in the same #
28# subgraph, we want to build B before A. But this means we'll have to #
29# remove cycles for all weights. #
30# - dependencies have another qualifier: before or after. The problem: if a #
31# package A depends on B with an "after" qualifier, and a package C depends #
32# on A with a "before" qualifier, C may need B to be able to use A. So the #
33# only safe way to consider "after" qualifiers is to consider that they are #
34# "before" deps for any parent of the packages considered. #
35# We'll therefore have a 3 pass procedure. First build the set of nodes #
36# reachable from the root set. Second, remove dangling edges (those pointing #
37# to packages outside the node set), and move "after" edges to "before" edges #
38# originating from the parents. Third remove cycles and generate a #
39# topological sort. #
40# #
41# TODO: document each pass
42# Data layout: #
43# TODO: needs also to document the .tree files, and the "f" qualifier
44# #
45# A node of the tree is represented by a text file <nodeName>.dep. Each edge #
46# starting from this node is represented by a line in this file. We keep #
47# those files in the same directory. We introduce a special node named root, #
48# whose edges point to the list of nodes requested by the user. Each line #
49# contains three fields: #
50# - the weight of the edge #
51# - the "before" (b) or "after" (a) qualifier #
52# - the name of the destination of the edge #
[7dc8595]53# #
[2140f22]54# TODO: The following is obsolete
[d7c07e0]55# Circular dependencies: #
[7dc8595]56# #
[d7c07e0]57# In case we find a cirdular dependency, it has the form : #
58# parent->dependency_0->...->dependency_n->dependency_0 #
59# If we want to build dependency_n before dependency_0, no problem: #
60# we just prune the tree at dependency_n. If we want to build first #
61# dependency_0, we need to put dependency_n as a dependency of parent, #
62# then erase and rebuild the subtree from there. Now, we may have met #
63# another circular dependency in the subtree, and erasing the tree makes #
64# us forget the decision which was made. So, after first generating the #
65# list of dependencies from packages.xml, we keep the generated list in #
66# a file <nodeName>.odep, which we modify according to the decision which #
67# was made. #
[e4b1293]68#---------------------------------------------------------------------------#
69
70# Global variables:
[e576789]71# A string of spaces for indenting:
[3c10176]72declare -a spaceSTR=" "
[2140f22]73# When we are backing up from a circular dependency, `parentNode'
74# contains the node which has an edge entering the cycle
75declare parentNode
[3c10176]76
[2140f22]77#---------------------#
78generate_subgraph() { #
79#---------------------#
[3c10176]80: <<inline_doc
[2140f22]81 function: Create a subgraph of all the nodes reachable from the node
82 represented by the file whose name is $1. The edges considered
83 are those with maximal weight DEP_LEVEL (recursive function).
84 input vars: $1 : file name corresponding to the node whose edges will be
85 followed for the DFS
86 $2 : weight of the edge leading to this node
87 $3 : depth (root is 1)
88 $4 : qualifier (a for after, b for before)
[7ac11f0]89 externals: vars: DEP_LEVEL contains 1 if we want to build the
90 tree only for required dependencies,
91 2 if we want also recommended ones,
[2140f22]92 3 if we want also optional ones, but only
93 for the requested packages,
94 4 if we want all the dependencies
[7ac11f0]95 MAIL_SERVER contains the name of the MTA we want to use.
96 files: ../xsl/dependencies.xsl: stylesheet for creating the
97 .dep files
[e91a15d]98 ../packages.xml: File containing packages id
[7ac11f0]99 and dependencies
[2140f22]100 returns: 0 if the tree has been successfully created
101 output: files: for each node reachable from $1, a file <node>.dep.
102 on error: nothing
103 on success: nothing
104inline_doc
105
106local depFile=$1
107local -i weight=$2
108local -i depth=$3
109local qualifier=$4
110local -i spacing=0
111local priostring
112local buildstring
113local id_of_dep
114local prio_of_dep
115local build_of_dep
116local dep_level
117
118if (( depth < 10 )); then spacing=1; fi
119case $weight in
120 1) priostring=required ;;
121 2) priostring=recommended ;;
122 3) priostring=optional ;;
123esac
124case $qualifier in
125 a) buildstring=runtime ;;
126 b) buildstring= ;;
127esac
128dep_level=$DEP_LEVEL
129if [ "$dep_level" = 3 ] && [ "$depth" -gt 2 ]; then dep_level=2; fi
130if [ "$dep_level" -gt 3 ]; then dep_level=3; fi
131echo -en "\nNode: $depth${spaceSTR:0:$(( depth + spacing ))}${RED}${depFile%.dep}${OFF} $priostring $buildstring"
132
133depth=$(( depth + 1 ))
134if (( depth < 10 )); then spacing=1; else spacing=0; fi
135# Start of loop
136{
137while read prio_of_dep build_of_dep id_of_dep; do
138case $prio_of_dep in
139 1) priostring=required ;;
140 2) priostring=recommended ;;
141 3) priostring=optional ;;
142 4) priostring=external ;;
143esac
144case $build_of_dep in
145 a) buildstring=runtime ;;
146 b) buildstring= ;;
147esac
148# Has this entry already been seen
149 if [ -f ${id_of_dep}.dep ] ; then
150# Just display it and proceed.
151 echo -en "\nEdge: $depth${spaceSTR:0:$((depth + spacing))}${MAGENTA}${id_of_dep}${OFF} $priostring $buildstring"
152 continue
153 fi
154# Is the weight higher than requested?
155 if [ "$prio_of_dep" -gt $dep_level ]; then
156# Just display it and proceed.
157 echo -en "\n Out: $depth${spaceSTR:0:$((depth + spacing))}${YELLOW}${id_of_dep}${OFF} $priostring $buildstring"
158 continue
159 fi
160# Otherwise, let's build the corresponding subgraph.
161 xsltproc --stringparam idofdep "$id_of_dep" \
162 --stringparam MTA "$MAIL_SERVER" \
163 -o ${id_of_dep}.dep \
164 ../xsl/dependencies.xsl ../packages.xml
165
166 if [[ -s ${id_of_dep}.dep ]]; then # this dependency has dependencies
167 generate_subgraph ${id_of_dep}.dep $prio_of_dep $depth $build_of_dep
168 else # id_of_dep has no dependencies, just touch the file and display
169 touch ${id_of_dep}.dep
170 echo -en "\nLeaf: $depth${spaceSTR:0:$((depth + spacing))}${CYAN}${id_of_dep}${OFF} $priostring $buildstring"
171 fi
172done
173} <$depFile
174depth=$(( depth - 1 ))
175if (( depth < 10 )); then spacing=1; else spacing=0; fi
176echo -en "\n End: $depth${spaceSTR:0:$((depth + spacing))}${GREEN}${depFile%.dep}${OFF}"
177return 0
178}
179
180#-----------#
181path_to() { #
182#-----------#
183: <<inline_doc
184 function: check whether there is a path from $1 to $2 on the graph
185 input vars: $1 contains the filename of the starting node.
186 $2 contains the name of the node to reach
187 $3 contains the weight above which we do not want to
188 follow an edge
189 "seen" (global) contains the list of already seen nodes
190 returns: 0 if the node has been found
191 1 if not
192 on error: nothing
193 on success: nothing
194inline_doc
195local start=$1
196local seek=$2
197local prio=$3
198local prio_of_dep
199local build_of_dep
200local id_of_dep
201local r
202
203if test "${start%.dep}" = "$seek"; then return 0; fi
[59afe73]204seen="$seen${start%.dep} "
[5afaf7c]205if test -s $start; then
[2140f22]206 {
207 while read prio_of_dep build_of_dep id_of_dep; do
208 if test "$prio" -lt "$prio_of_dep"; then continue; fi
[5afaf7c]209 if ! test "${seen% $id_of_dep *}" = "$seen"; then continue; fi
[2140f22]210 if path_to ${id_of_dep}.dep $seek $prio; then return 0; fi
211 done
212 } < $start
213fi
214return 1
215}
216#------------------#
217clean_subgraph() { #
218#------------------#
219: <<inline_doc
220 function: Remove dangling edges and create groups of deps for "after"
221 deps: A-before->B-after->C becomes:
222 A -before-> Bgroupxx -before-> B
223 \
224 -before-> C
225 the name of the group is chosen so that it is unlikely as
226 a package name (so that it is removed when building the
227 xml book).
228 input vars: None
229 files: <node>.dep files containing dangling edges and
230 "after" qualifiers
231 returns: 0
232 output: files: <node>.dep files containing no dangling edges and
233 no "after" qualifiers
234 on error: nothing
235 on success: nothing
236inline_doc
237
238local node
239local id_of_dep
240local prio_of_dep
241local build_of_dep
242local lines_to_remove
243local lines_to_change
244local parent
245local p
246local b
247local start
248local seen
249
250for node in $(ls *.dep); do
251 if test $node = root.dep; then continue; fi
252 lines_to_remove=
253 {
254 while read prio_of_dep build_of_dep id_of_dep; do
255 if ! test -f ${id_of_dep}.dep; then
256 lines_to_remove="$lines_to_remove $id_of_dep"
257 continue
258 fi
259 done
260 } <$node
261 for id_of_dep in $lines_to_remove; do
262 sed "/\ $id_of_dep\$/d" -i $node
263 done
264done
265for node in $(grep -l ' a ' *.dep); do
266 lines_to_remove=
267 if ! [ -e ${node%.dep}groupxx.dep ]; then
[f1edc8e]268 b=0
[2140f22]269 for parent in $(grep -l ${node%.dep}\$ *); do
270 p=0
271 for start in $(grep ' a ' $node | cut -d' ' -f3); do
272 seen=" "
273 if path_to ${start}.dep ${parent%.dep} 3; then p=1; break; fi
274 done
275 if test $p = 0; then
[f1edc8e]276 b=1
[2140f22]277 sed -i "s/\ ${node%.dep}\$/&groupxx/" $parent
278 fi
279 done
280 echo "1 b ${node%.dep}" > ${node%.dep}groupxx.dep
[f1edc8e]281 if test $b = 0; then echo "1 b ${node%.dep}groupxx" >> root.dep; fi
[2140f22]282 fi
283 {
284 while read prio_of_dep build_of_dep id_of_dep; do
285 if test $build_of_dep = a; then
286 if ! grep -q ${id_of_dep} ${node%.dep}groupxx.dep; then
287 echo "$prio_of_dep b ${id_of_dep}" >> ${node%.dep}groupxx.dep
288 fi
289 lines_to_remove="$lines_to_remove $id_of_dep"
290 fi
291 done
292 } <$node
293 for id_of_dep in $lines_to_remove; do
294 sed "/\ $id_of_dep\$/d" -i $node
295 done
296done
297for node in $(grep -l ' f ' *); do
298 lines_to_change=
299 {
300 while read prio_of_dep build_of_dep id_of_dep; do
301 if test $build_of_dep = f; then
302 if ! test -f ${id_of_dep}-pass1.dep; then
303 cp ${id_of_dep}{,-pass1}.dep;
304 fi
305 lines_to_change="$lines_to_change $id_of_dep"
306 unset lr
307 {
308 while read p b start; do
309 seen=" "
310 if path_to ${start}.dep ${node%.dep} $p; then
311 lr="$lr $start"
312 fi
313 done
314 } < ${id_of_dep}-pass1.dep
315 for p in $lr; do
316 sed "/\ $p\$/d" -i ${id_of_dep}-pass1.dep
317 done
318 fi
319 done
320 } <$node
321 for id_of_dep in $lines_to_change; do
[d80302b]322 sed "/\ $id_of_dep\$/"'{s/[[:digit:]] f /1 b /;s/$/-pass1/}' -i $node
[2140f22]323 done
324done
325} # End clean_subgraph
326
327#----------------------------#
328generate_dependency_tree() { #
329#----------------------------#
330: <<inline_doc
331 function: Create a subtree of the dependency tree
332 (recursive function)
333 input vars: $1 : file with a list of targets (child nodes)
334 the first line of the file is the link
335 $2 : priority (1=req, 2=rec, 3=opt)
[e4b1293]336 returns: 0 if the tree has been successfully created
337 1 if we are backing up to the parent of a circular dep
[2140f22]338 and there are only required deps in the cycle
339 2 if we are backing up to the parent of a circular dep
340 and there are recommended deps and no optional deps in the
341 cycle
342 3 if we are backing up to the parent of a circular dep
343 and there are optiional deps in the cycle
344 modifies: vars: ParentNode is set when return is not 0
[e4b1293]345 output: files: for each <pkg> with dependencies in $1,
[2140f22]346 a file <pkg>.tree and its dependencies
[3c10176]347 on error: nothing
348 on success: nothing
349inline_doc
350
[2140f22]351local depFile=$1
[7dc8595]352local priority=$2
[e576789]353local -a rootlink
[7dc8595]354local -a priolink
[e576789]355local -a otherlink
356local -i depth
357local -i count=0
358local id_of_dep
[2140f22]359local build_of_dep
360local prio_of_dep
[e576789]361local parent
362local lines_to_remove=
363local srootlink
[7dc8595]364local priostring
365local dpriostring
366local i
[e576789]367
368{
[2140f22]369read -a rootlink
[e576789]370depth=${#rootlink[*]}
[2140f22]371read -a priolink
[e576789]372srootlink="${rootlink[*]} "
[7dc8595]373case $priority in
374 1) priostring=required ;;
375 2) priostring=recommended ;;
376 3) priostring=optional ;;
377esac
[2140f22]378# start of depFile
379echo -en "\nNode: $depth${spaceSTR:0:$depth}${RED}${depFile%.tree}${OFF} $priostring"
[e576789]380
[2140f22]381while read prio_of_dep build_of_dep id_of_dep; do
382 case $prio_of_dep in
[7dc8595]383 1) dpriostring=required ;;
384 2) dpriostring=recommended ;;
385 3) dpriostring=optional ;;
[2140f22]386 esac
[e576789]387# count entries in file
388 (( count++ ))
389# Has this entry already been seen?
[2140f22]390 if [ -f ${id_of_dep}.tree ]; then # found ${id_of_dep}.tree already in tree
391 otherlink=($(head -n 1 ${id_of_dep}.tree))
[2e1c1c3]392 if [ -z "${otherlink[*]}" ]
[2140f22]393 then echo otherlink empty for $id_of_dep.tree
[e576789]394 echo This should not happen, but happens to happen...
395 exit 1
396 fi
[2140f22]397#Do not use "${rootlink[*]}" =~ "${otherlink[*]}": case rootlink=(1 11)
[e576789]398# and otherlink=(1 1)
[e4b1293]399 if [[ ${srootlink#"${otherlink[*]} "} != ${srootlink} ]]; then # cir. dep
[7dc8595]400 echo -en "\nCirc: $((depth+1))${spaceSTR:0:$((depth+1))}${YELLOW}${id_of_dep}${OFF} $dpriostring"
[2140f22]401# Find lowest priority in branch from parent to depFile:
[7dc8595]402 p2=0
403 for (( i=${#otherlink[*]}; i < $depth ; i++ )) ; do
404 if (( ${priolink[i]} > $p2 )); then p2=${priolink[i]}; fi
405 done
406 if (( $prio_of_dep >= $p2 )); then # prune
[e4b1293]407 lines_to_remove="$lines_to_remove $id_of_dep"
[2140f22]408 sed -i "/$id_of_dep/d" ${depFile/.tree/.dep}
409 else # find and set parent, then return lowest priority
410# The parent has the same link without the last entry.
411# We do not need otherlink anymore so just destroy the last element
412 unset otherlink[-1]
413 parentNode=$(grep ^"${otherlink[*]}"\$ -l *)
414 return $p2
[e4b1293]415 fi
[2140f22]416 else # not circular: prune tree (but not .dep, since it may happen that
[7ac11f0]417 # the tree is destroyed and rebuilt in another order)
[e576789]418 lines_to_remove="$lines_to_remove $id_of_dep"
[e4b1293]419 fi # circular or not
420 continue # this dependency has already been seen, and the associated
421 # subtree computed. We are done
422 fi # Has this entry already been seen?
[2140f22]423# So, this entry has not already been seen. Let's build the corresponding
424# subtree. First check there is a subtree...
425# Use -s, because it may happen that after removing lines, .dep exists
[d7c07e0]426# but is empty.
[2140f22]427 if [[ -s ${id_of_dep}.dep ]]; then # this dependency has dependencies
428 sed "1i${rootlink[*]} $count\\
429${priolink[*]} $prio_of_dep" < ${id_of_dep}.dep \
430 > ${id_of_dep}.tree # add link and priolink
431 generate_dependency_tree ${id_of_dep}.tree $prio_of_dep
[e576789]432# Test return value, in case we exchange dependencies
[2140f22]433 p2=$?
434 case $p2 in
435 0) # Normal return
436 ;;
437 $prio_of_dep) # we remove this dep, but since it may become unreachable,
438 # move it to be built later (as a dep of parent).
439 tree_erase ${id_of_dep}.tree
440 lines_to_remove="$lines_to_remove $id_of_dep"
441 sed -i "/${id_of_dep}/d" ${depFile/.tree/.dep}
442 echo "$prio_of_dep b $id_of_dep" >> $parentNode
443# must be added to .dep in case parentNode is destroyed when erasing
444# the tree
445 echo "$prio_of_dep b $id_of_dep" >> ${parentNode/.tree/.dep}
446 continue
447 ;;
448 *) # We are backing up
449 return $p2
450 ;;
451 esac
452 else # id_of_dep has no dependencies, just record the link in a file
453 # and print
454 echo "${rootlink[*]} $count" > ${id_of_dep}.tree
455 echo -en "\nLeaf: $(($depth+1))${spaceSTR:0:$(($depth+1))}${CYAN}${id_of_dep}${OFF} $dpriostring"
456 fi
[e576789]457done
[2140f22]458echo -en "\n End: $depth${spaceSTR:0:$depth}${GREEN}${depFile%.tree}${OFF}"
459} <$depFile
[e576789]460# It may happen that a file is created with several times
461# the same line. Normally, all those lines but one
462# would be flagged to be removed (or all of them if
[e4b1293]463# the dependency appeared before). A simple sed /$line/d
[e576789]464# destroys all the lines. We should instead remove
465# only one for each appearance of it in lines_to_remove.
[e4b1293]466# so first get the position of last line and then delete
[e576789]467# that line
468for line in $lines_to_remove
[2140f22]469 do lineno=$(sed -n /^[[:digit:]]\ b\ $line\$/= $depFile | tail -n1)
470 sed -i ${lineno}d $depFile
[e576789]471done
472return 0
473}
[3c10176]474
[2140f22]475
[e576789]476#---------------#
477tree_browse() { #
478#---------------#
479local file=$1
480local f
481
482#echo file=$file
[7dc8595]483for f in $(grep '[^0-9 ]' $file | sed 's/.* //'); do
[e576789]484# echo f=$f
[2140f22]485 if grep -q '[^0-9 ]' ${f}.tree ; then
486 tree_browse ${f}.tree
[3c10176]487 fi
[e576789]488 echo $f
489done
490}
[3c10176]491
[e576789]492#--------------#
493tree_erase() { #
494#--------------#
495local file=$1
496local f
[6c8095a]497local rootlink
[f37d08b]498local rootlink2
[e576789]499
500#echo file=$file
[6c8095a]501rootlink="$(head -n1 $file) "
[7dc8595]502for f in $(grep '[^0-9 ]' $file | sed 's/.* //'); do
[2140f22]503 if [ -f ${f}.tree ]; then
504 rootlink2="$(head -n1 ${f}.tree) "
[6c8095a]505# We want two things:
506# i) do not erase the file if it is in another branch
507# ii) do not erase the file if there is a circular dependency
508# for case i), we test that rootlink is contained in rootlink2
509# for case ii), we test that rootlink2 is not contained in
510# rootlink.
[f37d08b]511# See comment above about srootlink
[6c8095a]512 if [[ ${rootlink2#${rootlink}} != ${rootlink2} &&
513 ${rootlink#${rootlink2}} == ${rootlink} ]] ; then
[2140f22]514 tree_erase ${f}.tree
[e576789]515 fi
516 fi
517done
518rm -f $file
[3c10176]519}
Note: See TracBrowser for help on using the repository browser.