1 | #!/bin/bash
|
---|
2 |
|
---|
3 | # $Id$
|
---|
4 | # Acknowledgment:
|
---|
5 | # The following code is a no-modified version (except for this comment,
|
---|
6 | # the `Id' comment, and the inline_doc fragment) of an original work written by
|
---|
7 | # Ken Moffat and is included here with his permission.
|
---|
8 | #
|
---|
9 |
|
---|
10 | # FARCE: Farce Assists Rebuild Comparison Evaluation ;)
|
---|
11 | #
|
---|
12 | # to answer the question "can it rebuild itself?"
|
---|
13 | #
|
---|
14 | # We expect four arguments - first directory path, filelist
|
---|
15 | # containing the files in this directory which we wish to compare,
|
---|
16 | # second directory path, filelist for second directory.
|
---|
17 | #
|
---|
18 | # Yes, we could just compare everything in each tree, but the
|
---|
19 | # filelist script knows about files it can reasonably ignore,
|
---|
20 | # and this also allows us to build a sytem, boot it and get a
|
---|
21 | # list of files, build a full desktop environment, and only then
|
---|
22 | # build and boot the "can it build itself" test system and get
|
---|
23 | # _its_ filelist.
|
---|
24 | #
|
---|
25 | # What this script aims to do:
|
---|
26 | # ____________________________
|
---|
27 | #
|
---|
28 | # First, report files not in both builds.
|
---|
29 | #
|
---|
30 | # Then, confirm symlinks point to same targets.
|
---|
31 | #
|
---|
32 | # After that, compare individual files -
|
---|
33 | # if different, run the file name through 'expected'
|
---|
34 | # to pick out files that are unlikely to match (logs,
|
---|
35 | # pids, fstab [assumes '/' is a different device each time],
|
---|
36 | # count these as 'expected'.
|
---|
37 | #
|
---|
38 | # For whatever is left, check the file type - ar archives
|
---|
39 | # have their members extraced and compared (every member has
|
---|
40 | # a timestamp), gzipped files are compared beyond their
|
---|
41 | # timestamp, binaries, at least those using shared libs or
|
---|
42 | # which are shared objects, are copied and subjected to
|
---|
43 | # --strip-debug. If files match at this stage, count them as
|
---|
44 | # 'accepted'.
|
---|
45 | #
|
---|
46 | # As a last step for any file that doesn't match, copy it
|
---|
47 | # through some perl regexps to "process" it (convert any
|
---|
48 | # date, time, kernel-version information from standard formats
|
---|
49 | # into tokens, then see if the tokensi match.
|
---|
50 | #
|
---|
51 | # For details of the regexps, see the tokenize function.
|
---|
52 | # Those files that match after this are also counted as
|
---|
53 | # 'accepted'. Note that I don't always start from the kernel
|
---|
54 | # version that I'm going to build, so this copes with e.g. perl
|
---|
55 | # files that hardcode the kernel version.
|
---|
56 | #
|
---|
57 | # We now have files that don't match. A few of these seem to be
|
---|
58 | # common to all builds - some (members of) c++ libraries or ar
|
---|
59 | # archives, a few programs which perhaps use some sort of c++ code).
|
---|
60 | # The file name # is passed to the 'failure' function - these
|
---|
61 | # recognized filenames are labelled as 'predictable FAIL:',
|
---|
62 | # anything else is labelled as 'unexpected FAIL:'.
|
---|
63 | #
|
---|
64 | # output:
|
---|
65 | # stderr - files only in one of the builds, failure messages,
|
---|
66 | # and totals.
|
---|
67 | #
|
---|
68 | # farce-results - more details, including which files were treated
|
---|
69 | # as expected differences, files where neither copy could be read,
|
---|
70 | # files treated as accepted, with the reason (and member for ar
|
---|
71 | # archives). This data is typically up to 100 characters wide -
|
---|
72 | # sometimes it's a bit more, but it doesn't wrap too badly in a
|
---|
73 | # 100 character xterm.
|
---|
74 | #
|
---|
75 | # farce-extras - diffs for the files, or members, that didn't
|
---|
76 | # match. This file is to establish new regexps for picking up
|
---|
77 | # date/time/kernel-version formats.
|
---|
78 | #
|
---|
79 | # farce-identical - the names of the files which are identical
|
---|
80 | #
|
---|
81 | # farce-substitutions - whenever using tokenizeanddiff results in a
|
---|
82 | # difference being accepted, for both versions diff the before and
|
---|
83 | # after versions to show what got changed. If the file is a binary,
|
---|
84 | # the output may still be hard to read. Note that I _know_ glibc
|
---|
85 | # version strings pass one of the regexps looking for a kernel version
|
---|
86 | # - since I expect you to use the same version of glibc for each
|
---|
87 | # build, this is not a problem.
|
---|
88 | #
|
---|
89 | # farce-differ - the names of the files which could not be treated
|
---|
90 | # as matching (whether or not I regard the failure as predictable)
|
---|
91 | # for possible input to ICA processing.
|
---|
92 | #
|
---|
93 | # Copyright (C) 2005, 2006 Ken Moffat <ken@linuxfromscratch.org>
|
---|
94 | #
|
---|
95 | # All rights reserved.
|
---|
96 | #
|
---|
97 | # This program is free software; you can redistribute it and/or modify
|
---|
98 | # it under the terms of the GNU General Public License as published by
|
---|
99 | # the Free Software Foundation; either version 2 of the License, or (at
|
---|
100 | # your option) any later version.
|
---|
101 | #
|
---|
102 | # This program is distributed in the hope that it will be useful, but
|
---|
103 | # WITHOUT ANY WARRANTY; without even the implied warranty of
|
---|
104 | # MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or
|
---|
105 | # NON INFRINGEMENT. See the GNU General Public License for more
|
---|
106 | # details.
|
---|
107 | #
|
---|
108 | # You should have received a copy of the GNU General Public License
|
---|
109 | # along with this program; if not, write to the Free Software
|
---|
110 | # Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
|
---|
111 | #
|
---|
112 |
|
---|
113 | : <<inline_doc
|
---|
114 | desc: do farce analisys and report
|
---|
115 | usage: farce -directory $FARCELOGDIR/dir path1 list1 path2 list2
|
---|
116 | input vars: $1 farce log dir for this comparation
|
---|
117 | $2 full path to previous iteration
|
---|
118 | $3 full path to filelist for previos iteration
|
---|
119 | $4 full path to current iteration
|
---|
120 | $5 full path to filelist for current iteration
|
---|
121 | externals: --
|
---|
122 | modifies: --
|
---|
123 | returns: --
|
---|
124 | on error:
|
---|
125 | on success:
|
---|
126 | inline_doc
|
---|
127 |
|
---|
128 |
|
---|
129 | VERSION="002"
|
---|
130 |
|
---|
131 | # variables for output files
|
---|
132 | RESULT=farce-results
|
---|
133 | EXTRAS=farce-extras
|
---|
134 | IDENTICAL=farce-identical
|
---|
135 | SUBS=farce-substitutions
|
---|
136 | DIFFER=farce-differ
|
---|
137 |
|
---|
138 | # documenting the variables
|
---|
139 | # C1, C2 temp files to hold disassembled code
|
---|
140 | # D1, D2 temp directories for extracting member of ar archive
|
---|
141 | # DIFF temp file for diffing the filelists
|
---|
142 | # F1, F2 temp files for tokenizeanddiff
|
---|
143 | # FTYPE obsolete, commented out
|
---|
144 | # M1, M2 temp files to hold members of an ar archive
|
---|
145 | # MEMBERS temp file to list members of ar archive
|
---|
146 | # OP1, OP2 the original $PWD, needed when extracting members of ar
|
---|
147 | # archives
|
---|
148 | # P1, P2 paths to first and second build
|
---|
149 | # S1, S2 temp files for shared objects
|
---|
150 |
|
---|
151 | # functions
|
---|
152 | function dohelp() {
|
---|
153 | echo "`basename $0`: compare trees of files from a build"
|
---|
154 | echo "and lists of the files they contained"
|
---|
155 | echo ""
|
---|
156 | echo "`basename $0` [ -help | -version ] || path1 list1 path2 list2"
|
---|
157 | }
|
---|
158 |
|
---|
159 | function emessage() {
|
---|
160 | # write a string to both stderr and $RESULT
|
---|
161 | echo "$@" >&2
|
---|
162 | echo "$@" >&5
|
---|
163 | }
|
---|
164 |
|
---|
165 | function expected() {
|
---|
166 | # if we expect it to differ because of its name,
|
---|
167 | # allow it and report, return true ; else return false
|
---|
168 | case $1 in
|
---|
169 | /boot/grub/menu.lst)
|
---|
170 | # just in case somebody puts this into the main filesystem
|
---|
171 | true;;
|
---|
172 | /etc/aliases.db)
|
---|
173 | # some sort of database for postfix, not parsable
|
---|
174 | true;;
|
---|
175 | /etc/blkid.tab)
|
---|
176 | # includes dev name for rootfs
|
---|
177 | true;;
|
---|
178 | /etc/fstab)
|
---|
179 | # fstab, e.g. ' / ' will differ
|
---|
180 | true;;
|
---|
181 | /etc/group*)
|
---|
182 | true;;
|
---|
183 | /etc/hosts)
|
---|
184 | # with dhcp client, I add current ip address to this in a hook
|
---|
185 | true;;
|
---|
186 | /etc/ld.so.*)
|
---|
187 | # .conf and .cache can vary,
|
---|
188 | # particularly if one system has a full build when I run this
|
---|
189 | true;;
|
---|
190 | /etc/lilo.conf|/etc/yaboot.conf)
|
---|
191 | # bootloader control, I assume grub will all be on a separate
|
---|
192 | true;;
|
---|
193 | /etc/mtab)
|
---|
194 | # at a minimum, different '/'
|
---|
195 | true;;
|
---|
196 | /etc/ntp.drift)
|
---|
197 | true;;
|
---|
198 | /etc/passwd*)
|
---|
199 | true;;
|
---|
200 | /etc/shadow*)
|
---|
201 | true;;
|
---|
202 | /etc/ssh/*key|/etc/ssh/*pub)
|
---|
203 | # openssh keys
|
---|
204 | true;;
|
---|
205 | /misc/*)
|
---|
206 | # where I put buildscripts (which mostly won't change)
|
---|
207 | # and stamps containing name/time/space which will differ in the times
|
---|
208 | true;;
|
---|
209 | /root/*)
|
---|
210 | # expect .bash_history etc to differ - if we can read them
|
---|
211 | true;;
|
---|
212 | /usr/bin/lynx)
|
---|
213 | # part of my inital builds, I guess this uses anonymous namespaces
|
---|
214 | true;;
|
---|
215 | /usr/include/c++/*/*/bits/stdc++.h.gch/*)
|
---|
216 | # precompiled headers
|
---|
217 | true;;
|
---|
218 | /usr/lib*/libstdc++.a|/usr/lib*/libstdc++.so*|/usr/lib*/libsupc++.a)
|
---|
219 | # probably, anonymous namespaces
|
---|
220 | # libstdc++.a, libstdc++.so.n.n.n, libsupc++.a
|
---|
221 | true;;
|
---|
222 | /usr/share/info/dir)
|
---|
223 | # if one system has had extra stuff built, this will likely be bigger
|
---|
224 | true;;
|
---|
225 | /usr/share/man/whatis)
|
---|
226 | # if one system has had extra stuff built, this will likely be bigger
|
---|
227 | true;;
|
---|
228 | /var/lib/locate/locatedb)
|
---|
229 | # if one system has had extra stuff built, this will likely be bigger
|
---|
230 | true;;
|
---|
231 | /var/lib/nfs/*)
|
---|
232 | # allow nfs bookkeeping
|
---|
233 | true;;
|
---|
234 | /var/log/*)
|
---|
235 | true;;
|
---|
236 | /var/run/utmp)
|
---|
237 | true;;
|
---|
238 | /var/spool/fcron*)
|
---|
239 | true;;
|
---|
240 | /var/state/*)
|
---|
241 | # allow dhcp leases
|
---|
242 | true;;
|
---|
243 | /var/tmp/random-seed)
|
---|
244 | true;;
|
---|
245 | # following start with wildcards
|
---|
246 | *Image*|*.PPCBoot*|*vmlinuz*|*lfskernel*)
|
---|
247 | # compressed kernels, sometimes just building at a different
|
---|
248 | # date/time is enough to change the length of them, because the
|
---|
249 | # long format date and time is part of the compressed data
|
---|
250 | true;;
|
---|
251 | *pid*)
|
---|
252 | # pids, including e.g. /var/spool/postfix/pid/*
|
---|
253 | true;;
|
---|
254 | *)
|
---|
255 | # nothing else is expected to be different
|
---|
256 | false;;
|
---|
257 | esac
|
---|
258 | if [ $? -eq 0 ]; then
|
---|
259 | message "expected difference in $1"
|
---|
260 | let expected=$expected+1
|
---|
261 | case $TYPE in
|
---|
262 | AR)
|
---|
263 | let EXPAR=$EXPAR+1
|
---|
264 | ;;
|
---|
265 | ELF)
|
---|
266 | let EXPELF=$EXPELF+1
|
---|
267 | ;;
|
---|
268 | UNK)
|
---|
269 | let EXPUNK=$EXPUNK+1
|
---|
270 | ;;
|
---|
271 | # so far, no other valid types, so don't accumulate them
|
---|
272 | *)
|
---|
273 | emessage "internal error, expected difference for $1 of type $TYPE not allowed"
|
---|
274 | exit 2
|
---|
275 | ;;
|
---|
276 | esac
|
---|
277 | true
|
---|
278 | else
|
---|
279 | false
|
---|
280 | fi
|
---|
281 | }
|
---|
282 |
|
---|
283 | function failure() {
|
---|
284 | # first parm is filename or token
|
---|
285 | # second parm is the error message
|
---|
286 | # update the appropriate total
|
---|
287 | # and write to both stderr and the results
|
---|
288 | # by using emessage
|
---|
289 |
|
---|
290 | let different=$different+1
|
---|
291 | case $TYPE in
|
---|
292 | AR)
|
---|
293 | let DIFAR=$DIFAR+1
|
---|
294 | ;;
|
---|
295 | ELF)
|
---|
296 | let DIFELF=$DIFELF+1
|
---|
297 | ;;
|
---|
298 | GZ)
|
---|
299 | let DIFGZ=$DIFGZ+1
|
---|
300 | ;;
|
---|
301 | SYM)
|
---|
302 | let DIFSYM=$DIFSYM+1
|
---|
303 | ;;
|
---|
304 | UNK)
|
---|
305 | let DIFUNK=$DIFUNK+1
|
---|
306 | ;;
|
---|
307 | *)
|
---|
308 | emessage "internal error in failure() for TYPE $TYPE"
|
---|
309 | exit 2
|
---|
310 | ;;
|
---|
311 | esac
|
---|
312 | test -f ${P1}$1 && echo $1 >&9
|
---|
313 | emessage "FAIL: $2"
|
---|
314 | }
|
---|
315 |
|
---|
316 | function fatal() {
|
---|
317 | # unrecoverable error
|
---|
318 | echo $*
|
---|
319 | exit 1
|
---|
320 | }
|
---|
321 |
|
---|
322 | function filetype() {
|
---|
323 | TYPE=`file ${P1}${FILE}`
|
---|
324 | case $TYPE in
|
---|
325 | *'current ar archive'*)
|
---|
326 | let TOTAR=$TOTAR+1
|
---|
327 | TYPE=AR
|
---|
328 | ;;
|
---|
329 | *' ELF '*)
|
---|
330 | let TOTELF=$TOTELF+1
|
---|
331 | TYPE=ELF
|
---|
332 | ;;
|
---|
333 | *'gzip compressed data'*)
|
---|
334 | let TOTGZ=$TOTGZ+1
|
---|
335 | TYPE=GZ
|
---|
336 | ;;
|
---|
337 | *)
|
---|
338 | let TOTUNK=$TOTUNK+1
|
---|
339 | TYPE=UNK
|
---|
340 | ;;
|
---|
341 | esac
|
---|
342 | }
|
---|
343 |
|
---|
344 | function message() {
|
---|
345 | # write a string to $RESULT
|
---|
346 | echo $* >&5
|
---|
347 | }
|
---|
348 |
|
---|
349 | function onlyone() {
|
---|
350 | #report files only in one build
|
---|
351 | # text should go to both stderr and the results,
|
---|
352 | # but blank lines only go to the results
|
---|
353 | if [ $1 == '<' ]; then
|
---|
354 | emessage "File(s) only in the first build"
|
---|
355 | else
|
---|
356 | emessage "File(s) only in the second build"
|
---|
357 | fi
|
---|
358 | message ""
|
---|
359 | FILES=`cat $DIFF | grep "^$1" | cut -d ' ' -f 2`
|
---|
360 | for F in $FILES; do
|
---|
361 | emessage $F
|
---|
362 | let only=$only+1
|
---|
363 | done
|
---|
364 | message ""
|
---|
365 | }
|
---|
366 |
|
---|
367 | # 'test' functions are called with three arguments:
|
---|
368 | # the two pathes and the filename
|
---|
369 | # - we know the file is of this type, so see if we
|
---|
370 | # can get it to match by reasonalbe means.
|
---|
371 | # if not, treat it as different.
|
---|
372 | #
|
---|
373 | # NB if pathes are absolute, we need to prefix them
|
---|
374 | # with the original $PWD to access the .a files
|
---|
375 | #
|
---|
376 | function testar() {
|
---|
377 | # ar archives include timestamps for the members,
|
---|
378 | # but diff doesn't show file timestamps unless the data differs
|
---|
379 | # put out a message to help locate which archive any messages
|
---|
380 | # about the members refer to.
|
---|
381 |
|
---|
382 | # try just stripping them U1,2 undebuggable
|
---|
383 | U1=`mktemp` || fatal "cannot create a temporary file"
|
---|
384 | U2=`mktemp` || fatal "cannot create a temporary file"
|
---|
385 | cp ${1}${3} $U1
|
---|
386 | cp ${2}${3} $U2
|
---|
387 | strip --strip-debug $U1
|
---|
388 | strip --strip-debug $U2
|
---|
389 | cmp -s $U1 $U2
|
---|
390 | rm $U1 $U2
|
---|
391 | if [ $? -eq 0 ]; then
|
---|
392 | let accepted=$accepted+1
|
---|
393 | let ACCAR=$ACCAR+1
|
---|
394 | message "archive $3 matches after strip --strip-debug"
|
---|
395 | return
|
---|
396 | fi
|
---|
397 | # rest of this function retained primarily for pathologically bad builds
|
---|
398 | # put out a message in the log to help identify which archive has issues.
|
---|
399 | message "examining ar archive $3"
|
---|
400 | D1=`mktemp -d` || fatal "cannot create a temporary directory"
|
---|
401 | D2=`mktemp -d` || fatal "cannot create a temporary directory"
|
---|
402 | cd $D1
|
---|
403 | ar -x ${OP1}${1}${3}
|
---|
404 | cd $D2
|
---|
405 | ar -x ${OP2}${2}${3}
|
---|
406 | cd
|
---|
407 | # diff the members - true means they match
|
---|
408 | diff -Na $D1 $D2 >/dev/null
|
---|
409 | if [ $? -eq 0 ]; then
|
---|
410 | message "accept: $3 after diffing the members"
|
---|
411 | let accepted=$accepted+1
|
---|
412 | let ACCAR=$ACCAR+1
|
---|
413 | else
|
---|
414 | # process individual members to eliminate date/time/kernel-version
|
---|
415 | # first, check the members are the same
|
---|
416 | M1=`mktemp` || fatal "cannot create a temporary file"
|
---|
417 | M2=`mktemp` || fatal "cannot create a temporary file"
|
---|
418 | cd $D1
|
---|
419 | MEMBERS=
|
---|
420 | for F in *; do
|
---|
421 | MEMBERS="$MEMBERS $F"
|
---|
422 | done
|
---|
423 | cd
|
---|
424 | echo $MEMBERS | sort >$M1
|
---|
425 | cd $D2
|
---|
426 | MEMBERS=
|
---|
427 | for F in *; do
|
---|
428 | MEMBERS="$MEMBERS $F"
|
---|
429 | done
|
---|
430 | cd
|
---|
431 | echo $MEMBERS | sort >$M2
|
---|
432 | cmp -s $M1 $M2
|
---|
433 | if [ $? -ne 0 ]; then
|
---|
434 | # oh dear, different members
|
---|
435 | echo "list of members differs for archive $3" >&6
|
---|
436 | diff $M1 $M2 >&6
|
---|
437 | failure $3 "$3 list of members differs"
|
---|
438 | else
|
---|
439 | # members (names) are same,
|
---|
440 | # process each one
|
---|
441 | STATUS=0
|
---|
442 | for M in $MEMBERS; do
|
---|
443 | #avoid firing up perl on matching members
|
---|
444 | cmp -s $D1/$M $D2/$M
|
---|
445 | if [ $? -ne 0 ]; then
|
---|
446 | tokenizeanddiff $D1/$M $D2/$M $FILE:$M
|
---|
447 | if [ $? -eq 0 ]; then
|
---|
448 | message "member $M matches after processing"
|
---|
449 | else
|
---|
450 | message "member $M DIFFERS after processing"
|
---|
451 | STATUS=1
|
---|
452 | fi
|
---|
453 | fi
|
---|
454 | done
|
---|
455 | if [ $STATUS -eq 0 ]; then
|
---|
456 | let accepted=$accepted+1
|
---|
457 | let ACCAR=$ACCAR+1
|
---|
458 | else
|
---|
459 | let different=$different+1
|
---|
460 | let DIFAR=$DIFAR+1
|
---|
461 | echo $3 >&9
|
---|
462 | emessage "FAIL: in $3"
|
---|
463 | fi
|
---|
464 | fi
|
---|
465 | rm $M1 $M2
|
---|
466 | fi
|
---|
467 | rm -rf $D1 $D2
|
---|
468 | }
|
---|
469 |
|
---|
470 | function testgzip() {
|
---|
471 | # bytes 4,5,6,7 are the timestamp, so ignore these
|
---|
472 | cmp -s -i 8 ${1}${3} ${2}${3}
|
---|
473 | if [ $? -eq 0 ]; then
|
---|
474 | message "accept: $3 after ignoring gzip timestamp"
|
---|
475 | let accepted=$accepted+1
|
---|
476 | let ACCGZ=$ACCGZ+1
|
---|
477 | else
|
---|
478 | failure $3 " $3 even after ignoring gzip timestamp"
|
---|
479 | fi
|
---|
480 | }
|
---|
481 |
|
---|
482 | function testso() {
|
---|
483 | # shared object - first try stripping it
|
---|
484 | # in fact, this now handles ALL ELF files
|
---|
485 | S1=`mktemp` || fatal "cannot create a temporary file"
|
---|
486 | S2=`mktemp` || fatal "cannot create a temporary file"
|
---|
487 | cp ${1}${3} $S1
|
---|
488 | strip --strip-debug $S1
|
---|
489 | cp ${2}${3} $S2
|
---|
490 | strip --strip-debug $S2
|
---|
491 | cmp -s $S1 $S2
|
---|
492 | if [ $? -eq 0 ]; then
|
---|
493 | message "accept: $3 after --strip-debug"
|
---|
494 | let accepted=$accepted+1
|
---|
495 | let ACCELF=$ACCELF+1
|
---|
496 | else
|
---|
497 | tokenizeanddiff $S1 $S2 $3
|
---|
498 | if [ $? -ne 0 ]; then
|
---|
499 | failure $3 " $3 differs after stripping and processing"
|
---|
500 | else
|
---|
501 | message "accept: $3 after --strip-debug and processing"
|
---|
502 | let accepted=$accepted+1
|
---|
503 | let ACCELF=$ACCELF+1
|
---|
504 | fi
|
---|
505 | fi
|
---|
506 | rm $S1 $S2
|
---|
507 | }
|
---|
508 |
|
---|
509 | function tokenize() {
|
---|
510 | # use regexes to replace date/time/kernel-version text
|
---|
511 | # with tokens which may allow files to match even though
|
---|
512 | # they have hardcoded date/time/kernel-version.
|
---|
513 | # arguments are file to process, and where to put it.
|
---|
514 | # these regexes are somewhat long, and the order they
|
---|
515 | # are applied in is important (to stop short ones being
|
---|
516 | # used when a longer version would match).
|
---|
517 | # KV00 linux version date (e.g. as in the kernel itself)
|
---|
518 | # allow 2 or 3 groups of three alphas here - optional smp, with day, mon
|
---|
519 | # KV01 kernel version, including possible cpu details (that is for cdda2wav)
|
---|
520 | # KV02 just the version, in quotes e.g. "2.6.12.6" or '2.6.13', for perl stuff
|
---|
521 | # except that "|' gives me grif, so try a boundary
|
---|
522 | # also, it might need local version on the end, I really want
|
---|
523 | # quote2.\d+.\d+.{0,32}quote - it is the quotes that don't work.
|
---|
524 | # DT00 Day Mon .d+ hh:mm:ss TZN CCYY variations include non-caps and 'mon d'
|
---|
525 | # DT01 Mon .d+ CCYY hh:mm:ss
|
---|
526 | # DT02 hh:mm:ss Mon .d CCYY
|
---|
527 | # DT03 Mon .d CCYY
|
---|
528 | # DT04 Day Mon { ,d}d hh:mm:ss CCYY - for groff example postscript files
|
---|
529 | # (somewhat similar to DT00, but ' d' or ' dd' for day of month and no TZN )
|
---|
530 | # DT05 hh:mm:ss
|
---|
531 | # DT06 ISO date using space as separator
|
---|
532 | # DT07 ISO date using dash as separator
|
---|
533 | # DT08 ISO date using slash as separator
|
---|
534 | # DT09 fullmonth (capitalised), day number, comma, 4-digit year (groff 1.18.1 ps)
|
---|
535 | # DT10 dd, fullmonth (capitalised), 4-digit year (groff 1.18.1 manpages)
|
---|
536 | # DT11 '(xample comma space digit(s) backslash ) in groff memef.ps which is
|
---|
537 | # quite clearly the day of the month when it was compiled, preceded by 'example'
|
---|
538 | # with something weird between the e and the x.
|
---|
539 |
|
---|
540 | if [ $# -ne 2 ]; then
|
---|
541 | fatal "tokenizing called with $# arguments : $*"
|
---|
542 | fi
|
---|
543 |
|
---|
544 | cat $1 | perl -p \
|
---|
545 | -e 's/(L|l)inux.*\d\.\d\.\d+.* \#\d+( [A-Za-z][a-z]{2}){2,3} \d+ \d\d:\d\d:\d\d [A-Za-z]{3} \d{4}\b/%KV00%/g;' \
|
---|
546 | -e 's/(L|l)inux( (\w|_)+)?(-| |_)\d\.\d(\.\d+){1,2}((-|_)?(\w|_)+)?( |\000)*/%KV01%/g;' \
|
---|
547 | -e 's/\W2(\.\d+){2,3}(-|_)?((\w|_)+)?\s*\W/%KV02%/g;' \
|
---|
548 | -e 's/\b([A-Za-z][a-z]{2} ){2}( |\d)?\d \d\d:\d\d:\d\d [A-Za-z]{3} \d{4}\b/%DT00%/g;' \
|
---|
549 | -e 's/\b[A-Z][a-z]{2} ( |\d)\d \d{4} \d\d:\d\d:\d\d\b/%DT01%/g;' \
|
---|
550 | -e 's/\b\d\d:\d\d:\d\d [A-Z][a-z]{2} ( |\d)\d \d{4}\b/%DT02%/g;' \
|
---|
551 | -e 's/\b[A-Z][a-z]{2} ( |\d)\d \d{4}\b/%DT03%/g;' \
|
---|
552 | -e 's/\b([A-Z][a-z]{2} ){2}( |\d)\d \d\d:\d\d:\d\d \d{4}/%DT04%/g;' \
|
---|
553 | -e 's/\b\d\d:\d\d:\d\d\b/%DT05%/g;' \
|
---|
554 | -e 's/\b\d{4} \d\d \d\d\b/%DT06%/g;' \
|
---|
555 | -e 's/\b\d{4}-\d\d-\d\d\b/%DT07%/g;' \
|
---|
556 | -e 's/\b\d{4}\/\d\d\/\d\d\b/%DT08%/g;' \
|
---|
557 | -e 's/\b[A-Z][a-z]{2,} \d{1,2}, \d{4}/%DT09%/g;' \
|
---|
558 | -e 's/\b\d\d [A-Z][a-z]{2,} \d{4}/%DT10%/g;' \
|
---|
559 | -e 's/\(xample, \d{1,2}\\\)/%DT11%/g;' \
|
---|
560 | >$2
|
---|
561 | }
|
---|
562 |
|
---|
563 | function tokenizeanddiff() {
|
---|
564 | # Call tokenize for the inputs, then compare the results
|
---|
565 | # Input arguments are path/filename for old and new versions
|
---|
566 | # third parm is readable name (filename, or archivename:member)
|
---|
567 | # to help understand what is in the extras output.
|
---|
568 | # - sometimes called for files, but other times called for
|
---|
569 | # members of ar archives extracted into temporary directories
|
---|
570 | #message tokenizeanddiff called for $1 $2 $3
|
---|
571 | F1=`mktemp` || fatal "cannot create a temporary file"
|
---|
572 | F2=`mktemp` || fatal "cannot create a temporary file"
|
---|
573 | tokenize $1 $F1
|
---|
574 | tokenize $2 $F2
|
---|
575 |
|
---|
576 | # actually, cmp is probably more efficient
|
---|
577 | # but for picking up the pieces it will be better to
|
---|
578 | # use diff to see what got through.
|
---|
579 | cmp -s $F1 $F2
|
---|
580 | TOKENRESULT=$?
|
---|
581 | if [ $TOKENRESULT -ne 0 ]; then
|
---|
582 | echo "failure in $3..." >&6
|
---|
583 | diff -a $F1 $F2 >&6
|
---|
584 | rm $F1 $F2
|
---|
585 | false
|
---|
586 | else
|
---|
587 | # show what we did
|
---|
588 | echo "substitutions for $3" >&8
|
---|
589 | echo "build one" >&8
|
---|
590 | diff -a $1 $F1 >&8
|
---|
591 | echo "build two" >&8
|
---|
592 | diff -a $2 $F2 >&8
|
---|
593 | rm $F1 $F2
|
---|
594 | true
|
---|
595 | fi
|
---|
596 | }
|
---|
597 |
|
---|
598 | function validateargs() {
|
---|
599 | # validate the arguments
|
---|
600 | BAD=0
|
---|
601 | if ! [ -d $1 ]; then
|
---|
602 | echo "Error: first argument is not a directory" >&2
|
---|
603 | let BAD=$BAD+1
|
---|
604 | fi
|
---|
605 | NAME=`basename ${2%%-*}`
|
---|
606 | if [ $NAME != filelist ]; then
|
---|
607 | echo "Error: second argument is not a recognized filelist" >&2
|
---|
608 | let BAD=$BAD+1
|
---|
609 | fi
|
---|
610 | if ! [ -d $3 ]; then
|
---|
611 | echo "Error: third argument is not a directory" >&2
|
---|
612 | let BAD=$BAD+1
|
---|
613 | fi
|
---|
614 | NAME=`basename ${4%%-*}`
|
---|
615 | if [ $NAME != filelist ]; then
|
---|
616 | echo "Error: fourth argument is not a recognized filelist" >&2
|
---|
617 | let BAD=$BAD+1
|
---|
618 | fi
|
---|
619 | for I in $1 $2 $3 $4; do
|
---|
620 | if ! [ -r $I ]; then
|
---|
621 | echo "Error: cannot read $I" >&2
|
---|
622 | let BAD=$BAD+1
|
---|
623 | fi
|
---|
624 | done
|
---|
625 | if [ $1 == $3 ]; then
|
---|
626 | echo "Error: directory pathes are identical" >&2
|
---|
627 | let BAD=$BAD+1
|
---|
628 | fi
|
---|
629 | if [ $2 == $4 ]; then
|
---|
630 | echo "Error: filelist names are identical" >&2
|
---|
631 | let BAD=$BAD+1
|
---|
632 | fi
|
---|
633 | if [ $BAD -eq 0 ]; then
|
---|
634 | ARGS=valid
|
---|
635 | fi
|
---|
636 | }
|
---|
637 |
|
---|
638 | # Mainline
|
---|
639 | ARGS=unproven
|
---|
640 | OUTDIR=
|
---|
641 | if [ $# -eq 1 ]; then
|
---|
642 | case $1 in
|
---|
643 | -version|--version)
|
---|
644 | echo "`basename $0` version $VERSION"
|
---|
645 | exit 0
|
---|
646 | ;;
|
---|
647 | -help|--help)
|
---|
648 | dohelp
|
---|
649 | exit 0
|
---|
650 | ;;
|
---|
651 | esac
|
---|
652 | fi
|
---|
653 | if [ $1 = "--directory" ]; then
|
---|
654 | OUTDIR=$2
|
---|
655 | shift 2
|
---|
656 | grep '/$' $OUTDIR >/dev/null 2>&1 || OUTDIR=`echo $OUTDIR | sed 's%$%/%'`
|
---|
657 | echo "creating directory $OUTDIR"
|
---|
658 | mkdir -p $OUTDIR
|
---|
659 | if [ $? -ne 0 ]; then
|
---|
660 | echo "cannot mkdir $OUTDIR"
|
---|
661 | exit 1
|
---|
662 | fi
|
---|
663 | fi
|
---|
664 | if [ $# -eq 4 ]; then
|
---|
665 | validateargs $*
|
---|
666 | fi
|
---|
667 | if ! [ $ARGS == valid ]; then
|
---|
668 | dohelp
|
---|
669 | fatal "`basename $0`: error in arguments"
|
---|
670 | fi
|
---|
671 |
|
---|
672 | # ok, we're happy, lets hit these files
|
---|
673 | exec 5>${OUTDIR}$RESULT
|
---|
674 | exec 6>${OUTDIR}$EXTRAS
|
---|
675 | exec 7>${OUTDIR}$IDENTICAL
|
---|
676 | exec 8>${OUTDIR}$SUBS
|
---|
677 | exec 9>${OUTDIR}$DIFFER
|
---|
678 |
|
---|
679 | >${OUTDIR}$RESULT
|
---|
680 | if [ $? -ne 0 ]; then
|
---|
681 | fatal "cannot write to ${OUTDIR}$RESULT"
|
---|
682 | fi
|
---|
683 |
|
---|
684 | emessage "will compare:"
|
---|
685 | emessage " first build at $1 with files listed in $2"
|
---|
686 | emessage "second build at $3 with files listed in $4"
|
---|
687 |
|
---|
688 | let accepted=0
|
---|
689 | let different=0
|
---|
690 | let expected=0
|
---|
691 | let matched=0
|
---|
692 | let only=0
|
---|
693 | let predictable=0
|
---|
694 | let unreadable=0
|
---|
695 | let total=0
|
---|
696 |
|
---|
697 | # break down the accepted
|
---|
698 | let ACCAR=0
|
---|
699 | let ACCELF=0
|
---|
700 | let ACCGZ=0
|
---|
701 | let ACCUNK=0
|
---|
702 |
|
---|
703 | # break down definitely different
|
---|
704 | let DIFAR=0
|
---|
705 | let DIFELF=0
|
---|
706 | let DIFGZ=0
|
---|
707 | let DIFSYM=0
|
---|
708 | let DIFUNK=0
|
---|
709 |
|
---|
710 | # break down the expected differences
|
---|
711 | let EXPAR=0
|
---|
712 | let EXPELF=0
|
---|
713 | let EXPGZ=0
|
---|
714 | let EXPUNK=0
|
---|
715 |
|
---|
716 | # break down the identical files
|
---|
717 | let MATAR=0
|
---|
718 | let MATELF=0
|
---|
719 | let MATGZ=0
|
---|
720 | let MATSYM=0
|
---|
721 | let MATUNK=0
|
---|
722 |
|
---|
723 | # break down how many of each type
|
---|
724 | let TOTAR=0
|
---|
725 | let TOTELF=0
|
---|
726 | let TOTGZ=0
|
---|
727 | let TOTSYM=0
|
---|
728 | let TOTUNK=0
|
---|
729 |
|
---|
730 | # now identify differences between the two trees
|
---|
731 | DIFF=`mktemp` || fatal "cannot create a temporary file"
|
---|
732 | diff $2 $4 >$DIFF
|
---|
733 |
|
---|
734 | for RUN in '<' '>' ; do
|
---|
735 | grep -q "$RUN" $DIFF && onlyone "$RUN"
|
---|
736 | done
|
---|
737 |
|
---|
738 | rm $DIFF
|
---|
739 |
|
---|
740 | # and compare them
|
---|
741 | message "Results of file comparison:"
|
---|
742 | message ""
|
---|
743 |
|
---|
744 | # Strip any trailing slash from the path for tidyness,
|
---|
745 | # because the filenames all start with a slash. Unfortunately,
|
---|
746 | # unfortunately, '/' becomes empty, which breaks subroutines,
|
---|
747 | # so special case it.
|
---|
748 | # also, to process ar archives we need to extract them in temp
|
---|
749 | # directories - that means that after cd'ing we've broken any
|
---|
750 | # relative path, so save original pwd as necessary.
|
---|
751 | P1=`echo $1 | sed 's%/$%%'`
|
---|
752 | echo $1 | grep '^/' >/dev/null
|
---|
753 | if [ $? -ne 0 ]; then
|
---|
754 | # relative path
|
---|
755 | OP1=${PWD}/
|
---|
756 | #echo "setting OP1 to $OP1"
|
---|
757 | else
|
---|
758 | OP1=
|
---|
759 | #echo "$1 is an absolute path"
|
---|
760 | fi
|
---|
761 | test -z "$P1" && P1='/'
|
---|
762 | P2=`echo $3 | sed 's%/$%%'`
|
---|
763 | echo $3 | grep '^/' >/dev/null
|
---|
764 | if [ $? -ne 0 ]; then
|
---|
765 | # relative path
|
---|
766 | OP2=${PWD}/
|
---|
767 | #echo "setting OP2 to $OP2"
|
---|
768 | else
|
---|
769 | OP2=
|
---|
770 | #echo "$3 is an absolute path"
|
---|
771 | fi
|
---|
772 | test -z "$P2" && P2='/'
|
---|
773 |
|
---|
774 | echo "about to read $2"
|
---|
775 | while read FILE ; do
|
---|
776 | #echo "process $FILE"
|
---|
777 | #echo "test existence of ${P2}${FILE}"
|
---|
778 | # confirm it exists in second build
|
---|
779 | # we have already reported files only in one build
|
---|
780 | if [ -f ${P2}"${FILE}" ]; then
|
---|
781 | let total=$total+1
|
---|
782 | # check we can read both of them
|
---|
783 | # or count as unreadable - I used to separate only-one-unreadable,
|
---|
784 | # but if you compre '/' and a _copy_ of /mnt/lfs that assumption
|
---|
785 | # breaks, so be less picky.
|
---|
786 | if ! [ -r "${P1}${FILE}" ] || ! [ -r "${P2}${FILE}" ]; then
|
---|
787 | message "cannot read one or both versions of $FILE"
|
---|
788 | let unreadable=$unreadable+1
|
---|
789 | continue
|
---|
790 | fi
|
---|
791 | if [ -h "${P1}${FILE}" ]; then
|
---|
792 | # for symlink, look at what it points to
|
---|
793 | # exceptionally, do not call filetype
|
---|
794 | TYPE=SYM
|
---|
795 | let TOTSYM=$TOTSYM+1
|
---|
796 | SL1=`ls -l "${P1}${FILE}" | awk '{ print $11 }'`
|
---|
797 | SL2=`ls -l "${P2}${FILE}" | awk '{ print $11 }'`
|
---|
798 | if [ "$SL1" = "$SL2" ]; then
|
---|
799 | echo "symlink $FILE matches for $SL1" >&5
|
---|
800 | let matched=$matched+1
|
---|
801 | let MATSYM=$MATSYM+1
|
---|
802 | else
|
---|
803 | failure TARGET " symlink $FILE points to $SL1 and $SL2"
|
---|
804 | echo $FILE >&9
|
---|
805 | fi
|
---|
806 | else
|
---|
807 | # regular file, start by typing it for accounting,
|
---|
808 | # then compare it
|
---|
809 | filetype ${P1}${FILE}
|
---|
810 | cmp -s "${P1}${FILE}" "${P2}${FILE}"
|
---|
811 | if [ $? -eq 0 ]; then
|
---|
812 | let matched=$matched+1
|
---|
813 | case $TYPE in
|
---|
814 | AR)
|
---|
815 | let MATAR=$MATAR+1
|
---|
816 | ;;
|
---|
817 | ELF)
|
---|
818 | let MATELF=$MATELF+1
|
---|
819 | ;;
|
---|
820 | GZ)
|
---|
821 | let MATGZ=$MATGZ+1
|
---|
822 | ;;
|
---|
823 | UNK)
|
---|
824 | let MATUNK=$MATUNK+1
|
---|
825 | ;;
|
---|
826 | *)
|
---|
827 | echo "unexpected TYPE of $TYPE for $FILE" >&2
|
---|
828 | exit 2
|
---|
829 | ;;
|
---|
830 | esac
|
---|
831 | echo ${FILE} >&7
|
---|
832 | else
|
---|
833 | # seems different, can we do better ?
|
---|
834 | # test if we expect it to differ
|
---|
835 | expected $FILE
|
---|
836 | if [ $? -ne 0 ]; then
|
---|
837 | case $TYPE in
|
---|
838 | GZ)
|
---|
839 | testgzip $P1 $P2 $FILE ;;
|
---|
840 | AR)
|
---|
841 | testar $P1 $P2 $FILE ;;
|
---|
842 | ELF)
|
---|
843 | testso $P1 $P2 $FILE ;;
|
---|
844 | *)
|
---|
845 | # long-stop - strip dates from text files
|
---|
846 | tokenizeanddiff "${P1}${FILE}" "${P2}${FILE}" "$FILE"
|
---|
847 | if [ $? -eq 0 ]; then
|
---|
848 | message "accepted $FILE after processing"
|
---|
849 | let accepted=$accepted+1
|
---|
850 | let ACCUNK=$ACCUNK+1
|
---|
851 | else
|
---|
852 | failure "$FILE" " $FILE is different"
|
---|
853 | fi
|
---|
854 | ;;
|
---|
855 | esac
|
---|
856 | fi
|
---|
857 | fi
|
---|
858 | fi
|
---|
859 | fi
|
---|
860 | done < $2
|
---|
861 |
|
---|
862 | message ""
|
---|
863 | # write totals to stderr as well as the results file
|
---|
864 | emessage "$only files in only one of the builds"
|
---|
865 | emessage "$total files compared, of which"
|
---|
866 | emessage "$unreadable files could not be read, skipped"
|
---|
867 | emessage "$matched files are identical"
|
---|
868 | emessage "$expected files differed as expected"
|
---|
869 | emessage "$accepted files had allowable differences"
|
---|
870 | #emessage "$predictable files differed as they normally do"
|
---|
871 | emessage "$different files differed"
|
---|
872 |
|
---|
873 | # totals of different file types
|
---|
874 | emessage ""
|
---|
875 | emessage "$TOTAR ar archives"
|
---|
876 | emessage " of which $MATAR are identical"
|
---|
877 | emessage " of which $ACCAR are accepted after strip-debug or extracting, diffing, tokenizing"
|
---|
878 | emessage " of which $EXPAR differed as expected"
|
---|
879 | emessage " of which $DIFAR differed"
|
---|
880 | emessage "$TOTELF ELF executables or shared libraries"
|
---|
881 | emessage " of which $MATELF are identical"
|
---|
882 | emessage " of which $ACCELF are accepted after stripping and tokenizing"
|
---|
883 | emessage " of which $EXPELF differed as expected"
|
---|
884 | emessage " of which $DIFELF differed"
|
---|
885 | emessage "$TOTGZ gzipped files"
|
---|
886 | emessage " of which $MATGZ are identical"
|
---|
887 | emessage " of which $ACCGZ are accepted after comparing beyond timestamp"
|
---|
888 | emessage " of which $DIFGZ are different"
|
---|
889 | emessage "$TOTSYM symbolic links"
|
---|
890 | emessage " of which $MATSYM are identical"
|
---|
891 | emessage " of which $DIFSYM have different targets"
|
---|
892 | emessage "$TOTUNK other files"
|
---|
893 | emessage " of which $MATUNK are identical"
|
---|
894 | emessage " of which $ACCUNK are accepted after tokenizing"
|
---|
895 | emessage " of which $EXPUNK differed as expected"
|
---|
896 | emessage " of which $DIFUNK differed"
|
---|
897 |
|
---|