From: jhmeyers@miu.edu (John H Meyers)
Subject: BZDIR improved even further (v 3.2)
Date: 11 Nov 1997 00:00:00 GMT
Newsgroups: comp.sys.hp48

The purpose of BZDIR is to compress whole directories
as single objects, which is much more efficient than
compressing the contents of a directory one object at a time
(the latter actually sometimes *expands* the total memory used!)

The problem with compressing a whole directory is deciding
where to store it, since normally you can not replace the
original directory variable with a non-directory (string) by
attempting to store it into the same name; even if you could,
you would then lose your original directory structure and
the "tabbed" menu labels.

One way to do it could be to make the result a self-extracting
object and override the normal "Directory Not Allowed" protection
to store the result back into the same variable; such a method seems
to require the object to have its own name embedded in it, causing
problems if renamed, and the additional normal protections against
accidentallly purging or overwriting the directory variable itself
would also be lost (have you ever inadvertently clobbered a program
in this way? Well then, what about clobbering a whole directory?)

The way we do it here is to recall the whole current directory as a
single object, compress it, and then store the result (now a string)
into the empty directory which remains after we purge all variables;
this keeps the directory structure intact as it originally existed,
and retains the protection against accidental clobbering.

When the compressed string is later turned back into a complete
directory object, we then store it right back into the empty
original directory variable, in one single STO, bypassing the
"Directory Not Allowed" protection just this one time,
in order to restore the complete directory in one operation.

This is much faster than the previous method (which used a
list of individual objects), although compression is in general
reduced very slightly (a couple of bytes or so per variable).

BZDIR relies on Mika Heiskanen's BZ compressor, which is
available from many sources, including Goodies Disk #10/Utils
(you can choose a version using less or more "buffer" memory)
or <ftp://hpcvbbs.cv.hp.com/dist/hp48g/utils/compress/bz12.zip>.

BZ is also part of HACK.LIB on GD#10/Jazz; the most recent HACK
library can be obtained from <http://www.hut.fi/~mheiskan>.

If you happen to have the HACK library installed, then BZDIR will
automatically take advantage of its faster VARS and PG commands.

I am using this on both my GX (with HACK.LIB installed)
and my 32k G model (without HACK.LIB), and it may well
have saved me the cost of an upgrade or a RAM card!

During execution of BZDIR, the HP48 status annunciators will
flash wildly; this is actually caused by BZ, and is normal.

If you use BZDIR frequently, you might wish to assign it to a
keycode or a Custom Menu label (Chapter 30 in G/GX User's Guide),
or to give it a shorter name which you can readily type.

Note that compression of the HOME directory is disallowed.

Do not store any new, uncompressed variables into an
already-compressed directory, because when you uncompress the
current directory, its entire current contents will be overwritten,
which means that you would lose any newly added variables.

Older versions of BZDIR used a separate program to decompress
the self-extracting object in which each compressed directory
was stored; this version uses BZDIR for both compression
and decompression, without making a self-extracting object.

If you replace an older version of BZDIR with this version,
you can leave old compressed directories as they were, and the
original self-extracting objects will continue to work; however,
do not purge the old separate BZDXT program (decompressor)
until each old directory has been decompressed at least once.

Select one or more of the following programs, according to
whether or not you use HACK.LIB, and other instructions below;
store the selected programs into your HOME directory.

If you download (via Kermit) a portion of the remaining part
of this post which includes more than one program, then
the result will be one object containing all the programs;
in this case, when you first execute the received file,
it will just leave the separate individual programs
on the stack, and you can then just store each
separate program under its proper name.

If typing by hand, omit comments which begin with "@"

BZDIR begins with "safety checks" to ensure that you do not
try to compress the HOME directory, and that the program BZ
is currently available (if BZ is unavailable, then you might
end up wiping out the contents of your current directory,
instead of just compressing or expanding it).

BZDIR version 3.1 does this by compressing a small object
(the real number zero) to see whether it turns into a string
beginning with "BZ"; this seems reasonable and simple enough,
even though it takes about 1/2 second, for the safety it offers.

However, whenever BZ compresses anything, it always reserves a
rather large "hash table" (7K-11K); if the amount of free memory
is less than this amount, an "Insufficient Memory" error will occur,
even if you are only compressing a rather small object.

This is fine if you are trying to compress a directory, since
if you haven't enough memory to compress even a single real number,
then certainly there isn't enough memory to compress a whole directory.

However, if you are trying to *un*compress a directory, then less memory
may actually be required for BZ to uncompress the whole directory
than to compress one real number, in which case the report of
"Insufficient Memory" may be greatly exaggerated.

Therefore, I have listed below a version of BZDIR which verifies
the availability of BZ by *un*compressing something, rather than
by compressing something.  The "something" is once again
the real number zero; however, since the "compressed" version of
zero is made up of non-ascii characters, some of which are nulls,
a modest kluge is used to generate the string in such a manner
that we can continue to post the program in ascii mode,
in a form which you can even continue to edit normally.

Actually, take your pick of any of the several kluges offered,
since I tried out each one of them; the least complicated one
(also the longest program) has already been pre-selected, but
you can make whatever changes you want, per the included notes
(checksums are provided for each choice you might make).

You have probably noticed that BZ tells you during compression
how many bytes (and percentage of total) that you have saved;
there is only an 18-byte overhead to subtract from what BZ
tells you, for the variable name and directory header.

BZ itself is about 1500 bytes, and BZDIR adds a couple of hundred
more; you do not start saving overall until your total compression
adds up to the size of these programs; then your initial investment
begins to pay off.

If you obtain the stand-alone version of BZ (e.g. from Goodies Disk #10),
choose the version labeled BZ1 if you have 32K; choose the BZ4 version
if you have 128K or more, as suggested in the documentation; the program
size and results of compression are the same, but the "smaller" version
reserves less memory for the hash table, which is what may cause
"Insufficient Memory" before you have even begun to compress anything
(compression speed is enhanced by using a larger table, but only if
you can afford the required extra memory overhead).

Hiding variables with HIDE, HIDER, etc. was not ideal with the
old BZDIR, since it only compressed and restored those variables
which it could list using VARS (evidently leaving any hidden
variables alone); however, no particular danger seemed apparent.

This version of BZDIR compresses the directory object itself,
including anything hidden, but it can not remove any hidden
variables to save their redundant space.  When the directory
is restored, we think we are storing it in its entirety into
an empty variable, but if there were any hidden variables,
then they are being overwritten in a manner which may or
may not observe the necessary formalities; I am not
expert enough to know whether there is any danger in this.

The hazards mentioned in FPGDIR.DOC on Goodies Disk #5
may also still apply here; therefore I do not recommend
ever using this version of BZDIR in any directory where
you have "hidden" any variables.

Everyone else should get good results :)

-----------------------------------------------------------------

%%HP: T(3); @ \-> is "right-arrow", \=/ is "unequal", etc.
@ BZDIR (Version 3.2)

\<< PATH SIZE 1 \<= @ Not allowed in HOME

@ Uncomment only one of the next three paragraphs (or none),
@ to test for presence of BZ by uncompressing an object:

@ Create compressed object (real number zero):
@ [109.5 bytes, .202 seconds]
  0 41 51 5 106 0 0 144 0 21 "BZ"
  1 10 START SWAP CHR + NEXT "BZ" OBJ\-> OR

@ Alternative using Wickes' ASC\-> program;
@ Get ASC\-> from the FAQ, or from Goodies Disk #1:
@ [61.5 bytes, .075 seconds]
@ "C2A20C100024A55100090000A65033920311B" ASC\-> "BZ" OBJ\-> OR

@ Alternative via concatenating hex strings;
@ Dangerous if any mistake in downloading/copying:
@ [95 bytes, .067 seconds]
@ #6A00009000155A42h #293305h #518Ah SYSEVAL
@ 10796 #18CEAh SYSEVAL #5AB3h SYSEVAL "BZ" OBJ\-> OR
@ Supported entry points: &HXS, COERCE, CHANGETYPE

  { 298 DOERR } IFT @ Complete the "up-front" error checking

  "BZ VARS PG" @ *string* of commands to be executed below

  'dirbz..' DUP VTYPE 0 < @ Variable in which result is stored

  { PATH RCL ROT OBJ\-> @ Compress
    @ The following extra check may detect missing PG:
    DUP 1 2 SUB "BZ" \=/ { 514 DOERR } IFT @ Compressed object?
    SWAP STO }

  { RCL SWAP OBJ\-> @ Expand
    @ The following extra check may detect missing PG:
    DUP TYPE 15 \=/ { 514 DOERR } IFT @ Directory object?
    PATH SWAP OVER DUP SIZE GET @ Current directory variable
    #62A02h SYSEVAL EVAL } @ SAFESTO is "supported"

  IFTE 2 MENU \>> @ End in VAR menu so you can see when it's done.

-----------------------------------------------------------------

   Chosen method     Checksum    Size (on stack)
   --------------    --------    ---------------
   START ... NEXT     #9417h         342
   ASC\->             #CEB0h         294
   Hex strings         #952h         327.5
   (none of above)     #C84h         232.5
   (omit extra checks)               -78.5

-----------------------------------------------------------------

@ If you don't have HACK.LIB, then you need this version of 'PG':
@ 'name' or { list } ==> (everything purged, including directories)

\<< DUP TYPE 5 \=/ { 1 \->LIST } IFT
 IF DUP SIZE THEN
 1 \<< IF DUP VTYPE 15 == THEN PGDIR ELSE PURGE END \>> DOLIST
 ELSE DROP END \>> @ Checksum: #E859h, Size (on stack): 108.5 bytes
@ Store into 'PG' in HOME

@ S/SX owners who use this 'PG' program
@ also need the DOLIST program below.

-----------------------------------------------------------------

@ DOLIST simulator for S/SX only (for use with above program)
@ { list N } ... { list 1 } N object ==> { results }
@ The object is usually a program, or the name of a program.

@ The first element from each list is put on the stack, then
@ the object is evaluated; this is repeated for the next element
@ in each list, etc., until all elements have been processed.
@ All results finally remaining on the stack are collected into one list,
@ except that if no results at all are produced, then no list is returned.

@ This works the same as in the G/GX, but if you omit the count of
@ how many lists to use, it always defaults here to 1, rather than
@ trying to figure it out by analyzing the executable object.

\<< OVER TYPE 5 == { 1 SWAP } IFT SWAP 0 RND SWAP OVER 2 + ROLLD \->LIST
 DUP SIZE DEPTH 3 - \-> ?f ?l ?n ?d \<< 1 ?l 1 GET SIZE
 FOR ?i 1 ?n FOR ?j ?l ?j GET ?i GET NEXT ?f EVAL
 NEXT DEPTH ?d - \->LIST \>> DUP SIZE NOT { DROP } IFT
\>> @ Store into 'DOLIST' in HOME
@ Checksum: #C646h, Size (on stack): 201.5 bytes

@ Names beginning with "?" are used to avoid potential conflict with
@ any local variable names which might be used in the evaluated object.

----------------------------------------------------------------------
With best wishes from: John H Meyers <jhmeyers@mum.edu>