"xz -v < regular_file > out.xz" doesn't display the percentage
and estimated remaining time because it doesn't even try to
check the input file size when input is read from stdin.
This could be improved but for now there's just a comment
to remind about it.
It worked for one input file since the counters are zero when
xz starts but they weren't reset when starting a new file in
passthru mode. For example, if files A, B, and C are one byte each,
then "xz -dcvf A B C" would show file sizes as 1, 2, and 3 bytes
instead of 1, 1, and 1 byte.
Example:
$ xz -dc --single-stream good-0-empty.xz
xz: good-0-empty.xz: Internal error (bug)
The code, that is tries to catch some input file issues early,
didn't anticipate LZMA_STREAM_END which is possible in that
code only when --single-stream is used.
Now files with unsupported check will make xz display
a warning, set the exit status to 2 (unless --no-warn is used),
and then decompress the file normally. This is how it was
supposed to work since the beginning but this was broken by
the commit 231c3c7098, that is,
a little before 5.0.0 was released. The buggy behavior displayed
a message, set exit status 1 (error), and xz didn't attempt to
to decompress the file.
This doesn't matter today except for special builds that disable
CRC64 or SHA-256 at build time (but such builds should be used
in special situations only). The bug matters if new check type
is added in the future and an old xz version is used to decompress
such a file; however, it's likely that such files would use a new
filter too and an old xz wouldn't be able to decompress the file
anyway.
The first hunk in the commit is the actual fix. The second hunk
is a cleanup since LZMA_TELL_ANY_CHECK isn't used in xz.
There is a test file for unsupported check type but it wasn't
used by test_files.sh, perhaps due to different behavior between
xz and the simpler xzdec.
xz --flush-timeout=2000, old version:
1. xz is started. The next flush will happen after two seconds.
2. No input for one second.
3. A burst of a few kilobytes of input.
4. No input for one second.
5. Two seconds have passed and flushing starts.
The first second counted towards the flush-timeout even though
there was no pending data. This can cause flushing to occur more
often than needed.
xz --flush-timeout=2000, after this commit:
1. xz is started.
2. No input for one second.
3. A burst of a few kilobytes of input. The next flush will
happen after two seconds counted from the time when the
first bytes of the burst were read.
4. No input for one second.
5. No input for another second.
6. Two seconds have passed and flushing starts.
The same code sequence repeats so it's nicer as a separate function.
Note that in one case there was no test for opt_mode != MODE_TEST,
but that was only because that condition would always be true, so
this commit doesn't change the behavior there.
When input blocked, xz --flush-timeout=1 would wake up every
millisecond and initiate flushing which would have nothing to
flush and thus would just waste CPU time. The fix disables the
timeout when no input has been seen since the previous flush.
"xz -dcfv not_an_xz_file" crashed (all four options are
required to trigger it). It caused xz to call
lzma_get_progress(&strm, ...) when no coder was initialized
in strm. In this situation strm.internal is NULL which leads
to a crash in lzma_get_progress().
The bug was introduced when xz started using lzma_get_progress()
to get progress info for multi-threaded compression, so the
bug is present in versions 5.1.3alpha and higher.
Thanks to Filip Palian <Filip.Palian@pjwstk.edu.pl> for
the bug report.
The patch is quite long but it's mostly about adding new #ifdefs
to omit code when encoders or decoders have been disabled.
This adds two new #defines to config.h: HAVE_ENCODERS and
HAVE_DECODERS.
Previously, --block-list and --block-size only worked together
in threaded mode. Boundaries are specified by --block-list, but
--block-size specifies the maximum size for a Block. Now this
works in single-threaded mode too.
Thanks to James M Leddy for the original patch.
Now if --block-list is used in threaded mode, the encoder
won't need to flush at each Block boundary specified via
--block-list. This improves performance a lot, making
threading helpful with --block-list.
The flush timer was reset after LZMA_FULL_FLUSH but since
LZMA_FULL_BARRIER doesn't flush, resetting the timer is
no longer done.
Now liblzma only uses "mythread" functions and types
which are defined in mythread.h matching the desired
threading method.
Before Windows Vista, there is no direct equivalent to
pthread condition variables. Since this package doesn't
use pthread_cond_broadcast(), pre-Vista threading can
still be kept quite simple. The pre-Vista code doesn't
use anything that wasn't already available in Windows 95,
so the binaries should run even on Windows 95 if someone
happens to care.
When --flush-timeout=TIMEOUT is used, xz will use
LZMA_SYNC_FLUSH if read() would block and at least
TIMEOUT milliseconds has elapsed since the previous flush.
This can be useful in realtime-like use cases where the
data is simultanously decompressed by another process
(possibly on a different computer). If new uncompressed
input data is produced slowly, without this option xz could
buffer the data for a long time until it would become
decompressible from the output.
If TIMEOUT is 0, the feature is disabled. This is the default.
This commit affects the compression side. Using xz for
the decompression side for the above purpose doesn't work
yet so well because there is quite a bit of input and
output buffering when decompressing.
The --long-help or man page were not updated yet.
The details of this feature may change.
Testing for end of file was no longer correct after full flushing
became possible with --block-size=SIZE and --block-list=SIZES.
There was no bug in practice though because xz just made a few
unneeded zero-byte reads.
This switches units from microseconds to milliseconds.
New clock_gettime(CLOCK_MONOTONIC) will be used if available.
There is still a fallback to gettimeofday().
There was somewhat illogical behavior when --extreme was
specified and mixed with custom filter chains.
Before this commit, "xz -9 --lzma2 -e" was equivalent
to "xz --lzma2". After it is equivalent to "xz -6e"
(all earlier preset options get forgotten when a custom
filter chain is specified and the default preset is 6
to which -e is applied). I find this less illogical.
This also affects the meaning of "xz -9e --lzma2 -7".
Earlier it was equivalent to "xz -7e" (the -e specified
before a custom filter chain wasn't forgotten). Now it
is "xz -7". Note that "xz -7e" still is the same as "xz -e7".
Hopefully very few cared about this in the first place,
so pretty much no one should even notice this change.
Thanks to Conley Moorhous.
Now the following works as you would expect:
echo foo | xz > foo.xz
echo bar | xz >> foo.xz
( xz -dc --single-stream ; xz -dc --single-stream ) < foo.xz
Note that it doesn't work if the input is not seekable
or if there is Stream Padding between the concatenated
.xz Streams.
For several people, the limiter causes bigger problems that
it solves, so it is better to have it disabled by default.
Those who want to have a limiter by default need to enable
it via the environment variable XZ_DEFAULTS.
Support for environment variable XZ_DEFAULTS was added. It is
parsed before XZ_OPT and technically identical with it. The
intended uses differ quite a bit though; see the man page.
The memory usage limit can now be set separately for
compression and decompression using --memlimit-compress and
--memlimit-decompress. To set both at once, -M or --memlimit
can be used. --memory was retained as a legacy alias for
--memlimit for backwards compatibility.
The semantics of --info-memory were changed in backwards
incompatible way. Compatibility wasn't meaningful due to
changes in the memory usage limiter functionality.
The memory usage limiter info is no longer shown at the
bottom of xz --long -help.
The memory usage limiter support for removed completely from xzdec.
xz's man page was updated to match the above changes. Various
unrelated fixes were also made to the man page.
message_filters_to_str() converts the filter chain to
a string. message_filters_show() replaces the original
message_filters().
uint32_to_optstr() was also added to show the dictionary
size in nicer format when possible.
The opening of the destination file is now delayed a little.
The coder is initialized, and if decompressing, the memory
usage of the first Block compared against the memory
usage limit before the destination file is opened. This
means that if --force was used, the old "target" file won't
be deleted so easily when something goes wrong very early.
Thanks to Mark K for the bug report.
The above fix required some changes to progress message
handling. Now there is a separate function for setting and
printing the filename. It is used also in list.c.
list_file() now handles stdin correctly (gives an error).
A useless check for user_abort was removed from file_io.c.
a regular file.
Sparse file creation can be disabled with --no-sparse.
I don't promise yet that the name of this option won't
change before 5.0.0. It's possible that the code, that
checks when it is safe to use sparse output on stdout,
is not good enough, and a more flexible command line
option is needed to configure sparse file handling.
files as is to standard output.
This feature is needed to be more compatible with gzip's
behavior. This was more complicated to implement than it
sounds, because the way liblzma is able to return errors with
files of only a few bytes in size. xz now has its own file
type detection code and no longer uses lzma_auto_decoder().
