Merge trunk r3945 (refactor two README files)

README

@@ -2,55 +2,109 @@ $Id$
 
 1. INTRODUCTION::
 
-     This collection of scripts, known as jhalfs, strives to create
+     The scripts in this directory implement an automation of the building
-  accurate makefiles from the Linux From Scratch book series XML files.
+  of a GNU/LInux system, as described in the Linux From Scratch book series.
-  This software is an evolution of the original "jhalfs-0.2" code developed
+  The name of the project is jhalfs: in that name, "alfs" stands for
-  by Jeremy Huntwork.
+  "automated linux from scratch", and the initials "jh" have been kept since
-
+  the original "jhalfs-0.2" code developed by Jeremy Huntwork.
-     The usage of this script assumes you have read and are familiar with
-  the book(s) and, therefore, the configuration variables found in menuconfig
-  interface will have meaning to you.
 
      The list of supported books can be found at
-  http://wiki.linuxfromscratch.org/alfs/wiki/SupportedBooks
+  http://wiki.linuxfromscratch.org/alfs/wiki/SupportedBooks.
 
-  NOTES::
+     The documentation is split among various README.* files. Here is a list
-  *.  The resulting Makefile takes considerable time to run to completion.
+  of what is in which:
-  Lay in a supply of caffeine beverages.
+    - README (this file): instructions to use the LFS book. This should be
+      enough if you just want to build a base system as per the LFS book. It is
+      also a required reading for all the other projects.
+    - README.CLFS: supplementary instructions to use the CLFS book series.
+    - README.BLFS: instructions to install an automated build infrastructure
+      for the BLFS book. There are two ways to do so: (i) install the
+      tools at the end of an LFS build (CLFS is not supported in that case), or
+      (ii) install the tools on an already running system. Both methods are
+      described in that file.
+    - README.CUSTOM: instructions to run custom commands either during the xLFS
+      build, at the end of a xLFS build. Note that you will not find
+      instructions on how to write those commands, but some examples are
+      available.
+    - README.PACKAGE_MANAGEMENT: instructions to use package management during
+      the build (only for LFS, patches welcome for CLFS...)
+    - README.HLFS: very short file explaining why you cannot use HLFS with the
+      present tool version.
 
-  *.  It is recommended that you temporarily unpack your linux kernel,
+     Other sources of information are the context help in the menu interface,
-  run <make menuconfig>, configure the kernel as per the book and save
+  and the xLFS books themselves.
-  the resulting .config file.
-
-  *.  Read carefully this file and the other README.* files before using
-  this tool.
 
 2. PREREQUISITES::
 
-     To use this tool you MUST:
+     As said elsewhere, it is strongly advised that you first build manually
-
+  a complete system before attempting to automate the build.
-     - have experience building {c,h,b}LFS packages
-     - know how to edit and write shell scripts
-     - know how a Makefile works
-     - be able to trace build failures and to find what is causing them
-       (user error, package bug, {c,h,b}LFS command bug, or jhalfs code bug)
-
-     If you do not have the above skills, please don't use this tool.
 
+     Of course the "Host System Requirements" should be fulfilled. The needed
+  supplementary packages are detailed at the bottom of the page:
+  http://www.linuxfromscratch.org/alfs/download.html
 
 3. INSTALLATION::
 
-     No installation is required. You should just run <make> in this directory.
+     No installation is required. You may want to move the files in this
+  directory to a convenient location, and then follow the instructions below.
 
 4. CONFIGURATION::
 
-     Configuration is done through a menu based interface. See the section
+  4.1. CONFIGURATION OF THE TOOLS:
-     RUNNING, for details.
+       There is no configuration of the tools themselves. The various
+    parameters for the build are set through a menu driven interface. See
+    the section RUNNING below for details.
+
+  4.2. PRELIMINARY TASKS:
+       This tool has no support at all for creating a partition and a mount
+    point for the built system. You should follow the book up to the section
+    "Mounting the new partition". Note that the default name for the
+    partition mount point is "/mnt/build_dir", instead of /mnt/{c,}lfs.
+    You can change that default to anything you'd like in the menu, so you
+    may name it /mnt/lfs, or whatever you like. One important point is that
+    the user you are logged in as (and not the (c)lfs user) should have write
+    permission to the mounted directory. We'll use the name /mnt/build_dir
+    in the sequel.
+
+       The tool can download the needed packages for you, or you may download
+    them yourself. The tool may optionally use a package archive directory
+    where the downloaded packages are stored. That directory name may be made
+    available to the tool in two ways: (i) export the SRC_ARCHIVE variable,
+    for example SRC_ARCHIVE=/usr/src, (ii) enter the name at the "Package
+    Archive Directory" menu prompt. Note that the user should have write
+    permission to that directory. If a needed package is found in that
+    directory, it is copied to /mnt/build_dir/sources, if not, it is
+    downloaded to that directory and copied to /mnt/build_dir/sources,
+    except if found in /mnt/build_dir/sources, in which case, it is just
+    copied to $SRC_ARCHIVE. If you want the tool to download packages and you
+    do not want to archive them, just unset SRC_ARCHIVE, and keep the
+    default entry for "Package Archive Directory". If you choose to download
+    the packages by yourself, you should download (or copy) them to
+    /mnt/build_dir/sources directly.
+    
+       If you want to build the kernel as part of the automated build, select
+    "Build the kernel" in the menu. Then, a configuration file must be
+    provided. In order to do so, it is recommended to download the kernel
+    tarball, unpack it, run <make menuconfig>, configure the kernel as per
+    the book, and save the resulting .config file to a location where it can
+    be retrieved later on (a convenient location and name is
+    $SRC_ARCHIVE/config-<arch>-<kernel version>-<config details>).
+
+       Another file you may provide is the fstab file. To use it, select
+    "Use a custom fstab file" in the menu interface, and enter the name of
+    the file where asked. As for the kernel configuration, this file has to
+    be prepared before running the menu. A convenient location and name is
+    $SRC_ARCHIVE/fstablfs.
+
+       At a more advanced level, you may want to supply custom commands
+    to be run at the end of (C)LFS build. Scripts containing those commands
+    are located in the ./custom/config directory. Examples are given in
+    ./custom/examples. A template is provided as ./custom/template. See
+    README.CUSTOM for more details.
 
 5. RUNNING::
 
-     The command <make> will launch a menu based configuration program. You will
+     The command <make> will launch a menu based configuration program. The
-     recognize the layout from building the kernel or uClibc/BusyBox. The
   underlying menu code was borrowed from BusyBox and slightly modified for
   our use.
 
@@ -64,21 +118,42 @@ $Id$
   packages in BLFS. See README.BLFS to know how. If you choose any other
   book, you'll have to configure the settings and the build parameters
   from the menu. Note that you may choose to install the blfs tools onto
-     the newly built system (see below). It is not the same thing as choosing
+  the newly built system. It is not the same thing as choosing
   the BLFS book in the menu, which will install the blfs tools on the
   currently running system.
 
+     The "General Settings" menu is where the "Build Directory" name is to be
+  entered. Other entries in that menu select what the tool should do. The
+  "Run the Makefile" entry selects whether the tool will start the build
+  automatically after generating the needed files. The "Rebuild files" selects
+  whether to clean the build directory before doing anything else. To protect
+  against removing important files, this can only be done in an empty directory,
+  or a directory previously populated by the tool.
+
+     The "Build Settings" menu is where various options for the build can be
+  selected. Two options, "Use a custom fstab file" and "Build the kernel",
+  have been described above. "Do not use/display progress_bar", if set, will
+  prevent a progress bar to be displayed during the build. That may be useful
+  on slow machine. The other options should be self explanatory, using either
+  the online help or book reading.
+
+     The "Advanced Features" menu is for various maintenance tasks, like
+  testing the build instructions or reporting build statistics. One useful
+  option is "Optimization and parallelisation". It is not recommended to use
+  it for setting compiler optimization flags, although it is possible, but
+  if you select it, you'll be able to select the number of parallel `make'
+  jobs, which allows much faster builds on modern multicore CPUs.
 
      Once you have set the parameters and saved the configuration, the script
   is launched. Its aim is to extract instructions from the selected book
   to generate scripts, and to generate a Makefile, which allows running
   the scripts in the right order. The script verifies first that the host
-     can run it and build the xLFS system, then validates the configuration
+  can run itself and build the xLFS system, then validates the configuration
   and lists the parameters. At this point, you may choose to quit or to
   continue with the listed parameters. The script will then proceed to
   generate the Makefile and the build scripts, optionally download
   packages, and eventually verify the host prerequisite. If you have
-     selected "Run the makefile", the command make is launched in the
+  selected "Run the makefile", the command <make> is launched in the
   adequate directory, and the build begins. If not, you'll have to run
   "make" manually, for example: "make -C /mnt/build_dir/jhalfs", if you
   have used the default parameters (see the layout under $BUILDDIR in the
@@ -93,55 +168,9 @@ $Id$
 
   NOTE::
       If you run the jhalfs script directly the only function you can select
-         is to display the version number running <./jhalfs -v>
+      is to display the version number by running <./jhalfs -v>
 
-6. BLFS_TOOL SUPPORT::
+6. LAYOUT::
-
-     For books that support it (only LFS for jhalfs version 2.4),
-     there is an option to install an automated framework for building BLFS
-     packages. It is called blfs-tool. When you tick `BOOK Settings/Add
-     blfs-tool support' in jhalfs configuration menu, the tools are
-     installed in $BLFS_ROOT (default /blfs_root) on the xLFS system,
-     and a few dependencies (which you may select) are built at the
-     end of the jhalfs run, before the custom tools. The instructions for
-     building the dependencies are taken from the BLFS book.
-
-     (TODO: blfs-tools have not been tested with current (version 3.0) of CLFS,
-     and certainly need some adaptation to run)
-     WARNING:: If you add blfs-tool support on a CLFS Sysroot build
-               you MUST edit the scripts to fix the installation paths.
-
-     After booting the new xLFS system some steps are needed to finish
-     the installation of the automated tools:
-
-       - A user account must be created. You must be logged on that user
-         account to use blfs-tool. This is not strictly necessary,
-         since the packages can be built as root, too, but it is
-         never a good idea to build packages as root.
-
-       - Move /blfs-root to that user's home and change ownership of the
-         directory and files to the user.
-
-       - Give the user read and write privileges over the $TRACKING_DIR
-         directory and the files that it contains.
-
-       - Configure sudo, adding the needed privileges for the user. For
-         newer sudo version, do not forget to add a line Defaults secure_path=
-         containing /sbin and /usr/sbin (in /etc/sudoers), otherwise some
-         executables are not found.
-
-       - Although it is not strictly necessary, it is recommended to install
-         the bash shell startup files (as per `3.After LFS Configuration
-         Issues' of the BLFS book), as some instructions in BLFS rely on
-         their being present.
-
-     We assume that blfs-tool will be used on a running fresh xLFS system.
-     To use it to build BLFS packages from the chroot jail is also possible,
-     but not supported.
-
-     To know how to use blfs-tool, see README.BLFS.
-
-7. LAYOUT::
 
         /BLFS (see README.BLFS)
 
@@ -204,21 +233,16 @@ $Id$
         jhalfs
         blfs-tool
 
-8. FAQ::
+7. FAQ::
-    Q. "This 'help' file is very sparse"
+    Q. "It doesn't work"
-    A. Yes, it is. This tool, jhalfs, is for those who understand the LFS books
+    A. There are several reasons why it may be so. One possibility is the
-       and wish to automate the build. 99% of any problems that arise can be
+       following:jhalfs was designed to work against the development versions
-       solved by reading the book(s).
+       of the LFS series of books. Consequently changes in a book(s) sometimes
-
+       breaks older versions of jhalfs. Before you start pulling out your hair
-    Q. "It doesn't work!"
+       download the latest version of jhalfs to see if that solves your
-    A. Yes it does, try >> make
+       problem. Note that it may be the other way around. If you want to build
-       Remember you must have 'sudo' privileges.
+       an old version of the book, you may have to downgrade you jahlfs
-
+       version.
-    Q. "It still doesn't work"
-    A. jhalfs was designed to work against the development versions of the LFS
-       series of books. Consequently changes in a book(s) sometimes breaks older
-       versions of jhalfs. Before you start pulling out your hair download the
-       latest version of jhalfs to see if that solves your problem.
 
     Q. "How do I specify the build location?"
     A. The original LFS document worked against the well known location

README.BLFS

@@ -2,10 +2,6 @@ $Id$
 
 1. INTRODUCTION::
 
-     If you want to add blfs-tool support into an xLFS base system build,
-  read the "BLFS_TOOL SUPPORT" section found in the README and be sure
-  to follow the after-booting installation instructions.
-
      To automate package builds from the BLFS book instructions is a huge
   task. Some of the issues are: the BLFS book isn't linear; some package
   pages use a custom layout; there are circular dependencies; several
@@ -20,55 +16,88 @@ $Id$
 
 2. PREREQUISITES::
 
-     To use this tool you MUST:
+     In addition to a full LFS system, the following packages and their
+  dependencies are needed by this tool:
+    - required: libxml2, libxslt, DocBook XML DTD
+    - recommended: wget (to download the package tarballs) and sudo (to build
+      as a user)
+    - optional: lynx (allows to read the generated linearized book), GPM (to
+      cut and paste commands from the book), subversion (to update the book
+      sources), openssl (used by wget for all https:// sites)
+  Note that the optional dependencies are recommended for ease of use of the
+  tool.
 
-     - have experience building BLFS packages
+     You should also have the following personal skills:
-     - know how to edit and write shell scripts
+    - Ability to write and debug shell scripts: as said in the introduction,
-     - know how a Makefile works
+      not all the generated scripts can be used directly. They need to be
-     - be able to trace build failures and to find what is causing it
+      edited to produce an error free build.
-       (user error, package bug, BLFS command bug, or jhalfs code bug)
+    - Ability to debug build failures, like missing dependencies or
+      installation directories not known to the system (when you install in
+      /opt for example).
+    - Ability to choose the tools you need to configure and administrate
+      your system: in the BLFS book, nothing is mandatory, nothing is
+      useless. You are on your own in choosing what to build, but wrong
+      decisions may lead to a non functional system...
 
-     If you do not have the above skills, please don't use this tool.
+3. INSTALL::
 
+     There are two ways to install the BLFS tools on an LFS system, described
+  in paragraphs 3.1 and 3.2, respectively:
 
-3. USAGE::
+  3.1 INSTALLATION ON A RUNNING SYSTEM
 
-     Due to the complexity of the BLFS book, the scripts and Makefile
+     Select "Use Book --> Beyond Linux From Scratch" in the jhalfs menu:
-     generation is done in several steps:
+  The tools are installed in $HOME$BLFS_ROOT (the default for $BLFS_ROOT
+  is /blfs_root). The BLFS book is downloaded or copied to its directory.
+  The tracking directory (see below) is initialized but not created: before
+  the installation, you should ensure the tracking directory (default location
+  /var/lib/jhalfs/BLFS) exists and is writable by the user. After the
+  intallation, you should perform the following additional steps:
 
-  3.1  INSTALLED PACKAGES TRACKING SYSTEM::
+    - Configure sudo, adding the needed privileges for the user. For
+      newer sudo version, do not forget to add a line `Defaults secure_path='
+      containing /sbin and /usr/sbin (in /etc/sudoers), otherwise some
+      executables are not found.
 
-       This tool includes a very simple tracking system to log which packages
+    - Although it is not strictly necessary, it is recommended to install
-    have been installed using the tool. It is used to skip installed packages
+      the bash shell startup files (as per `3.After LFS Configuration
-    from target selection menu and to test if an installed package has been
+      Issues' of the BLFS book), as some instructions in BLFS rely on
-    updated in the BLFS book. Do not rely on this feature as a package
+      their being present.
-    management tool.
 
-       The tracking system itself is an XML file: instpkg.xml. It is
+  3.2 INSTALLATION ON A JUST BUILT xLFS SYSTEM
-    initialized when <make> is first run in blfs_root. It resides in a
-    directory, which is created when needed during the process of building
-    custom tools or blfs dependencies, right after xLFS. You can specify
-    that directory location in the blfs-tools sub-menu of jhalfs. You may
-    need to update permissions and/or ownership of this directory before
-    using the blfs tool (see README in jhalfs).
 
-       The default location of the tracking directory is /var/lib/jhalfs/BLFS.
+    For books that support it (only LFS for jhalfs version 2.4),
-    NB : after the initial build, that directory is only used to contain
+  there is an option to install the BLFS tools right after building
-    instpkg.xml, unless custom tools have been built. In the latter case,
+  the xLFS system: just tick `BOOK Settings/Add blfs-tool support' in
-    it also contains empty files whose name are "$PKG-$VERSION" for each
+  jhalfs configuration menu. The tools are installed in $BLFS_ROOT
-    versionned package built. The information about those packages is
+  (default /blfs_root) on the xLFS system, and the dependencies are built
-    included into instpkg.xml the next time the tool is run.
+  at the end of the jhalfs run, before the custom tools.
 
-  3.2  BLFS_TOOL INSTALLATION::
+  (TODO: blfs-tools have not been tested with current (version 3.0) of CLFS,
+  and certainly need some adaptation to run)
 
-    3.2.1 Normal install
+  After booting the new xLFS system some steps are needed to finish
-       The tools are installed just after the building of xLFS, if the
+  the installation of the automated tools:
-    appropriate options have been selected in the building menu, as per
+
-    jhalfs README. If you forgot to select the options and xLFS has been
+    - A user account must be created. You must be logged on that user
-    built, it is possible to go back to selecting the appropriate
+      account to use blfs-tool. This is not strictly necessary,
-    BLFS tools options in the jhalfs menu, then tick `Run makefile'
+      since the packages can be built as root, too, but it is
-    and not `Rebuild files'. You obtain a /blfs_root directory in the
+      never a good idea to build packages as root.
-    root directory of the new xLFS system, which contains the followings:
+
+    - Move /blfs-root to that user's home and change ownership of the
+      directory and files to the user.
+
+    - Give the user read and write privileges over the $TRACKING_DIR
+      directory and the files that it contains.
+
+    - Configure sudo and add the bash shell startup files, as described
+      above
+
+     We assume that the BLFS tools will be used on a booted xLFS system.
+  Using them to build BLFS packages in a chroot jail is also possible,
+  but not supported.
+
+  3.3 DIRECTORY LAYOUT IN THE $BLFS_ROOT DIRECTORY
 
     blfs-xml/*              SVN tree of the selected BLFS book version
     lib/constants.inc       functions libraries 
@@ -98,17 +127,7 @@ $Id$
                             database and the tracking file.
     envars.conf             envars needed when running the target build scripts
 
-    3.2.2 Install to an already running LFS/BLFS system
+    Working files: several files are generated when first running the tool
-       If you forgot to install the tools when building xLFS, or want to try
-    the tools, you can select the BLFS book from the jhalfs menu. It will
-    run a script, which creates the above hierarchy in your home directory and
-    initialize the tracking file. You have first to make sure that the tracking
-    dir exists and is writable by the user. You may also populate it with
-    (empty) files whose names are of the form package-version, for installed
-    packages, so that they are included into the tracking file.
-
-    3.3.3 Working files
-        Several files are generated during the process:
 
     packages.xml            auto-generated packages database
     Config.in               input file for the menu driven choices
@@ -118,13 +137,38 @@ $Id$
     book-html/*             the linearized book rendered in html
     scripts/*               the scriptlets
 
+  3.4 INSTALLED PACKAGES TRACKING SYSTEM:
+
+       This tool includes a very simple tracking system to log which packages
+    have been installed using the tool. It is used to skip installed packages
+    from target selection menu and to test if an installed package has been
+    updated in the BLFS book. Do not rely on this feature as a package
+    management tool.
+
+       The tracking system itself is an XML file: instpkg.xml. It is
+    initialized when <make> is first run in blfs_root. It resides in a
+    directory, which is created when needed during the process of building
+    custom tools or blfs dependencies, right after xLFS. You can specify
+    that directory location in the blfs-tools sub-menu of jhalfs. You may
+    need to update permissions and/or ownership of this directory before
+    using the blfs tool (see README in jhalfs).
+
+       The default location of the tracking directory is /var/lib/jhalfs/BLFS.
+    NB : after the initial build, that directory is only used to contain
+    instpkg.xml, unless custom tools have been built. In the latter case,
+    it also contains empty files whose name are "$PKG-$VERSION" for each
+    versionned package built. The information about those packages is
+    included into instpkg.xml the next time the tool is run.
+
+4. USAGE::
+
        From now on, all the work must be done from inside the installation
     root directory.
 
-       You may move that directory to the $HOME of a non root user, or build
+     Due to the complexity of the BLFS book, the scripts and Makefile
-    as root from that directory.
+     generation is done in several steps:
 
-  3.3  UPDATING BOOK SOURCES::
+  4.1  UPDATING BOOK SOURCES::
 
        If you are using the development book version and you want to update
     installed packages to the latest version found in that book, you need to
@@ -142,7 +186,7 @@ $Id$
     with a new version in the book will be available for target selection
     and used to solve dependencies.
 
-  3.4  CONFIGURING AND PARSING THE BOOK::
+  4.2  CONFIGURING AND PARSING THE BOOK::
 
        The next step is to create a book and build scripts in dependency
     build order for one or several packages.
@@ -180,7 +224,7 @@ $Id$
        There is also another directory, "dependencies" that contains files
     generated while resolving dependencies.
 
-  3.5  EDITING BUILD SCRIPTS::
+  4.3  EDITING BUILD SCRIPTS::
 
        Now it is time to review the generated book and scripts, making any
     changes to the scripts necessary to fix generation bugs or to suit your
@@ -197,7 +241,7 @@ $Id$
       Also, review and edit envars.conf. This file is used to set global envars
     needed by the build scripts.
 
-  3.6  CREATING THE MAKEFILE::
+  4.4  CREATING THE MAKEFILE::
 
        When the build scripts are ready to be run, the Makefile can be
     created. Create an empty directory (for example "mkdir work") and cd
@@ -206,7 +250,7 @@ $Id$
     Review the Makefile, and, if all looks sane, start the build by running
     "make".
 
-4. GENERATED BUILD SCRIPTS ISSUES::
+5. GENERATED BUILD SCRIPTS ISSUES::
 
       In this section, known issues with the generated build scripts are
    discussed. They are due to build procedures and/or BLFS layout
@@ -222,14 +266,14 @@ $Id$
    to another name (using the xxx-a- fields), and possibly renaming the xxx-a-
    fields of each involved script.
 
-   4.1  BLFS BOOTSCRIPTS::
+   5.1  BLFS BOOTSCRIPTS::
 
         Normally, bootscript installation should work. On the other hand, the
      book does not give instruction for running them, so you might have to
      manually insert "/etc/init.d/rc.d/<initscript> start" at some place during
      the build.
 
-   4.2  PACKAGE CONFIGURATION::
+   5.2  PACKAGE CONFIGURATION::
 
         For those packages that have a "Configuration" section, you should
      edit the build script to fit the needs of your system. Sometimes, the
@@ -237,7 +281,7 @@ $Id$
      line 'source /etc/profile', which ensures that the proper environment
      variables are used.
 
-   4.3  PAGES WITH TWO OR MORE PACKAGES::
+   5.3  PAGES WITH TWO OR MORE PACKAGES::
 
         For example: sane, poppler, audacious, freetts, which, etc.
 
@@ -248,7 +292,7 @@ $Id$
      name (for example PKG1, PKG2, etc) after each download. The unpacking
      instructions may need to be repeated for each tarball in turn.
 
-   4.4  XORG7
+   5.4  XORG7
 
         The book has special page layouts for the Xorg7 packages. The tool
      breaks those pages into individual pages for each packages in the linear
@@ -257,13 +301,13 @@ $Id$
         To build the whole Xorg7 chapter, select twm. The (recommended)
      dependency chain brings in the whole set of Xorg packages.
 
-   4.5  PATCHES
+   5.5  PATCHES
 
         Please, make sure that all scripts have the commands to download/apply
      the required patches. Due to book layout issues, some patches may be
      missing (as of BLFS 8.0, all the patches seem to be downloaded).
 
-   4.6  ROOT COMMANDS
+   5.6  ROOT COMMANDS
 
         If building as a normal user (the default setting), be sure that all
      commands that require root privileges are run using sudo. Also make sure
@@ -292,7 +336,7 @@ $Id$
 
         Due to book layout issues, some sudo commands may be missing.
 
-   4.7  OTHERS
+   5.7  OTHERS
 
         There may be other issues that we are not aware of. If you find
      any, please report it to <alfs-discuss@linuxfromscratch.org>.