The latest release of Tinderbox can be downloaded from http://tinderbox.marcuscom.com.
Tinderbox lives in the MarcusCom CVS repository under the portstools module.
Recent FreeBSD (development happens on -CURRENT only, but Tinderbox is known to work on RELENG_6 and RELENG_7, too), Perl 5.8 or later (lang/perl5.8), and either MySQL 4.1 or later (databases/mysql41-server) or PostgreSQL 7.4 or later (databases/postgres74-server).
If MySQL is used, the MySQL Perl module must also be installed (databases/p5-DBD-mysql41). If PostgreSQL is used, the Pg Perl module must als be installed (databases/p5-DBD-Pg).
Please note, however, that is is recommended that either MySQL 5.0 or PostgreSQL 8.1 (or later) be used. There are currently no plans to make this a requirement, but it may happen in the future.
PHP 4.x is now EOL, so the web front-end (found in the webui subdirectory) requires PHP 5 (lang/php5), PEAR::DB (databases/pear-DB), PHP Session (www/php5-session), and one of the PHP MySQL (databases/php5-mysql) or PHP Pgsql (databases/php5-pgsql) depending on your choice of backend database above. This code should not require "register_globals=on" in your php.ini, but there have been sporadic reports of it failing without this set.
If you will be sending emails from Tinderbox (e.g. build failure or build completion emails) you must install net/p5-Net to get the Net::SMTP Perl module.
Tinderbox now determines where to do all of its work relative to the location of the tc program, passed around to other functions by means of the ${pb} environmental variable (think: package build). Historically, this was the /space subdirectory (tc was usually found in /space/scripts/tc).
Henceforth, the top-level directory will be referred to as ${pb}, which you should substitute for whatever root directory you are using.
Run tc to setup configuration files and initialize the Tinderbox database:
# cd ${pb}/scripts && ./tc Setup
If you are going to be using the web front-end, edit webui/inc_ds.php.dist for your database setup. Then copy this file to inc_ds.php.
If your Tinderbox host does not have administrative access to the database server, you must perform the next few steps by hand. Else, skip to step 7.
For PostgreSQL users, make sure the Tinderbox user owns the Tinderbox database as well as all the tables within that database.
Populate the database with the Tinderbox schema:
MySQL
# cd sql
# ./genschema mysql | mysql -u{DB_ADMIN} -p -h {DB_HOST}
{DB_NAME}
PostgreSQL
# cd sql
# ./genschema pgsql | psql -U {DB_USER} -W -h {DB_HOST} -d {DB_NAME}
Where {DB_HOST} and {DB_NAME} are the values of $DB_HOST and $DB_NAME from ds.ph respectively, {DB_ADMIN} is the database administrator username, and {DB_USER} is the Tinderbox user. Note: do not do this if you are upgrading! If you do, you will overwrite all of your previous data. If schema changes are required for upgrading, a separate upgrade schema file will be included, and instructions will be available at http://tinderbox.marcuscom.com.
Create the required Jails using the tc command. A
Jail is nothing more than a version of FreeBSD. For example, to
create a jail for FreeBSD 6.3-RELEASE:
# cd ${pb}/scripts && ./tc createJail -j
6.3 -d "FreeBSD 6.3-RELEASE" -t
RELENG_6_3_0_RELEASE -u CVSUP
Note: all Jail names MUST begin with their
FreeBSD major version number. That is, the following is an
illegal jail name: FreeBSD-6.3.
Hint: It is recommended that the Jail begin with the FreeBSD major.minor numbers (i.e. 6.3-FreeBSD instead of just 6-FreeBSD) as this may prove useful when using things such as Hooks.
NOTE: The tc script use cvsup12 as
their default cvsup mirror. If you would like to use
another server, enable cvsup use the -H and
-C command line arguments to the tc script
respectively. For example, to use cvsup2.freebsd.org and
enable cvsup compression for all Jail Jail updates, use
the following command:
# cd ${pb}/scripts && ./tc createJail -j
6-STABLE -d "FreeBSD 6-STABLE" -t RELENG_6 -u
CVSUP -C -H cvsup2.freebsd.org
Tinderbox may undergo a variety of changes between versions. In
order to make sure your Tinderbox stays fully operational after
copying over a new version's distribution, you should run the
following command:
# cd ${pb}/scripts && ./tc Upgrade
Note: This command may require administrative access to the database. If such access is not available from your Tinderbox host, you may have to load an upgrade schema file manually. The upgrade script will provide such instructions if needed.
Tinderbox upgrades to 3.0 are supported only from Tinderbox 2.4.x. If you are running a release prior to 2.4.0, you must first upgrade to the latest 2.4 release, then upgrade to 3.0.
The 3.0 upgrade will migrate all data, configuration, and scripts from 2.x EXCEPT any customized port fail reasons and port fail patterns. These will have to be re-entered by hand after completing the 3.0 upgrade.
CAVEAT: If you are doing a major upgrade (i.e. moving from one major version of Tinderbox to another), be sure to backup your database BEFORE running tc Upgrade. Major upgrades may drop and recreate the database, and this has the potential for data loss.
To run a Tinderbox build, and track the progress in the
database, you must first add the port you wish to build to the
database using the ${pb}/scripts/tc
application:
# cd ${pb}/scripts && ./tc addPort -b
{BUILD} -d {PORT DIRECTORY}
Where {BUILD} is the name of the Build for which this
port should be built, and {PORT DIRECTORY} is the directory
within the PortsTree where this port can be found. For example, to
build the GNOME 2 Desktop port for the Build
6.3-FreeBSD:
# cd ${pb}/scripts && ./tc addPort -b
6.3-FreeBSD -d x11/gnome2
Note: a port does not have to be added to the database for Tinderbox to build it. If you just want to do a quick ad hoc port build, forgo the previously mentioned step.
Note: in Tinderbox 2.x, addPort took a -r argument which recursively added ports to the datastore. In Tinderbox 3.0 and higher, this option is assumed. If you do not want to enable recursion, specify the -R argument to addPort.
To start a Tinderbox build, use the tc command:
# cd ${pb}/scripts && ./tc tinderbuild -b
{BUILD} {PORT DIRECTORY}
For example, to build the GNOME 2 Desktop for the Build
6.3-FreeBSD:
# cd ${pb}/scripts && ./tc tinderbuild -b
6.3-FreeBSD x11/gnome2
TIP: The example above will run the build in the foreground with all messages and errors echoing to the terminal. To capture all of this, it is recommended to redirect tinderbuild output to a log file. For example:
The tinderbuild function also accepts some additional command line arguments:
| -init | updates the Jail then updates the Build |
| -nullfs | uses nullfs instead of NFS to mount Jail and PortsTree file systems |
| -cleanpackages | removes all packages already built for the specified Build |
| -updateports | updates the Build's PortsTree (NOTE: dangerous if doing parallel runs with the same PortsTree) |
| -skipmake | skips the Makefile generation stage (NOTE: only use this option if a good Makefile already exists) |
| -noduds | skips the duds file generation stage (NOTE: packages which are forbidden or ignored will be built) |
| -noclean | does not clean up the Build hierarchy after the port build completes |
| -plistcheck | makes any plist verification problems (e.g. leftover files) fatal |
| -cleandistfiles | removes all files and directories in the distfile cache prior to starting the build |
| -fetch-original | ignores the distfile cache, and fetches all distfiles from their respective sources |
| -nolog | disables log analysis code |
| -trybroken | builds ports marked as BROKEN (this does NOT require -noduds) |
| -jobs | starts n number of parallel port builds (NOTE: the default is 1, and for best results should not exceed the number of physical CPUs in the Tinderbox host) |
| -onceonly | only performs one build pass (i.e. tinderbuild Phase 1) |
| -norebuild | do not force a rebuild of packages specified on the command line |
To obtain the version of Tinderbox, run the command:
# cd ${pb}/scripts && ./tc tbversion
This command simply cats the .version file in ${pb}. The .version file is created during distfile generation. If the contents of the Tinderbox distribution are copied to another location, make sure that the .version file is copied as well.
To update existing Jails:
# cd ${pb}/scripts && ./tc makeJail -j
6.3
The output of the Jail build will go to stdout. The output of the update command (e.g. cvsup) will go to ${pb}/jails/{JAIL}/update.log (where {JAIL} is the name of the Jail in question).
To update existing PortsTrees, use the tc (Tinderbox
Controller) application with the updatePortsTree command.
For example:
# cd ${pb}/scripts && ./tc updatePortsTree -p
FreeBSD
If you want to clone various aspects of an existing Build, use
the tc application with the copyBuild command.
For example:
# cd ${pb}/scripts && ./tc copyBuild -s
6.3-FreeBSD -d 6.4-FreeBSD
The source and destination Builds (i.e. 6.3-FreeBSD and 6.4-FreeBSD respectively in this example) must already exist. By default, copyBuild will copy the environment file, OPTIONS, and ports from the source Build to the destination Build. It can also optionally copy ccache data and packages. The following options are supported by copyBuild:
| -s | Source Build name |
| -d | Destination Build name |
| -c | (optional) copy ccache data |
| -E | (optional) do NOT copy environment files |
| -O | (optional) do NOT copy OPTIONS data |
| -P | (optional) do NOT copy ports |
| -p | (optional copy packages (including last build status, version, and size) |
Over time, Builds may become cluttered with old log files and
packages. To cleanup old, unreferenced, files, use the
${pb}/scripts/tc application with the
tbcleanup command:
# cd ${pb}/scripts && ./tc tbcleanup
The tbcleanup command supports a few arguments which will test it to perform additional cleanups:
| -d | Cleanup unreferenced distfiles from the distfile cache |
| -E | Do NOT remove old error logs (regular port build logs are still removed) |
| -p | Cleanup stale packages |
If you want to terminate a running tinderbuild, run the
command:
# cd ${pb}/scripts && ./tc tbkill -b
BUILD
This will gracefully terminate a running tinderbuild for Build
BUILD. If you want to force the tinderbuild to die, then
specify the kill signal:
# cd ${pb}/scripts && ./tc tbkill -b BUILD -s
9
Note: It may take a few minutes after executing a graceful termination of the tinderbuild before all processes exit. This is because the processes are cleaning up the build environment. If, after five minutes, the build is still running, then you should consider killing it with signal 9.
If you encounter problems with Tinderbox, it helps to see what is going on inside a Build. Tinderbox operational logs can be found under ${pb}/builds/{BUILD} (where {BUILD} is the Build name). This is where tinderbuild output should be redirected (see Using Tinderbox above). The make.0 and make.1 logs contain the initial build setup for each port. The reason there are two logs is that tinderbuild runs in two phases. The second build phase is identical to the first, and is run to catch any transient problems that may have occurred in the first phase.
The full build log of each port will be copied to ${pb}/logs/{BUILD} (where {BUILD} is the Build name). If the port failed to build successfully, the log will also be copied to ${pb}/errors/{BUILD} (where {BUILD} is the Build name).
Sometimes, the log alone is not sufficient for figuring out why a port failed to build. In such cases, one must also see the port's work directory. To have Tinderbox save this, create an empty file called .keep in the port's directory, and the work directory will be tarred, compressed, and copied to ${pb}/wrkdirs/{BUILD} (where {BUILD} is the Build name).
When it becomes too difficult to figure out the problem based on
the wrkdir, it may become necessary to access the Build itself. To
do this, touch a file called .sleepme in the port's
directory. The moment the .sleepme file is detected by the
build system, the port build will suspend just before executing
make build. You can access the Build with the
command:
# cd ${pb}/scripts && ./tc enterBuild -b
{BUILD} -d {PORT_DIRECTORY}
When you have finished, remove the .sleepme file, and the port build will continue.
By default, Jails and PortsTrees are updated using csup (/usr/bin/csup). Every time a new Jail or PortsTree is created, they will inherit the default update type. This update type is stored in ${pb}/scripts/etc/env/GLOBAL as defaultUpdateType. The allowed values are CVSUP, CSUP, USER, and NONE. for CVSUP and CSUP there is also a defaultUpdateHost. This can also be changed in ${pb}/scripts/etc/env/GLOBAL.
A Jail or PortsTree's update type can also be set when creating the Jail or PortsTree. To do this, specify the -u option to the appropriate create command. If the value of the update type is CVSUP, then /usr/local/bin/cvsup will be used to update the Jail or PortsTree. If the value is NONE, then no updates can be performed once the Jail or PortsTree is created. Instead, it is assumed these trees already exist in the appropriate format.
If the value is USER, then an update.sh script must be created under the Jail or PortsTree's root directory. This script will be called when an update is required. For example, if you have a Jail called 6.3, an executable update.sh script must be placed under ${pb}/jails/6.3. If you have a PortsTree called FreeBSD, and executable update.sh script must be placed under ${pb}/portstrees/FreeBSD.
If you want to mount /ports inside your
PortsTree or /src inside your Jail via
nullfs or NFS from another location, use -m switch to
create.
Example for NFS:
./tc createPortsTree -p FreeBSD -m
server:/directory
./tc createJail -j 6-FreeBSD -m
server:/directory
Example for nullfs:
./tc createPortsTree -p FreeBSD -m
/directory
./tc createJail -j 6-FreeBSD -m
/directory
Tinderbuild will ensure that these file systems are correctly mounted so you do not need to mount them by your own before calling tinderbuild.
If you want to change this settings later, use:
./tc setPortsMount -p <portstreename> -m
<mount path>
./tc setSrcMount -j <jailname> -m <mount
path>
Caching distfiles in a local repository can greatly improve
build times. As long as there is sufficient disk space, enabling a
local (or NFS) distfile cache is easy. Just use the following
command:
./tc configDistfile -c <mount point>
Where <mount point> is either an NFS specification, or fully qualified path (in the case of nullfs) in which to store downloaded distfiles. For example:
NFS:
./tc configDistfile -c
localhost:/space/distfiles
nullfs:
./tc configDistfile -c /d/distfiles
Another excellent way of accelerating builds is to use the compiler cache, ccache. To use ccache support, you must first create a tar file with ccache and various symlinks within a /opt directory. Your tarball contents should be:
opt opt/ccache opt/gcc -> ccache opt/cc -> ccache opt/g++ -> ccache opt/c++ -> ccache
This tarball must be called ccache.tar, and be placed in the Jail directory for each Jail that will use ccache (e.g. ${pb}/jails/6.3).
Once the tarball is created, run (-e for enabling,
-d for disabling, -c specifies directory,
-s maximal size):
./tc configCcache -e -c /ccache -s 2G
Then run your builds as you normally would. To debug ccache, add -l /ccache.log switch to the command.
Then, in the root of each build directory, there will be a ccache.log that will let you know if the cache is working.
It is possible to export environment variables on a global, per-Jail, per-PortsTree, and/or per-Build basis. This is done by creating a file named GLOBAL, jail.{JAIL}, portstree.{PORTSTREE}, or build.{BUILD} respectively, and placing it in ${pb}/scripts/etc/env.
For example, if you want a particular Build (e.g. 6.3-Perl56) to use Perl 5.6.2 instead of Perl 5.8.x, create a file named build.6.3-Perl56 in ${pb}/scripts/etc/env that contains the following:
PERL_VER=5.6.2 PERL_VERSION=5.6.2
Likewise, if you want to enable debugging for a particular PortsTree (e.g. FreeBSD-debug), create the following portstree.FreeBSD-debug in ${pb}/scripts/etc/env:
CFLAGS="-O -g -pipe" STRIP=
Every time ./tc addPort is called, it will determine if a port is already present in the datastore. If it is, the port's properties (i.e. MAINTAINER, COMMENT, etc.) are updated. If you just want rescan all of the ports currently in the datastore, and update their properties, use the command ./tc rescanPorts. It accepts many of the same options as addPort.
It is possible to manipulate port OPTIONS on a per-Build basis. To do this, create a directory under which all build OPTIONS subdirectories will go (e.g. ${pb}/options). For each Build that you wish to use to use OPTIONS, create a subdirectory named for that build (e.g. 6-STABLE-FreeBSD). This directory should look like /var/db/ports in that it contains subdirectories for each OPTIONS-supporting port.
For example, if you wanted to build net/wireshark with RTP support for the Build 6-STABLE-FreeBSD, you would have the following directory structure:
${pb}/options/6-STABLE-FreeBSD/wireshark/options
The contents of the wireshark/options file word be:
_OPTIONS_READ=wireshark-0.99.4 WITH_RTP=true WITH_SNMP=true WITH_ADNS=true WITH_PCRE=true
Once this structure is setup, then enable OPTIONS support in Tinderbox, and specify the path to the OPTIONS directory structure:
# cd ${pb}/scripts
# ./tc configOptions -e
# ./tc configOptions -o /options
The configOptions command takes the standard host arguments as well as -e (enable OPTIONS support), -d (disable OPTIONS support), and -o (set OPTIONS source directory tree).
If you would like to interactively choose OPTIONS for your ports, specify the -o option to ./tc addPort. This will perform a make config on all ports, displaying the ncurses OPTIONS dialog. All OPTIONS settings will be automatically saved to the correct location. Each time addPort or rescanPorts is run with the -o flag, all existing OPTIONS settings will be purged. If you want to retain existing OPTIONS, use the -O flag instead.
A hook is a callout which gives Tinderbox the ability to run custom code at pre-determined times. For example, you can establish a Hook to run just before a Port is built, after a Build is extracted, before a PortsTree is updated, etc. Pre-condition hooks can even cause an operation to terminate if so desired. To see a list of all available Hooks, run ./tc listHooks. To add a new command for an existing Hook, use the command ./tc updateHookCmd -h <hook> -c <command>. If you wish to disable a Hook, run ./tc updateHookCmd -h <hook> -c (note: do not specify a command after the -c option).
To use tinderd scheduling via the web you must first create a User:
# cd ${pb}/scripts && ./tc addUser -u {USER} -e {EMAIL} -p {PASSWORD} -w
(Where {USER} is a username, {EMAIL} is the user's email address, and {PASSWORD} is the user's password for Tinderbox web access.)
If you want to enable web access for a previously created user, you must update that user's account to give them a password as well as web access:
# cd ${pb}/scripts && ./tc updateUser -u {USER} -e {EMAIL} -p {PASSWORD} -w
(Where {USER} is a username, {EMAIL} is the user's email address, and {PASSWORD} is the user's password for Tinderbox web access.)
Then you have to define one web administrator who has full rights on all Builds and is the only account that can add other users:
# cd ${pb}/scripts && ./tc setWwwAdmin -u {USER}
(Where {USER} is the web administrator's username.)
After that, just browse to the Tinderbox web site with your web browser and login with {USER} and {PASSWORD}. You can now create and modify other users easily by using the "Add User" or "Modify User" links.
If you want to use Tinderbox to test many different ports one
after the other you probably want tinderd.
tinderd runs as a daemon and looks to see if something was
added to the ports to build queue. You can add different
ports for different builds for different hosts with different
priorities. tinderd will automatically pick up the port
with the highest priority for its host and starts building it. That
repeats until the queue is empty. After the queue empties
tinderd will sleep for a configurable amount of time
(default: 120 seconds) thereafter it starts searching for new queue
entries again. If you need tinderd to check the queue before the
sleep timer has expired, send the tinderd process a SIGHUP:
# kill -HUP {PID of tinderd}
Where {PID of tinderd} is the process ID of the tinderd script as seen in the output of ps(1)
You can start tinderd (it will stay in foreground by default). If you want to run tinderd automatically when the system boots, copy the ${pb}/etc/rc.d/tinderd script to /usr/local/etc/rc.d. Be sure to check out the various rc.conf variables documented in this script before using it. NOTE: the script used to be called tinderd.sh. If you have a /usr/local/etc/rc.d/tinderd.sh script, delete it, then copy the new tinderd script to the rc.d directory.
Now use:
# cd ${pb}/scripts && ./tc
addBuildPortsQueueEntry -b {BUILD} -d {PORT
DIRECTORY}
To add a port to the queue. tinderd will automatically pick it up, run a tinderbuild on it, and will delete the entry after tinderbuild completed. (Where {BUILD} is a Build name, and {PORT DIRECTORY} is a directory under {BUILDS}'s PortsTree (e.g. x11/gnome2).)
$MCom: tinderweb/README.html,v 1.23 2009/02/24 23:49:25 marcus Exp $