UP(3) UP(3)
UP
up - user mode pipeline library
SYNOPSIS
#include <up.h>
UP * upAllocate();
upAddDevice(UP *, ...);
upSetupDone(UP *);
UP_BOOL upReadWrite(UP, int timeout);
upClose(UP *);
DESCRIPTION
The up library provides a user mode pipeline capability.
A user mode pipeline is similar in concept to a normal UNIX pipeline
in which various filter programs are used to transform a stream of
data. But in this case, the pipeline is run in a single process by
passing the data stream through various filtering modules to transform
data. This avoids context switches, provides better error handling,
avoids most uses of temporary files, and allows for the pipeline to be
dynamically changed. A user mode pipeline can handle multiple files
during the processing such as unpacking a tar archive, compressing
each file separately, and then repacking the resulting files into
another tar archive.
In the following discussion, the various filtering modules are called
devices even though the term is somewhat misleading since no hardware
is involved. However, data is read from them and is sent to them, and
so the term has some validity.
A user mode pipeline consists of a list of two or more devices
arranged from "left" to "right" in a straight line. The leftmost
device is the "reading" device, and must generate data for the
pipeline. The reading device can generate data from external sources
such as files or standard input, or from internal data such as a
string or data generated by the calling program. The rightmost device
is the "writing" device and must consume data from the pipeline. The
writing device can send data to an external source such as files or
standard output, or to internal data such as a string to be read by
the calling program. All devices in the middle of the pipeline (if
any) are "filters" which pass the data through them, transforming the
data as required.
The interface to a device is uniform so that new types of devices can
be easily added to the library. The interface provides for storing
input and output into buffers which are shared between adjacent
devices to minimize copying. The interface also provides
synchronization between the devices for the correct transfer of file
names, data, and end of files.
- 1 - Formatted: March 13, 2025
UP(3) UP(3)
MODES OF OPERATION
Data in the pipeline can be processed using one of three different
modes of operation. Which of these modes is used depends on the needs
of the calling program and its sophistication.
Firstly, data can be "written" to the left end of the pipeline by the
calling program which will "push" the data through the pipeline to
some output device which consumes it (such as writing it to a file).
In this mode of operation, the calling program blocks until the
pipeline processes enough data to make room for storing the program's
new data.
Secondly, data can be "read" from the right end of the pipeline, which
will "pull" the data through the pipeline from some input device which
generates it (such as reading from a file). In this mode of
operation, the calling program blocks until the pipeline processes
enough data to provide some to return to the calling program.
Finally, data can be transferred "asychronously" through the pipeline
by the calling program so that the program can generate input data if
desired, consume output data if desired, and never block. This mode
of operation makes use of the "poll" system call if necessary, and
allows the calling program to run the pipeline in an event-driven
manner.
CREATING A USER MODE PIPELINE
A user mode pipeline is based on the UP structure which is allocated
by calling the upAllocate routine. The returned UP structure is then
used for all further user mode pipeline calls. Once allocated,
devices can be attached sequentially to the pipeline. The routines to
add devices have names of the form upAddDevice, where Device is
replaced by the desired device. The argument list for each of these
devices varies depending on the needs of the device.
Devices are added from "left" to "right". That is, the data in the
pipeline will flow from the first device added to the last device
added. The first device must be one of the devices which can read
data from some external source. The last device must be one of the
devices which can write data to some external source. Intermediate
devices must be one of the devices which are filters, which read data
from another device and write data to another device.
Some devices require that the size of their data be known before the
data is seen. (For example, this is true for the PipeTar device,
because the headers for the files contain the file sizes.) If one of
these devices is used, and the file size is not known from the
previously added devices, then the special PipeFindSize device is
automatically added before the device which requires it. The
PipeFindSize device will buffer a file's data in memory or in a
temporary file, and will then supply the actual file size with the
file name to the next device.
- 2 - Formatted: March 13, 2025
UP(3) UP(3)
The automatically added PipeFindSize device uses default parameters
for the amount of memory it can consume, and whether or not file sizes
should sometimes be believed. You can explicity add your own
PipeFindSize device in the pipeline in order to override these default
parameters.
When all of the devices have been added to the pipeline, the
upSetupDone routine must be called. After this call, no further
devices can be added to the pipeline, and the pipeline is ready to do
I/O.
RUNNING THE PIPELINE
The usual way to run the pipeline is asychrounously by calling
upReadWrite repeatedly until the pipeline is finished. Each call
performs one cycle of I/O for the devices in the pipeline.
The routine accepts a timeout value in milliseconds similar to select,
so that the call does not need to block if that is not wanted. A
value of -1 means an indefinite wait, whereas a value of 0 means no
wait at all.
If there is an error, then the upReadWrite routine returns UP_FALSE.
If there is no error, then the upIsFinished routine needs to be called
to see whether or not the pipeline is finished. If it is not
finished, then you can call upReadWrite again to perform more I/O.
When the pipeline is finished, you call upClose to close all of the
devices and free their resources. It returns UP_TRUE if there were no
problems. After the pipeline is closed, it can be re-used by new
devices just like when it was first allocated.
If the pipeline is no longer needed, calling upFree will close the
devices (if necessary) and free the UP structure.
LIST OF FUNCTIONS
Function
Description
upAddReadEndpoint
Add a device to allow reading from the pipeline
upAddWriteEndpoint
Add a device to allow writing to the pipeline
upAddReadNull
Add a device which reads no data
upAddWriteNull
Add a device which writes no data
- 3 - Formatted: March 13, 2025
UP(3) UP(3)
upAddReadFd
Add a device which reads data from a file descriptor
upAddWriteFd
Add a device which writes data to a file descriptor
upAddReadDirectory
Add a devices which reads files from a directory
upAddWriteDirectory
Add a device which writes files into a directory
upAddReadString
Add a device which reads data from a string
upAddWriteString
Add a device which writes data to a string
upAddReadFile
Add a device which reads data from a file
upAddReadFileList
Add a device which reads data from a list of files
upAddWriteFile
Add a device which writes data to a file or a sequence of files
upAddProcessString
Add a device which sends data through an external process
upAddProcessArgs
Add a device which sends data through an external process
upAddProgress
Add a device which provides progress information
upAddLogFile
Add a device which logs information
upAddCat
Add a device which concatenates its input files into one output
file
upAddGnuZip
Add a device which compresses its input files using GZIP
upAddGnuUnzip
Add a device which uncompresses its intput files using GUNZIP
upAddCompress
Add a device which compresses its input files using COMPRESS
- 4 - Formatted: March 13, 2025
UP(3) UP(3)
upAddUnCompress
Add a device which uncompresses its input files using UNCOMPRESS
upAddUUDecode
Add a device which uuencodes its input files into one output file
upAddUUEncode
Add a device which uudecodes its input file into output files
upAddArchive
Add a device which archives its input files using AR
upAddUnArchive
Add a device which unarchives its input file using AR
upAddTar
Add a device which tars its input files using TAR
upAddUnTar
Add a device which untars its input files using TAR
upAddList
Add a device which produces a listing of the files
upAddGrep
Add a device which searches for lines which match a regular
expression
upAddFindSize
Add a device which calculates the true size of files
upAddFilter
Add a device which selects which files are transferred through it
upAddRemoveData
Add a device which transmits file names and status but removes
the file data
upAddFindFileData
Add a device which reads the data for the file names sent through
it
upAddMakeFileList
Add a device which reads a file name list to produce empty output
files
upGetVersion
Return the version of the up library
upAllocate
Allocate a new pipeline
- 5 - Formatted: March 13, 2025
UP(3) UP(3)
upClose
Close a pipeline
upFree
Close and free a pipeline
upSetupDone
Mark a pipeline ready for use
upReadWrite
Run one cycle of I/O for a pipeline
upPoll
Wait for the pipeline to have some I/O
upPollList
Return a list of polling entries needed by the pipeline
upRead
Read data from the right end of a pipeline
upWrite
Write data to the left end of a pipeline
upFlush
Flush data through a pipeline
upCanRead
Return whether data can be read from the right end of a pipeline
upCanWrite
Return whether data can be written to the left end of a pipeline
upIsFinished
Return whether the pipeline has finished
upSetSystemPath
Set the PATH list to be used when securely executing external
programs
upGetSystemPath
Return the PATH list used when securely executing external
programs
upSetTemporaryDirectoryPath
Set the directory to be used for creating temporary files
upGetTemporaryDirectoryPath
Return the directory to be used for creating temporary files
- 6 - Formatted: March 13, 2025
UP(3) UP(3)
upClearError
Clear the error information from the pipeline
upGetErrorCode
Get the error code for a pipeline error
upGetDeviceCount
Return the number of devices which are in the pipeline
upGetErrorString
Return a string describing the last error for the pipeline
upPrintError
Print the latest error on the pipeline to stderr
upGetWarningCount
Return the count of warnings which have occurred for the pipeline
upGetMaximumWarnings
Return the maximum number of warnings which can be printed to
stderr
upSetMaximumWarnings
Set the maximum number of warnings which can be printed to stderr
upSetDebugFlags
Set debugging flags for printing of internal pipeline status
SEE ALSO
up(1) - A utility which creates and runs a user mode pipeline
according to the supplied arguments.
LICENSE
The up library is licensed under the GNU Lesser Public License. This
means that you may freely use this library in your own program, using
whatever license you wish for your program, and with no restrictions
on the distribution or use of your program. However, any
modifications to the up library itself are covered by the license.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
USA
AUTHOR
David I. Bell
dbell@canb.auug.org.au
- 7 - Formatted: March 13, 2025
