packages icon



 UP(3)                                                                 UP(3)




 UP
      up - user mode pipeline library

 SYNOPSIS
      #include <up.h>

      UP * upAllocate();
      upAddDevice(UP *, ...);
      upSetupDone(UP *);
      UP_BOOL upReadWrite(UP, int timeout);
      upClose(UP *);

 DESCRIPTION
      The up library provides a user mode pipeline capability.

      A user mode pipeline is similar in concept to a normal UNIX pipeline
      in which various filter programs are used to transform a stream of
      data.  But in this case, the pipeline is run in a single process by
      passing the data stream through various filtering modules to transform
      data.  This avoids context switches, provides better error handling,
      avoids most uses of temporary files, and allows for the pipeline to be
      dynamically changed.  A user mode pipeline can handle multiple files
      during the processing such as unpacking a tar archive, compressing
      each file separately, and then repacking the resulting files into
      another tar archive.

      In the following discussion, the various filtering modules are called
      devices even though the term is somewhat misleading since no hardware
      is involved.  However, data is read from them and is sent to them, and
      so the term has some validity.

      A user mode pipeline consists of a list of two or more devices
      arranged from "left" to "right" in a straight line.  The leftmost
      device is the "reading" device, and must generate data for the
      pipeline.  The reading device can generate data from external sources
      such as files or standard input, or from internal data such as a
      string or data generated by the calling program.  The rightmost device
      is the "writing" device and must consume data from the pipeline.  The
      writing device can send data to an external source such as files or
      standard output, or to internal data such as a string to be read by
      the calling program.  All devices in the middle of the pipeline (if
      any) are "filters" which pass the data through them, transforming the
      data as required.

      The interface to a device is uniform so that new types of devices can
      be easily added to the library.  The interface provides for storing
      input and output into buffers which are shared between adjacent
      devices to minimize copying.  The interface also provides
      synchronization between the devices for the correct transfer of file
      names, data, and end of files.




                                    - 1 -           Formatted:  July 1, 2022






 UP(3)                                                                 UP(3)




 MODES OF OPERATION
      Data in the pipeline can be processed using one of three different
      modes of operation.  Which of these modes is used depends on the needs
      of the calling program and its sophistication.

      Firstly, data can be "written" to the left end of the pipeline by the
      calling program which will "push" the data through the pipeline to
      some output device which consumes it (such as writing it to a file).
      In this mode of operation, the calling program blocks until the
      pipeline processes enough data to make room for storing the program's
      new data.

      Secondly, data can be "read" from the right end of the pipeline, which
      will "pull" the data through the pipeline from some input device which
      generates it (such as reading from a file).  In this mode of
      operation, the calling program blocks until the pipeline processes
      enough data to provide some to return to the calling program.

      Finally, data can be transferred "asychronously" through the pipeline
      by the calling program so that the program can generate input data if
      desired, consume output data if desired, and never block.  This mode
      of operation makes use of the "poll" system call if necessary, and
      allows the calling program to run the pipeline in an event-driven
      manner.

 CREATING A USER MODE PIPELINE
      A user mode pipeline is based on the UP structure which is allocated
      by calling the upAllocate routine.  The returned UP structure is then
      used for all further user mode pipeline calls.  Once allocated,
      devices can be attached sequentially to the pipeline.  The routines to
      add devices have names of the form upAddDevice, where Device is
      replaced by the desired device.  The argument list for each of these
      devices varies depending on the needs of the device.

      Devices are added from "left" to "right".  That is, the data in the
      pipeline will flow from the first device added to the last device
      added.  The first device must be one of the devices which can read
      data from some external source.  The last device must be one of the
      devices which can write data to some external source.  Intermediate
      devices must be one of the devices which are filters, which read data
      from another device and write data to another device.

      Some devices require that the size of their data be known before the
      data is seen.  (For example, this is true for the PipeTar device,
      because the headers for the files contain the file sizes.) If one of
      these devices is used, and the file size is not known from the
      previously added devices, then the special PipeFindSize device is
      automatically added before the device which requires it.  The
      PipeFindSize device will buffer a file's data in memory or in a
      temporary file, and will then supply the actual file size with the
      file name to the next device.



                                    - 2 -           Formatted:  July 1, 2022






 UP(3)                                                                 UP(3)




      The automatically added PipeFindSize device uses default parameters
      for the amount of memory it can consume, and whether or not file sizes
      should sometimes be believed.  You can explicity add your own
      PipeFindSize device in the pipeline in order to override these default
      parameters.

      When all of the devices have been added to the pipeline, the
      upSetupDone routine must be called.  After this call, no further
      devices can be added to the pipeline, and the pipeline is ready to do
      I/O.

 RUNNING THE PIPELINE
      The usual way to run the pipeline is asychrounously by calling
      upReadWrite repeatedly until the pipeline is finished.  Each call
      performs one cycle of I/O for the devices in the pipeline.

      The routine accepts a timeout value in milliseconds similar to select,
      so that the call does not need to block if that is not wanted.  A
      value of -1 means an indefinite wait, whereas a value of 0 means no
      wait at all.

      If there is an error, then the upReadWrite routine returns UP_FALSE.
      If there is no error, then the upIsFinished routine needs to be called
      to see whether or not the pipeline is finished.  If it is not
      finished, then you can call upReadWrite again to perform more I/O.

      When the pipeline is finished, you call upClose to close all of the
      devices and free their resources.  It returns UP_TRUE if there were no
      problems.  After the pipeline is closed, it can be re-used by new
      devices just like when it was first allocated.

      If the pipeline is no longer needed, calling upFree will close the
      devices (if necessary) and free the UP structure.

 LIST OF FUNCTIONS
      Function
           Description

      upAddReadEndpoint
           Add a device to allow reading from the pipeline

      upAddWriteEndpoint
           Add a device to allow writing to the pipeline

      upAddReadNull
           Add a device which reads no data

      upAddWriteNull
           Add a device which writes no data





                                    - 3 -           Formatted:  July 1, 2022






 UP(3)                                                                 UP(3)




      upAddReadFd
           Add a device which reads data from a file descriptor

      upAddWriteFd
           Add a device which writes data to a file descriptor

      upAddReadDirectory
           Add a devices which reads files from a directory

      upAddWriteDirectory
           Add a device which writes files into a directory

      upAddReadString
           Add a device which reads data from a string

      upAddWriteString
           Add a device which writes data to a string

      upAddReadFile
           Add a device which reads data from a file

      upAddReadFileList
           Add a device which reads data from a list of files

      upAddWriteFile
           Add a device which writes data to a file or a sequence of files

      upAddProcessString
           Add a device which sends data through an external process

      upAddProcessArgs
           Add a device which sends data through an external process

      upAddProgress
           Add a device which provides progress information

      upAddLogFile
           Add a device which logs information

      upAddCat
           Add a device which concatenates its input files into one output
           file

      upAddGnuZip
           Add a device which compresses its input files using GZIP

      upAddGnuUnzip
           Add a device which uncompresses its intput files using GUNZIP

      upAddCompress
           Add a device which compresses its input files using COMPRESS



                                    - 4 -           Formatted:  July 1, 2022






 UP(3)                                                                 UP(3)




      upAddUnCompress
           Add a device which uncompresses its input files using UNCOMPRESS

      upAddUUDecode
           Add a device which uuencodes its input files into one output file

      upAddUUEncode
           Add a device which uudecodes its input file into output files

      upAddArchive
           Add a device which archives its input files using AR

      upAddUnArchive
           Add a device which unarchives its input file using AR

      upAddTar
           Add a device which tars its input files using TAR

      upAddUnTar
           Add a device which untars its input files using TAR

      upAddList
           Add a device which produces a listing of the files

      upAddGrep
           Add a device which searches for lines which match a regular
           expression

      upAddFindSize
           Add a device which calculates the true size of files

      upAddFilter
           Add a device which selects which files are transferred through it

      upAddRemoveData
           Add a device which transmits file names and status but removes
           the file data

      upAddFindFileData
           Add a device which reads the data for the file names sent through
           it

      upAddMakeFileList
           Add a device which reads a file name list to produce empty output
           files

      upGetVersion
           Return the version of the up library

      upAllocate
           Allocate a new pipeline



                                    - 5 -           Formatted:  July 1, 2022






 UP(3)                                                                 UP(3)




      upClose
           Close a pipeline

      upFree
           Close and free a pipeline

      upSetupDone
           Mark a pipeline ready for use

      upReadWrite
           Run one cycle of I/O for a pipeline

      upPoll
           Wait for the pipeline to have some I/O

      upPollList
           Return a list of polling entries needed by the pipeline

      upRead
           Read data from the right end of a pipeline

      upWrite
           Write data to the left end of a pipeline

      upFlush
           Flush data through a pipeline

      upCanRead
           Return whether data can be read from the right end of a pipeline

      upCanWrite
           Return whether data can be written to the left end of a pipeline

      upIsFinished
           Return whether the pipeline has finished

      upSetSystemPath
           Set the PATH list to be used when securely executing external
           programs

      upGetSystemPath
           Return the PATH list used when securely executing external
           programs

      upSetTemporaryDirectoryPath
           Set the directory to be used for creating temporary files

      upGetTemporaryDirectoryPath
           Return the directory to be used for creating temporary files





                                    - 6 -           Formatted:  July 1, 2022






 UP(3)                                                                 UP(3)




      upClearError
           Clear the error information from the pipeline

      upGetErrorCode
           Get the error code for a pipeline error

      upGetDeviceCount
           Return the number of devices which are in the pipeline

      upGetErrorString
           Return a string describing the last error for the pipeline

      upPrintError
           Print the latest error on the pipeline to stderr

      upGetWarningCount
           Return the count of warnings which have occurred for the pipeline

      upGetMaximumWarnings
           Return the maximum number of warnings which can be printed to
           stderr

      upSetMaximumWarnings
           Set the maximum number of warnings which can be printed to stderr

      upSetDebugFlags
           Set debugging flags for printing of internal pipeline status

 SEE ALSO
      up(1) - A utility which creates and runs a user mode pipeline
      according to the supplied arguments.

 LICENSE
      The up library is licensed under the GNU Lesser Public License.  This
      means that you may freely use this library in your own program, using
      whatever license you wish for your program, and with no restrictions
      on the distribution or use of your program.  However, any
      modifications to the up library itself are covered by the license.

      You should have received a copy of the GNU Lesser General Public
      License along with this library; if not, write to the Free Software
      Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
      USA

 AUTHOR
      David I. Bell
      dbell@canb.auug.org.au







                                    - 7 -           Formatted:  July 1, 2022