4.14 Configuration

There is no real difference between a command line option and a configuration parameter. In a package, which has one or many commands, each command has several options. Some of the options are command specific others are inherited from the package. A third group of options is inherited from the siliconBrain frame (like the --help option). The options, which are inherited from the package are the configuration parameters.

The normal command specific options are specified in a command specification program (executable specification), which outputs the option description in xml or other formats. The package configuration parameters are simple the options of a command, with the same name as the package. Whether the command really exists or not does not matter. These options are generated into each command implementation as inherited options. So each configuration parameter can be overwritten by using an option.

So the first source of configuration are the command line parameters. In the option handling of each command, the second location of looking for options is the environment. For each option and thus for each configuration parameter too, there is a corresponding environment variable packageName_optionName.

Third place are executable configurations: files with the name packageName.configuration residing in certain directories (current, home, $PATH, etc). These programs (implemented in any language) output the option contents in XML format. This output is read by the standard option handling generated by siliconBrain. The list of directories can be configured by setting the environment variable siliconBrainConfigurationPath to a colon separated list of directories, with the entry "" being the $PATH. The default is .:~:/etc:$PATH for (current, home, $PATH, etc).

At the beginning I thought that the very last location for a configuration look up is the packages path. But I decided against this because it should be possible to just delete a package directory and then reinstall it. All configuration information should be kept. So there should be nothing changeable inside the package directory. always think of the package directory residing on a read only CD-ROM. The second reason is, that the normal case presumably is that with a new version of a package the old configuration should be taken. If this is not the wish, siliconBrainConfigurationPath can be used to handle different configurations.

An option will have the value, which has been found first. So command line arguments overwrite environment variables, overwrite the configuration in the current directory overwrites ..... Flag options are normally specified without a value (--myFlag). To be able to set an option to false even if it is configured to true in a subsequent configuration, it is possible to set a flag option explicitly to false: --myFlag=false or --no-myFlag. And for reasons of symmetry --myFlag=true is also allowed.

The --complete option avoids reading all stages every time. Whenever in the places to look the --complete option is set, the lookup in the subsequent places is suppressed. So for example, if yo specify --complete on the command line no further option or configuration lookup is performed.

In each package there is a generated program packageName.configurationReader, which does the standard option handling (reading command line arguments, reading environment variables, reading executable configuration output and each time stopping, if --complete is found). This program then does nothing than outputting the aggregated information in various format: xml, bash, environment, perl and others.

This can be included in other languages like (for bash)

     eval $(packageName.configurationReader $* -- --bash)

The configuration can be pumped into environment by:

     eval $(packageName.configurationReader -- --bashEnvironemnt)

where export statements are generated. The configurationReader always set the --complete option. So after executing the last example, all commands of that package, will stop after reading the environment. The above eval statement can for example be place in a .bashrc.

Before starting I thought, that the output of the executable configurations should be of the form:

     # comment
     key = value
     anotherKey = another value

rather than xml. The reason for that was, that it should be easy for other programs in arbitrary languages to read and process the configurations. Other languages include bash, emacs lisp and perl. Using xml would require to have an xml parser in any language. But for the reason to have a consistent and common option, environment and configuration handling I have invented the configurationReader. This handles options, environment and configurations and then outputs the result in a host language friendly way. This means that the configurations can output xml without having the need to process this xml in all different languages. On the other side to produce xml is far more easy than to parse it. The following example of a siliconBrain.configuration file is the so called trivial configuration implementation:

     #!/bin/bash
     
     cat <<EOF
     <siliconBrain>
        <publishTargetHost> siliconbrain.com </publishTargetHost>
        <publishTargetArchive> ftp/anon/pub </publishTargetArchive>
        <publishTargetWeb> www/htdocs </publishTargetWeb>
     </siliconBrain>
     EOF

With this concept we fulfill the following requirentments of a configuration environment:

• Configurations are held in files, which in turn are recognized by their file extension: The executable configurations have the extension .configuration.
• Configuration contents is pure ascii and thus is easy to create in any language: the configuration content is xml, which is easy to build in any language. We do not use xml attributes which makes the creation even easier.
• Configurations do reside in cvs and can be created with an arbitrary editor (as opposed to a special program): For each configuration, which is a program, there is a program source, which can as normally stored in cvs. Further on configurations can be written in any language (C, bash, perl, ...) so an administrator can choose the language, which fits best in her environment.
• Configurations do cascade in site specific, user specific, local: The configurationReader and the standard option handling for C-programs, do read first the command line, then the environment variables, then a configuration found in the current working directory, then one in the users $HOME directory then in the $PATH, then in /etc.
• The same package can be run with different configurations on the same computer with the same userid: By using different configurations in different current working directories, this is possible. If the ./myPackage.configuration does not set the --complete flag, then a higher level configuration can set the common parameters.
• After a version change, the old configuration files should be supported still: the option handling xml parser just ignores options, which are unknown. This is a little support for a package author to fulfill this request.
• Sometimes a configuration configures the build phase of a package. In that case the configuration should be part of the cvs tree: Because configurations are programs, there source can reside in the normal cvs tree. If they are C-programs, the make process must just make sure, that they are compiled into the package root directory.
• It should be possible and easy to use the configurations from all languages: The configurationReader can output the found and aggregated parameters in xml, bash, perl or other languages.
• It should be possible to gain performances by suppressing a complex configuration hierarchy parsing: The --complete option can suppress any further lookup in higher levels. So for example setting the environment variable myPackage_complete will suppress the search, execution and parsing of executable configurations. With the configurationReaders --bashEnvironment format, it is easy to pump a configuration into ones environment.
• It should be possible to read all or part of a configuration from a database: because a configuration is a program, information can be read from anywhere, to be converted to xml.
• The configuration language should provide means to avoid redundancies: Because there is no extra configuration language (observing the “never invent a new language” rule), but instead any language can be used (the option handling does not interpret this language source, but rather the output which should be xml). So all means of programming are available. You can write your configurations in c++ and use oo techniques.
• Configuration syntax should be easy for non programmers to be build: the trivial implementation (see example above) is a bash program, which just cat out the xml.
• Configurations should be processable with sed and grep: To be honest I am not sure. The output of a configuration is xml. Is this processable with grep?
• Configurations should be writable in any language: This is true as long as the language allows writing xml to stdout.
• Configurations should be allowed to be implemented in mixed languages: (For example a company admin writes a complex perl configuration file and a completes with a short trivial bash based configuration snippet.) This can be accomplished by chaining the configurations in the sense that one calls a second, modifies its contends then output the final xml. Or by a simple placing the complex perl into /etc/packageName.configuration and the trivial bash into ./packageName.configuration. This is the intended solution.

There is an environment variable siliconBrainConfigurationPath with a colon separated list of directories, where to search for configurations.

The option specification, which describes the configuration parameters of a package is the same (not just equal) to the option specification of a command with the name of the package. So if the package packageName has a command packageName.command then the options of that command are the same as the configuration parameters of the package. This is intended for packages with a definite main command, which for example starts the server.

The configurationReader uses siliconBrainLib's xmlPrinter to output its information. This means, that it can do the output in various formats (for example xml). And it does it colorized in case of stdout being a tty. Also the configurationReader uses the short descriptions of the options to create comments in the generated output, so each option is described a little. Only those options (and with it configuration parameters (member that the configuration parameters are available as inherited command line options)) which have been specified somewhere are outputted. The rest is present just via its commented short description:

     ~/sandbox/siliconBrain> siliconBrain.configurationReader -- --xml
     
     <!-- siliconBrain main configuration.
        general datahandling application framework -->
     <siliconBrain>
     
        <!-- publishTargetHost (value) : IP address or domain name where package should be put to -->
        <publishTargetHost> siliconbrain.com </publishTargetHost>
     
        <!-- publishTargetArchive (value) : directory on the publishTargetHost, where the tar.gz should be put to. -->
        <publishTargetArchive> ftp/anon/pub </publishTargetArchive>
     
        <!-- publishTargetWeb (value) : directory on the publishTargetHost, where the HTMLilized sourcetree should be put to. -->
        <publishTargetWeb> www/htdocs </publishTargetWeb>
     
        <!-- help (flag ) : Print a short help message, listing the options. -->
     
        <!-- verbose (flag ) : Let this command talk to you a lot. -->
     
        <!-- version (flag ) : Display version information. -->
     
        <!-- output (value) : File to which output is written. -->
     
        <!-- complete (flag ) : Indicate completeness of specified options. No further lookup in configuration chain. -->
        <complete/>
     </siliconBrain>
     ~/sandbox/siliconBrain>

It is possible to chain configurations in a way: a user can write a configuration in her home directory, which calls a standard configuration and then uses sed or awk to manipulate some entries.

Be aware the the suffix .configuration does not indicate the format of the file, so that you can chose a certain editor or editing mode. Rather this suffix indicates, that it is an executable file, which, when called, outputs xml. In case that the configuration is implemented in c, the configuration file itself is not readable at all. In this case you need the source.

It is recommended to use siliconBrainLib's SiliconBrainPrinter aka xmlPrinter to write the sources of your configuration, if they tend to be a little complex. This is a c library of functions (also used by the configurationReader) to output xml. If you do, you are programming in c, you have all the possibilities of c and the resulting program outputs xml in color in a standardized inteted manner, and in other syntaxes (like perl or bash).

I have decided for an executable configuration to be able to use a real full blown programming language for writing the configuration. So in creating configurations you have all means of reusability, avoiding copied code, factor out common information, using loops, defining variables, commenting. On the other side you can read databases or other sources of information to support kind of dynamic configuration.

If you use xml as a language for sources, or to put it into other words, if you have xml files in cvs normally you corrupt a lot of programming rules. Mostly the “never copy code” rule. But xml is a perfect protocol language, a language used for programs to communicate with each other. By deciding for executable configurations, I have the advantages of full programming languages, with the advantages of xml as protocol.