Wrapper script¶
Service providers are encouraged to write a wrapper script that acts as the glue between CLAM and the NLP Application(s). CLAM will execute the wrapper script, and the wrapper script will in turn invoke the actual NLP Application(s). Using a wrapper script offers more flexibility than letting CLAM directly invoke the NLP Application, and allows the NLP Application itself to be totally independent of CLAM.
When CLAM starts the wrapper script, it creates a clam.xml
file containing the selection of parameters and input
files provided by the user. It call the wrapper script with the arguments as specified in COMMAND
in
the serviceconf. There are some important things to take into account:
- All user-provided input has to be read from the specified input
directory. A full listing of this input will be provided in the
clam.xml
data file. If you choose not to use this, but use$PARAMETERS
instead, then you must take care that your application can identify the file formats by filename, extension or otherwise. - All user-viewable output must be put in the specified output directory. Output files must be generated in accordance with the profiles that describe this generation.
- The wrapper should periodically output a small status message to
$STATUSFILE
. While this is not mandatory, it offers valuable feedback to the user on the state of the system. - The wrapper script is always started with the current working directory set to the selected project directory.
- Wrapper scripts often invoke the actual application using some kind
of
system()
call. Take care never to pass unvalidated user-input to the shell! This makes you vulnerable for code injection attacks. The CLAM Data API offers the functionclam.common.data.shellsafe()
to help protect you.
The wrapper script can be written in any language. Python developers
will have the big advantage that they can directly tie into the CLAM
Data API, which handles things such as reading the clam.xml
data
file, makes all parameters and input files (with metadata) directly
accessible, and offers a function to protect your variables against code
injection when passing them to the shell. Using the Python for your
wrapper is therefore recommended.
If you used clamnewproject
to begin your new clam service, two
example wrapper scripts will have been created for you, one in Python
using the CLAM Data API, and one using bash shell script. Choose one.
These generated scripts are heavily commented to guide you in setting
your wrapper script up. This documentation will add some further
insights.
Data API¶
The key function of CLAM Data API is to parse the CLAM XML Data file
that the clam webservice uses to communicate with clients. This data is
parsed and all its components are made available in an instance of a
CLAMData
class.
Suppose your wrapper script is called with the following command definition:
COMMAND = "/path/to/wrapperscript.py $DATAFILE $STATUSFILE $OUTPUTDIRECTORY"
Your wrapper scripts then typically starts in the following fashion:
import sys
import clam.common.data
datafile = sys.argv[1]
statusfile = sys.argv[2]
outputdir = sys.argv[3]
clamdata = clam.common.data.getclamdata(datafile)
The first statements parse the command line arguments. The last
statement returns a CLAMData`
instance, which contains all data your
wrapper might need, representing the state of the project and all user
input. It is highly recommended to read the API reference for CLAMData
. A few of the attributes available are:
clamdata.system_id
clamdata.project
clamdata.user
clamdata.status
clamdata.parameters
clamdata.input
clamdata.program
Any global parameters set by the user are available from the
clamdata
instance, by using it like a Python dictionary, where the
keys correspond to the Parameter ID:
parameter = clamdata['parameter_id']
The CLAM API also has facilities to use a status file to relay progress feedback to the web-interface. Using it is as simple as importing the library and writing messages at strategic points during your program’s execution:
import clam.common.status
clam.common.status.write(statusfile, "We are running!")
Progress can also be expressed through an additional completion parameter, holding a value between \(0\) and \(1\). The web-application will show a progress bar if such information is provided:
clam.common.status.write(statusfile,
"We're half way there! Hang on!", 0.5)
If you have a specific input file you want to grab, you may obtain it
from your clamdata
instance with :meth:`CLAMData.inputtemplate:
inputfile = clamdata.inputfile('some-inputtemplate-id')
inputfilepath = str(inputfile)
The variable inputfile
in the above example is an instance of CLAMFile
, inputfilepath
in the above
example will contain the full path to the file that was uploaded by the user for the specified input template.
Once you have a file, you can easily obtain any associated metadata parameters in a dictionary-like fashion, for instance:
author = inputfile.metadata['author']
When you have multiple input files, you may want to iterate over all of them. The name of the inputtemplate can be obtained from the metadata:
for inputfile in clamdata.input:
inputfilepath = str(inputfile)
inputtemplate = inputfile.metadata.inputtemplate
The core of your wrapper script usually consists of a call to your
external program. In Python this can be done through os.system()
.
Consider the following fictitious example of a program that translates
an input text to the language specified by a global parameter.
os.system("translate -l " + clamdata['language'] + " " + \
str(clamdata.inputfile('sourcetext')) + \
+ " > " + outputdir + "/output.txt"))
However, at this point you need to be aware of possible malicious use,
and make sure nobody can perform a code injection attack. The key here
is to never pass unvalidated data obtained from user-input directly to
the shell. CLAM’s various parameters have their own validation options;
the only risk left to mitigate is that of string input. If the global
parameter language would be a free string input field, a user may
insert malicious code that gets passed to the shell. To prevent this,
use the shellsafe()
function from the CLAM Data API.
shellsafe = clam.common.data.shellsafe #just a shortcut
os.system("translate -l " + shellsafe(clamdata['language'],"'") + \
" " + \
shellsafe(str(clamdata.inputfile('sourcetext')),'"') + \
" > " + shellsafe(outputdir + "/output.txt") ))
Each variable should be wrapped in shellsafe
. The second argument to
shellsafe expresses whether to wrap the variable in quotes, and if so,
which quotes. Quotes are mandatory for values containing spaces or other
symbols otherwise forbidden. If no quotes are used, shellsafe does more
stringent checks to prevent code injection. A Python exception is raised
if the variable is not deemed safe, and the shell will not be invoked.
CLAM itself will detect and produce an error log.
Program¶
A program (programme) describes exactly what output files will be generated on the basis of what input files. It is the concretisation of the profiles. Profiles specify how input relates to output in a generic sense, using input and output templates. The program lists what exact output files will be generates, with filenames, on the basis of exactly which input files. The program is a read-only construct generated from the profiles and the input. It is present in the CLAM XML response, the clam XML data file, and accessible to your wrapper script.
Keep in mind that this method allows you to iterate over the output files prior to their actual creation. Because it contains exact information on output and input files. It is the most elegant method to set up your wrapper script, avoiding any duplication of file names and allowing your wrapper to be set up in a filename agnostic way.
In the following example. We obtain all output files and corresponding
output templates using the CLAMData.getoutputfiles()
. For each output file,
we can request the input files (and corresponding input templates) using
the CLAMData.getinputfiles()
.
Consider the following example that simply concatenates all input texts
(input template inputtext
) to a single output text (output template
outputtext
) using the unix cat
tool:
for outputfile, outputtemplate in clamdata.program.getoutputfiles():
outputfilepath = str(outputfile)
if outputtemplate == 'outputtext':
inputfiles_safe = ""
for inputfile, inputtemplate in clamdata.program.getinputfiles(outputfilename):
inputfilepath = str(inputfile)
if inputtemplate == 'inputtext': #check is a bit obsolete in this case
inputfiles_safe += " " + shellsafe(inputfilepath)
if inputfiles_safe:
os.system("cat " + inputfiles_safe + " > " + shellsafe(outputfilepath))
The outputfile
and inputfile
variables are again instances of CLAMFile
.
Their metadata parameters can be accesses through
outputfile.metadata[’parameter_id’]
and
inputfile.metadata[’parameter_id’]
.
Examples¶
Some example webservice configuration files and wrapper scripts are
included in clam/config
and clam/wrappers
respectively, often
similarly named.
One notable examples that are heavily commented:
textstats
– A simple text statistics/frequency list example for CLAM. It is a portable sample that has no external dependencies, the implementation is pure Python and done entirely in the wrapper script.
Some real-life CLAM webservice can be found in https://github.com/proycon/clamservices
Data API Reference¶
-
class
clam.common.data.
AbstractConverter
(id, **kwargs)¶ -
acceptforinput
= []¶
-
acceptforoutput
= []¶
-
convertforinput
(filepath, metadata)¶ Convert from target format into one of the source formats. Relevant if converters are used in InputTemplates. Metadata already is metadata for the to-be-generated file. ‘filepath’ is both the source and the target file, the source file will be erased and overwritten with the conversion result!
-
convertforoutput
(outputfile)¶ Convert from one of the source formats into target format. Relevant if converters are used in OutputTemplates. Sourcefile is a CLAMOutputFile instance.
-
label
= '(ERROR: label not overriden from AbstractConverter!)'¶
-
-
class
clam.common.data.
AbstractMetaField
(key, value=None)¶ This abstract class is the basis for derived classes representing metadata fields of particular types. A metadata field is in essence a (key, value) pair. These classes are used in output templates (described by the XML tag
meta
). They are not used byCLAMMetaData
-
static
fromxml
(node)¶ Static method returning an MetaField instance (any subclass of AbstractMetaField) from the given XML description. Node can be a string or an etree._Element.
-
resolve
(data, parameters, parentfile, relevantinputfiles)¶
-
xml
(operator='set', indent='')¶ Serialize the metadata field to XML
-
static
-
class
clam.common.data.
Action
(*args, **kwargs)¶ This action paradigm allows you to specify actions, each action allows you to tie a URL to a script or Python function, and may take a number of parameters you explicitly specify. Each action is strictly independent of other actions, and completely separate of the projects, and by extension also of any files within projects and any profiles. Unlike projects, which may run over a long time period and are suited for batch processing, actions are intended for real-time communication. Typically they should return an answer in at most a couple of seconds.i
Positional Arguments:
- a Parameter instance or a Viewer instance.
Keyword arguments:
id
- The ID of the action (mandatory)name
- A human readable name, used in the interfacedescription
- A human readable description of the action, used in the interfacecommand
- The command to run, this is analogous to the COMMAND in the service configuration file and may contain parameters (most notably $PARAMETERS)function
- The python function to call (use either this orcommand
)parameters
- List of parameter instances. By defailt, they will be passed in the order defined to the command or function.parameterstyle
- Set topositional
(default) orkeywords
. Changes the way arguments are passed to the function.viewers
- List of viewer instances.mimetype
- The mimetype of the output (when no viewers are used).method
- The HTTP Method to allow, set to string GET, POST or the None value to allow all methods.returncodes404
- A list of command exit codes that will be mapped to HTTP 404 Not Found (defaults to: [4])returncodes403
- A list of command exit codes that will be mapped to HTTP 403 Permission Denied (defaults to: [3])returncodes200
- A list of command exit codes that will be mapped to HTTP 200 Ok (defaults to: [0])allowanonymous
- Boolean indicating whether this action can be used without any authentication.
-
static
fromxml
(node)¶ Static method returning an Action instance from the given XML description. Node can be a string or an etree._Element.
-
xml
(indent='')¶
-
exception
clam.common.data.
AuthRequired
(msg='')¶ Raised on HTTP 401 - Authentication Required error. Service requires authentication, pass user credentials in CLAMClient constructor.
-
exception
clam.common.data.
AuthenticationRequired
¶ This Exception is raised when authentication is required but has not been provided
-
exception
clam.common.data.
BadRequest
¶ Raised on HTTP 400 - Bad Request erors
-
class
clam.common.data.
CLAMData
(xml, client=None, localroot=False, projectpath=None, loadmetadata=True)¶ Instances of this class hold all the CLAM Data that is automatically extracted from CLAM XML responses. Its member variables are:
baseurl
- The base URL to the service (string)projecturl
- The full URL to the selected project, if any (string)status
- Can be:clam.common.status.READY
(0),``clam.common.status.RUNNING`` (1), orclam.common.status.DONE
(2)statusmessage
- The latest status message (string)completion
- An integer between 0 and 100 indicating- the percentage towards completion.
parameters
- List of parameters (but use the methods instead)profiles
- List of profiles ([ Profile ]
)program
- A Program instance (or None). Describes the expected outputfiles given the uploaded inputfiles. This is the concretisation of the matching profiles.input
- List of input files ([ CLAMInputFile ]
); useinputfiles()
instead for easier accessoutput
- List of output files ([ CLAMOutputFile ]
)projects
- List of project IDs ([ string ]
)corpora
- List of pre-installed corporaerrors
- Boolean indicating whether there are errors in parameter specificationerrormsg
- String containing an error messageoauth_access_token
- OAuth2 access token (empty if not used, string)
Note that depending on the current status of the project, not all may be available.
-
baseurl
= None¶ String containing the base URL of the webserivice
-
commandlineargs
()¶ Obtain a string of all parameters, using the paramater flags they were defined with, in order to pass to an external command. This is shell-safe by definition.
-
corpora
= None¶ List of pre-installed corpora
-
errormsg
= None¶ String containing an error message if an error occured
-
errors
= None¶ Boolean indicating whether there are errors in parameter specification
-
get
(parameter_id, default=None)¶
-
input
= None¶ List of input files ([ CLAMInputFile ])
-
inputfile
(inputtemplate=None)¶ Return the inputfile for the specified inputtemplate, if
inputtemplate=None
, inputfile is returned regardless of inputtemplate. This function may only return 1 and returns an error when multiple input files can be returned, useinputfiles()
instead.
-
inputfiles
(inputtemplate=None)¶ Generator yielding all inputfiles for the specified inputtemplate, if
inputtemplate=None
, inputfiles are returned regardless of inputtemplate.
-
inputtemplate
(template_id)¶ Return the inputtemplate with the specified ID. This is used to resolve a inputtemplate ID to an InputTemplate object instance
-
inputtemplates
()¶ Return all input templates as a list (of InputTemplate instances)
-
loadmetadata
= None¶ True)
Type: Automatically load metadata for input and output files? (default
-
matchingprofiles
()¶ Generator yielding all matching profiles
-
output
= None¶ List of output files ([ CLAMOutputFile ])
-
outputtemplate
(template_id)¶ Get an output template by ID
-
parameter
(parameter_id)¶ Return the specified global parameter (the entire object, not just the value)
-
parametererror
()¶ Return the first parameter error, or False if there is none
-
parameters
= None¶ This contains a list of (parametergroup, [parameters]) tuples.
-
parseresponse
(xml, localroot=False)¶ Parses CLAM XML, there’s usually no need to call this directly
-
passparameters
()¶ Return all parameters as {id: value} dictionary
-
profiles
= None¶ List of profiles ([ Profile ])
-
program
= None¶ Program instance. Describes the expected outputfiles given the uploaded inputfiles. This is the concretisation of the matching profiles.
-
projects
= None¶ List of projects ([ string ])
-
projecturl
= None¶ String containing the full URL to the project, if a project was indeed selected
-
status
= None¶ The current status of the service, returns clam.common.status.READY (1), clam.common.status.RUNNING (2), or clam.common.status.DONE (3)
-
statusmessage
= None¶ The current status of the service in a human readable message
-
class
clam.common.data.
CLAMFile
(projectpath, filename, loadmetadata=True, client=None, requiremetadata=False)¶ -
attachviewers
(profiles)¶ Attach viewers and converters to file, automatically scan all profiles for outputtemplate or inputtemplate
-
basedir
= ''¶
-
copy
(target, timeout=500)¶ Copy or download this file to a new local file
-
delete
()¶ Delete this file
-
exists
()¶
-
loadmetadata
()¶ Load metadata for this file. This is usually called automatically upon instantiation, except if explicitly disabled. Works both locally as well as for clients connecting to a CLAM service.
-
metafilename
()¶ Returns the filename for the metadata file (not full path). Only used for local files.
-
read
()¶ Loads all lines in memory
-
readlines
()¶ Loads all lines in memory
-
store
(fileid=None, keep=False)¶ Put a file in temporary public storage, returns the ID if the file is local, returns a dictionary with keys ‘id’, ‘filename’ and ‘url’ if the file is remote.
-
validate
()¶ Validate this file. Returns a boolean.
-
-
class
clam.common.data.
CLAMInputFile
(projectpath, filename, loadmetadata=True, client=None, requiremetadata=False)¶ -
basedir
= 'input'¶
-
-
class
clam.common.data.
CLAMMetaData
(file, **kwargs)¶ A simple hash structure to hold arbitrary metadata. This is the basis for format classes.
-
allowcustomattributes
= True¶
-
attributes
= None¶
-
classmethod
formatxml
(indent='')¶ Render an XML representation of the format class
-
static
fromxml
(node, file=None)¶ Read metadata from XML. Static method returning an CLAMMetaData instance (or rather; the appropriate subclass of CLAMMetaData) from the given XML description. Node can be a string or an etree._Element.
-
httpheaders
()¶ HTTP headers to output for this format. Yields (key,value) tuples. Should be overridden in sub-classes!
-
items
()¶ Returns all items as (key, value) tuples
-
mimetype
= 'text/plain'¶
-
save
(filename)¶ Save metadata to XML file
-
schema
= ''¶
-
validate
()¶ Validate the metadata. Possibly extracts additional metadata from the actual file into the metadata file. This method calls a format’s custom validator() function which you can override per format, additionally it also validates any constraints that are set. The validatation method implements some caching so your validator() function is never called more than once.
-
validateconstraints
()¶ Validates the constraints (if any). Called by validate(), no need to invoke directly
-
validator
()¶ This method can be overriden on derived classes and has no implementation here, should return True or False. Additionaly, if there is metadata IN the actual file, this method should extract it and assign it to this object. Will be automatically called from constructor. Note that the file (CLAMFile) is accessible through self.file, which is guaranteerd to exist when this method is called.
-
xml
(indent='')¶ Render an XML representation of the metadata
-
-
class
clam.common.data.
CLAMOutputFile
(projectpath, filename, loadmetadata=True, client=None, requiremetadata=False)¶ -
basedir
= 'output'¶
-
-
class
clam.common.data.
CLAMProvenanceData
(serviceid, servicename, serviceurl, outputtemplate_id, outputtemplate_label, inputfiles, parameters=None, timestamp=None)¶ Holds provenance data
-
static
fromxml
(node)¶ Return a CLAMProvenanceData instance from the given XML description. Node can be a string or an lxml.etree._Element.
-
xml
(indent='')¶ Serialise provenance data to XML. This is included in CLAM Metadata files
-
static
-
exception
clam.common.data.
ConfigurationError
¶ This Exception is raised when authentication is required but has not been provided
-
class
clam.common.data.
Constraint
(constrainttype, **kwargs)¶ -
static
fromxml
(node)¶ Static method returns a Constraint instance from the given XML description. Node can be a string or an etree._Element.
-
test
(metadata)¶
-
xml
(indent='')¶ Produce Constraint XML
-
static
-
class
clam.common.data.
CopyMetaField
(key, value=None)¶ In CopyMetaField, the value is in the form of templateid.keyid, denoting where to copy from. If not keyid but only a templateid is specified, the keyid of the metafield itself will be assumed.
-
resolve
(data, parameters, parentfile, relevantinputfiles)¶
-
xml
(indent='')¶ Serialize the metadata field to XML
-
-
class
clam.common.data.
ForbidMeta
(**kwargs)¶
-
exception
clam.common.data.
FormatError
(value)¶ This Exception is raised when the CLAM response is not in the valid CLAM XML format
-
class
clam.common.data.
Forwarder
(id, name, url, description='', type='zip', tmpstore=True, encodeurl=True)¶
-
exception
clam.common.data.
HTTPError
¶ This Exception is raised when certain data (such a metadata), can’t be retrieved over HTTP
-
class
clam.common.data.
InputSource
(**kwargs)¶ -
check
()¶ Checks if this inputsource is usable in INPUTSOURCES
-
isdir
()¶
-
isfile
()¶
-
xml
(indent='')¶
-
-
class
clam.common.data.
InputTemplate
(template_id, formatclass, label, *args, **kwargs)¶ This class represents an input template. A slot with a certain format and function to which input files can be uploaded
-
static
fromxml
(node)¶ Static method returning an InputTemplate instance from the given XML description. Node can be a string or an etree._Element.
-
generate
(file, validatedata=None, inputdata=None, user=None)¶ Convert the template into instantiated metadata, validating the data in the process and returning errors otherwise. inputdata is a dictionary-compatible structure, such as the relevant postdata. Return (success, metadata, parameters), error messages can be extracted from parameters[].error. Validatedata is a (errors,parameters) tuple that can be passed if you did validation in a prior stage, if not specified, it will be done automatically.
-
json
()¶ Produce a JSON representation for the web interface
-
match
(metadata, user=None)¶ Does the specified metadata match this template? returns (success,metadata,parameters)
-
matchingfiles
(projectpath)¶ Checks if the input conditions are satisfied, i.e the required input files are present. We use the symbolic links .*.INPUTTEMPLATE.id.seqnr to determine this. Returns a list of matching results (seqnr, filename, inputtemplate).
-
validate
(postdata, user=None)¶ Validate posted data against the inputtemplate
-
xml
(indent='')¶ Produce Template XML
-
static
-
exception
clam.common.data.
NoConnection
¶ Raised when a connection can’t be established
-
exception
clam.common.data.
NotFound
(msg='')¶ Raised on HTTP 404 - Not Found Errors
-
class
clam.common.data.
OutputTemplate
(template_id, formatclass, label, *args, **kwargs)¶ -
findparent
(inputtemplates)¶ Find the most suitable parent, that is: the first matching unique/multi inputtemplate
-
static
fromxml
(node)¶ Static method return an OutputTemplate instance from the given XML description. Node can be a string or an etree._Element.
-
generate
(profile, parameters, projectpath, inputfiles, provenancedata=None)¶ Yields (inputtemplate, inputfilename, inputmetadata, outputfilename, metadata) tuples
-
generatemetadata
(parameters, parentfile, relevantinputfiles, provenancedata=None)¶ Generate metadata, given a filename, parameters and a dictionary of inputdata (necessary in case we copy from it)
-
getparent
(profile)¶ Resolve a parent ID
-
xml
(indent='')¶ Produce Template XML
-
-
class
clam.common.data.
ParameterCondition
(**kwargs)¶ -
allpossibilities
()¶ Returns all possible outputtemplates that may occur (recusrively applied)
-
evaluate
(parameters)¶ Returns False if there’s no match, or whatever the ParameterCondition evaluates to (recursively applied!)
-
static
fromxml
(node)¶ Static method returning a ParameterCondition instance from the given XML description. Node can be a string or an etree._Element.
-
match
(parameters)¶
-
xml
(indent='')¶
-
-
exception
clam.common.data.
ParameterError
(msg='')¶ Raised on Parameter Errors, i.e. when a parameter does not validate, is missing, or is otherwise set incorrectly.
-
class
clam.common.data.
ParameterMetaField
(key, value=None)¶ -
resolve
(data, parameters, parentfile, relevantinputfiles)¶
-
xml
(indent='')¶ Serialize the metadata field to XML
-
-
exception
clam.common.data.
PermissionDenied
(msg='')¶ Raised on HTTP 403 - Permission Denied Errors (but only if no CLAM XML response is provided)
-
class
clam.common.data.
Profile
(*args)¶ -
static
fromxml
(node)¶ Return a profile instance from the given XML description. Node can be a string or an etree._Element.
-
generate
(projectpath, parameters, serviceid, servicename, serviceurl)¶ Generate output metadata on the basis of input files and parameters. Projectpath must be absolute. Returns a Program instance.
-
match
(projectpath, parameters)¶ Check if the profile matches all inputdata and produces output given the set parameters. Returns a boolean
-
matchingfiles
(projectpath)¶ Return a list of all inputfiles matching the profile (filenames)
-
out
(indent='')¶
-
outputtemplates
()¶ Returns all outputtemplates, resolving ParameterConditions to all possibilities
-
xml
(indent='')¶ Produce XML output for the profile
-
static
-
class
clam.common.data.
Program
(projectpath, matchedprofiles=None)¶ A Program is the concretisation of Profile. It describes the exact output files that will be created on the basis of what input files. This is in essence a dictionary structured as follows:
{outputfilename: (outputtemplate, inputfiles)}
in whichinputfiles
is a dictionary{inputfilename: inputtemplate}
-
add
(outputfilename, outputtemplate, inputfilename=None, inputtemplate=None)¶ Add a new path to the program
-
getinputfile
(outputfile, loadmetadata=True, client=None, requiremetadata=False)¶ Grabs one input file for the specified output filename (raises a KeyError exception if there is no such output, StopIteration if there are no input files for it). Shortcut for getinputfiles()
-
getinputfiles
(outputfile, loadmetadata=True, client=None, requiremetadata=False)¶ Iterates over all input files for the specified outputfile (you may pass a CLAMOutputFile instance or a filename string). Yields (CLAMInputFile,str:inputtemplate_id) tuples. The last three arguments are passed to its constructor.
-
getoutputfile
(loadmetadata=True, client=None, requiremetadata=False)¶ Grabs one output file (raises a StopIteration exception if there is none). Shortcut for getoutputfiles()
-
getoutputfiles
(loadmetadata=True, client=None, requiremetadata=False)¶ Iterates over all output files and their output template. Yields (CLAMOutputFile, str:outputtemplate_id) tuples. The last three arguments are passed to its constructor.
-
inputpairs
(outputfilename)¶ Iterates over all (inputfilename, inputtemplate) pairs for a specific output filename
-
outputpairs
()¶ Iterates over all (outputfilename, outputtemplate) pairs
-
update
([E, ]**F) → None. Update D from dict/iterable E and F.¶ If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
-
-
class
clam.common.data.
RequireMeta
(**kwargs)¶
-
exception
clam.common.data.
ServerError
(msg='')¶ Raised on HTTP 500 - Internal Server Error. Indicates that something went wrong on the server side.
-
class
clam.common.data.
SetMetaField
(key, value=None)¶ -
resolve
(data, parameters, parentfile, relevantinputfiles)¶
-
xml
(indent='')¶ Serialize the metadata field to XML
-
-
exception
clam.common.data.
TimeOut
¶ Raised when a connection times out
-
class
clam.common.data.
UnsetMetaField
(key, value=None)¶ -
resolve
(data, parameters, parentfile, relevantinputfiles)¶
-
xml
(indent='')¶ Serialize the metadata field to XML
-
-
exception
clam.common.data.
UploadError
(msg='')¶ Raised when something fails during upload
-
clam.common.data.
buildarchive
(project, path, fmt)¶ Build a download archive, returns the full file path
-
clam.common.data.
escape
(s, quote)¶
-
clam.common.data.
escapeshelloperators
(s)¶
-
clam.common.data.
getclamdata
(filename, custom_formats=None, custom_viewers=None)¶ This function reads the CLAM Data from an XML file. Use this to read the clam.xml file from your system wrapper. It returns a CLAMData instance.
If you make use of CUSTOM_FORMATS, you need to pass the CUSTOM_FORMATS list as 2nd argument.
-
clam.common.data.
getformats
(profiles)¶
-
clam.common.data.
loadconfig
(callername, required=True)¶ This function loads an external configuration file. It is called directly by the service configuration script and complements the configuration specified there. The function in turn automatically searches for an appropriate configuration file (in several paths). Host and system specific configuration files are prioritised over more generic ones.
callername
- A string representing the name of settings module. This is typically set to __name__
Example:
loadconfig(__name__)
-
clam.common.data.
loadconfigfile
(configfile, settingsmodule)¶ This function loads an external configuration file. It is usually not invoked directly but through
loadconfig()
which handles searching for the right configuration file in the right paths, with fallbacks.
-
clam.common.data.
parsexmlstring
(node)¶
-
clam.common.data.
processhttpcode
(code, allowcodes=None)¶ Return the success code or raises the appropriate exception when the code repesents an HTTP error code
-
clam.common.data.
processparameter
(postdata, parameter, user=None)¶
-
clam.common.data.
processparameters
(postdata, parameters, user=None)¶
-
clam.common.data.
profiler
(profiles, projectpath, parameters, serviceid, servicename, serviceurl, printdebug=None)¶ Given input files and parameters, produce metadata for outputfiles. Returns a list of matched profiles (empty if none match), and a program.
-
clam.common.data.
resolveconfigvariables
(value, settingsmodule)¶ Resolves standard environment variables, encoded in curly braces
-
clam.common.data.
resolveinputfilename
(filename, parameters, inputtemplate, nextseq=0, project=None)¶
-
clam.common.data.
resolveoutputfilename
(filename, globalparameters, localparameters, outputtemplate, nextseq, project, inputfilename)¶
-
clam.common.data.
sanitizeparameters
(parameters)¶ Construct a dictionary of parameters, for internal use only
-
clam.common.data.
shellsafe
(s, quote='', doescape=True)¶ Returns the value string, wrapped in the specified quotes (if not empty), but checks and raises an Exception if the string is at risk of causing code injection
-
clam.common.data.
unescapeshelloperators
(s)¶