dorie.parscraper¶
Contents
What is dorie.parscraper?¶
dorie.parscraper is a python package with the purpose of automatic creation of DORiE parameter files in various formats. The developers only need to maintain one single XML file that contains all the relevant information for each parameter.
It provides the following components:
Readers
for parsing both the XML file containing the meta-information for each parameter and the DORiE C++ source files,- a
matching function
that complements the information from the XML file with the location where each parameter is actually used in the code, Writers
for various output formats (e.g. to dynamically create the .ini files used in runs of the main program, or to create a user cheat sheet with further explanations),- a
wrapper script
controlling the scraping process.
Another purpose is to identify mismatches between XML meta-information and source code and notify the developers. Therefore, a warning is issued if either a parameter is only found in the XML file, or only found in the DORiE source code.
How to use dorie.parscraper¶
The dorie.parscraper scripts are called via CMake while configuring DORiE. This is to make sure that the generated ini files are accessible as soon as the system tests are created.
The dorie.parscraper scripts¶
-
scrape_folder.py
This script basically just parses the command line arguments, implements special handling of warnings and exceptions to keep the output as clean as possible, and calls
dorie.parscraper.wrapper.scrape_folder
.Command line options:
-
--xml
¶
Path to the XML file holding parameter meta-information
-
--source
¶
The C++ source folder to scrape
-
--out
¶
Output file(s)
-
--css
¶
Path to a CSS file to include into HTML output (optional)
-
--debug
¶
Show warnings (optional, default off)
-
The dorie.parscraper core modules¶
Parameter class¶
-
class
dorie.parscraper.parameter.
Parameter
(category, key)[source]¶ Implements a class for storing DORiE parameters.
Parameters: - category (str) – The category of the parameter, indicated in the DUNE infiles by square brackets.
- key (str) – The name of the parameter, at the left hand side of the equal sign in DUNE inifiles.
-
_code_found
= None¶ Whether the parameter was found in the code at all
-
_sources
= None¶ List of all the positions in the code where the parameter was found, encoded in a tuple: (filename, lineno, C++ type)
-
category
= None¶ Parameter category
-
comment
= None¶ Extra comment showing up in the generated ini files
-
definition
= None¶ Long explanation of the parameter, shown in the documentation
Do not issue an error if parameter is not found
-
key
= None¶ Parameter name
-
suggestion
= None¶ Suggested (default) value
-
values
= None¶ Possible values
Matching function¶
-
dorie.parscraper.match_parameters.
match
(xml_parameters, source_parameters, base_path)[source]¶ Expects the output of
dorie.parscraper.readers.xml
anddorie.parscraper.readers.source()
as input, and tries to match the parameters with each other. Records, where the parameters specified in the XML file are found in the code (file name, line number, and which type they are cast to), and supplements the Parameter instances with this information.Returns an OrderedDict with categories as keys and lists of Parameter instances as values. Parameters that are only found in the source files, but are not specified in XML, are not included in the output. Parameters that are only specified in the XML file but not in the code are returned, but a warning is issued.
Parameters: - xml_parameters – Output of
dorie.parscraper.readers.xml.parse()
- source_parameters – Output of
dorie.parscraper.readers.source.parse()
- xml_parameters – Output of
Readers¶
XML reader¶
-
dorie.parscraper.readers.xml.
parse
(filename)[source]¶ Reads and parses a DORiE parameter XML file, using the
xml.etree.ElementTree
package.The XML file is expected to have the following structure:
Available attributes are all attributes of the
dorie.parscraper.parameter.Parameter
class, except the ones starting with an underscore.Parameters: filename (str) – Path to the input XML file Return type: OrderedDict Returns: all parsed category names as keys, and a list of all parameters in the corresponding category as values (instances of Parameter
)Raises: XMLWarning – if unknown, redundant, or malformatted XML elements are encountered
C++ source reader¶
-
dorie.parscraper.readers.source.
parse
(filename)[source]¶ Reads a source file and looks for patterns like
inifile.get<type>("category.test.parameter",default)
When found, the relevant information gets scraped and assembled in a dict.
Note
You can prevent a parameter to be matched by adding
// %HIDDEN%
to the end of the corresponding line.Parameters: filename (str) – Full path to the input file Returns: list of dicts (one dict for each matching pattern in the source file), each containing all found information
Writers¶
.ini file writer¶
-
dorie.parscraper.writers.ini.
write
(parameters, out, *args, **kwargs)[source]¶ Writes the contents of the
parameters
input dict to a .ini file. The output is structured as[key] # value.comment value.key = value.suggestion
to match the format of the DUNE input files. A missing comment attribute leads to omitting of the comment line; a missing suggestion to the value
UNDEFINED
.Parameters: - parameters (dict) – dict with categories as keys and iterables of Parameter instances as values.
- out (str) – Path to the output file. Must be writable.
.rst file writer¶
-
dorie.parscraper.writers.rst.
write
(parameters, out, path_base, *args, **kwargs)[source]¶ Writes the contents of the
parameters
input dict to a .rst file. The output is structured ascategory
Parameter Definition Possible values Default Queried at p.key p.definition p.values p.suggestion all sources of p Parameters: - parameters (dict) – dict with categories as keys and iterables of
dorie.parscraper.parameter.Parameter
instances as values. - out (str) – Path to the output file. Must be writable.
- path_base (str) – Base path to the source files. Needed for creation of working links to the files.
- parameters (dict) – dict with categories as keys and iterables of
.html file writer¶
-
dorie.parscraper.writers.html.
write
(parameters, out, path_base, *args, **kwargs)[source]¶ Writes the contents of the
parameters
input dict to a .html file. The output is structured ascategory
Parameter Definition Possible values Default Queried at p.key p.definition p.values p.suggestion all sources of p Parameters: - parameters (dict) – dict with categories as keys and iterables of
dorie.parscraper.parameter.Parameter
instances as values. - out (str) – Path to the output file. Must be writable.
- path_base (str) – Base path to the source files. Needed for creation of working links to the files.
- css (str) – Path to a CSS file containing the styling of the HTML output. Is included into the HTML output (inline). Optional.
- parameters (dict) – dict with categories as keys and iterables of