dorie.parscraper

Contents

• dorie.parscraper
  • What is dorie.parscraper?
    • How to use dorie.parscraper
  • The dorie.parscraper scripts
  • The dorie.parscraper core modules
    • Parameter class
    • Warnings
    • Matching function
    • Wrapper
    • Readers
      • XML reader
      • C++ source reader
    • Writers
      • .ini file writer
      • .rst file writer
      • .html file writer

What is dorie.parscraper?

dorie.parscraper is a python package with the purpose of automatic creation of DORiE parameter files in various formats. The developers only need to maintain one single XML file that contains all the relevant information for each parameter.

It provides the following components:

• Readers for parsing both the XML file containing the meta-information for each parameter and the DORiE C++ source files,
• a matching function that complements the information from the XML file with the location where each parameter is actually used in the code,
• Writers for various output formats (e.g. to dynamically create the .ini files used in runs of the main program, or to create a user cheat sheet with further explanations),
• a wrapper script controlling the scraping process.

Another purpose is to identify mismatches between XML meta-information and source code and notify the developers. Therefore, a warning is issued if either a parameter is only found in the XML file, or only found in the DORiE source code.

How to use dorie.parscraper

The dorie.parscraper scripts are called via CMake while configuring DORiE. This is to make sure that the generated ini files are accessible as soon as the system tests are created.

The dorie.parscraper scripts

scrape_folder.py

This script basically just parses the command line arguments, implements special handling of warnings and exceptions to keep the output as clean as possible, and calls dorie.parscraper.wrapper.scrape_folder.

Command line options:

--xml

Path to the XML file holding parameter meta-information

--source

The C++ source folder to scrape

--out

Output file(s)

--css

Path to a CSS file to include into HTML output (optional)

--debug

Show warnings (optional, default off)

The dorie.parscraper core modules

Parameter class

class dorie.parscraper.parameter.Parameter(category, key)

Implements a class for storing DORiE parameters.

Parameters:

• category (str) – The category of the parameter, indicated in the DUNE infiles by square brackets.
• key (str) – The name of the parameter, at the left hand side of the equal sign in DUNE inifiles.

_code_found = None

Whether the parameter was found in the code at all

_sources = None

List of all the positions in the code where the parameter was found, encoded in a tuple: (filename, lineno, C++ type)

category = None

Parameter category

comment = None

Extra comment showing up in the generated ini files

definition = None

Long explanation of the parameter, shown in the documentation

hidden = None

Do not issue an error if parameter is not found

key = None

Parameter name

suggestion = None

Suggested (default) value

values = None

Possible values

Warnings

exception dorie.parscraper.warnings.OutputWarning

exception dorie.parscraper.warnings.ScraperWarning

exception dorie.parscraper.warnings.XMLWarning

Matching function

dorie.parscraper.match_parameters.match(xml_parameters, source_parameters, base_path)

Expects the output of dorie.parscraper.readers.xml and dorie.parscraper.readers.source() as input, and tries to match the parameters with each other. Records, where the parameters specified in the XML file are found in the code (file name, line number, and which type they are cast to), and supplements the Parameter instances with this information.

Returns an OrderedDict with categories as keys and lists of Parameter instances as values. Parameters that are only found in the source files, but are not specified in XML, are not included in the output. Parameters that are only specified in the XML file but not in the code are returned, but a warning is issued.

Parameters:

• xml_parameters – Output of dorie.parscraper.readers.xml.parse()
• source_parameters – Output of dorie.parscraper.readers.source.parse()

Wrapper

dorie.parscraper.wrapper.scrape_folder.scrape(xml_file, source_folder, out, css=None, debug=False)

Readers

XML reader

dorie.parscraper.readers.xml.parse(filename)

Reads and parses a DORiE parameter XML file, using the xml.etree.ElementTree package.

The XML file is expected to have the following structure:

<dorie> ‣ <category name=”…”> ‣ <parameter name=”…”> ‣ (attributes)

Available attributes are all attributes of the dorie.parscraper.parameter.Parameter class, except the ones starting with an underscore.

Parameters:filename (str) – Path to the input XML file Return type:OrderedDict Returns:all parsed category names as keys, and a list of all parameters in the corresponding category as values (instances of Parameter) Raises:XMLWarning – if unknown, redundant, or malformatted XML elements are encountered

C++ source reader

dorie.parscraper.readers.source.parse(filename)

Reads a source file and looks for patterns like

inifile.get<type>("category.test.parameter",default)

When found, the relevant information gets scraped and assembled in a dict.

Note

You can prevent a parameter to be matched by adding // %HIDDEN% to the end of the corresponding line.

Parameters:filename (str) – Full path to the input file Returns:list of dicts (one dict for each matching pattern in the source file), each containing all found information

Writers

.ini file writer

dorie.parscraper.writers.ini.write(parameters, out, *args, **kwargs)

Writes the contents of the parameters input dict to a .ini file. The output is structured as

[key]
# value.comment
value.key = value.suggestion

to match the format of the DUNE input files. A missing comment attribute leads to omitting of the comment line; a missing suggestion to the value UNDEFINED.

Parameters:

• parameters (dict) – dict with categories as keys and iterables of Parameter instances as values.
• out (str) – Path to the output file. Must be writable.

.rst file writer

dorie.parscraper.writers.rst.write(parameters, out, path_base, *args, **kwargs)

Writes the contents of the parameters input dict to a .rst file. The output is structured as

category

Parameter	Definition	Possible values	Default	Queried at
p.key	p.definition	p.values	p.suggestion	all sources of p

Parameters:

• parameters (dict) – dict with categories as keys and iterables of dorie.parscraper.parameter.Parameter instances as values.
• out (str) – Path to the output file. Must be writable.
• path_base (str) – Base path to the source files. Needed for creation of working links to the files.

.html file writer

dorie.parscraper.writers.html.write(parameters, out, path_base, *args, **kwargs)

Writes the contents of the parameters input dict to a .html file. The output is structured as

category

Parameter	Definition	Possible values	Default	Queried at
p.key	p.definition	p.values	p.suggestion	all sources of p

Parameters:

• parameters (dict) – dict with categories as keys and iterables of dorie.parscraper.parameter.Parameter instances as values.
• out (str) – Path to the output file. Must be writable.
• path_base (str) – Base path to the source files. Needed for creation of working links to the files.
• css (str) – Path to a CSS file containing the styling of the HTML output. Is included into the HTML output (inline). Optional.