wetsuite.helpers.koop_parse

Given document-style XML data such as that of CVDR XML documents, tries to capture most of the interesting structure in easier-to-digest python data form, and lessen the nested nature without quite throwing it away.

Whenever we hit an <al>, we emit a dict that details all the interesting elements between body and this <al>

This is intended as the lower-level half of potential cleanup, grouping/splitting logic, and such. ...though it is specific to a handful of XML schemas, so not universal. For wider applicability you may want to look to helpers.split TODO

    {
        'path':      '/cvdr/body/regeling/regeling-tekst/hoofdstuk[1]/artikel[1]/al',
        'parts': [   {'what': 'hoofdstuk',  'hoofdstuklabel': 'Hoofdstuk', 'hoofdstuknr': '1', 'hoofdstuktitel': 'Algemene bepalingen',},
                     {'what': 'artikel',    'artikellabel': 'Artikel',     'artikelnr': '1:1', 'artikeltitel': 'Begripsomschrijvingen',}  ],
        'merged':    {'hoofdstuklabel': 'Hoofdstuk',
                      'hoofdstuknr': '1',
                      'hoofdstuktitel': 'Algemene bepalingen'
                      'artikellabel': 'Artikel',
                      'artikelnr': '1:1',
                      'artikeltitel': 'Begripsomschrijvingen',
                      },
        'text-flat': 'In deze verordening wordt verstaan dan wel mede verstaan onder:'
    }

Fetch interesting metadata from manifest WTI XML. TODO: finish

Merge the result of the above, into a flatter structure

Takes individual SRU result record as an etree subtrees, picks out BWB-specific metadata (merging the separate metadata sections). Returns a dict

Give title text, estimate whether the content has much to say

Given the document (as an etree object), this is a quick and dirty 'give me mainly the plaintext in it', skipping any introductions and such.

Not structured data, intended to assist generic "how do these setences parse" code

TODO:

• review this, it makes various assumptions about document structure
• review the handling of certain elements, like lijst, table, definitielijst
• see if there are contained elements to ignore, like maybe <redactie type="vervanging"> ?
• generalize to have a parameter ignore_tags=['li.nr', 'meta-data', 'kop', 'tussenkop', 'plaatje', 'adres', 'specificatielijst', 'artikel.toelichting', 'citaat', 'wetcitaat']

Fetch the most interesting metadata from parsed toestand XML. TODO: finish

Fetch interesting metadata from parsed WTI XML. TODO: finish, and actually do it properly -- e.g. look to the schema as to what may be omitted, may repeat, etc.

Extracts metadata from a CVDR SRU search result's individual record, or CVDR content xml's root.

Because various elements can repeat - and various things frequently do (e.g. 'source'), each value is a list. using flatten=True MAY CLOBBER DATA and is only recommended for quick and dirty debug prints, not for real use.

        { 'creator': [{'attr': {'scheme': 'overheid:Gemeente'}, 'text': 'Zuidplas'}], ... }

        { 'creator': 'Zuidplas (overheid:Gemeente)', ... }

When indexing, you may care to transform varied forms (e.g. "112779_1" and "CVDR112779_1") into just one normalized form.

Note that this does not work on workids - we raise an exception for them.

Picks the parameters from a juriconnect (jci) style identifier string.

Mostly a helper function, used e.g. by cvdr_sourcerefs. Duplicates code in meta.py - TODO: centralize that

Would turn "BWB://1.0:c:BWBR0008903&artikel=12&g=2011-11-08" into a dict where 'artikel' maps to ['12'], etc.

Given a CVDR style identifier string (sometimes called JCDR), gives a more normalized version, e.g. useful for indexing.

For example:

    cvdr_parse_identifier('101404_1')     ==  ('101404', '101404_1')
    cvdr_parse_identifier('CVDR101405_1') ==  ('101405', '101405_1')
    cvdr_parse_identifier('CVDR101406')   ==  ('101406',  None     )
    cvdr_parse_identifier('1.0:101407_1') ==  ('101407', '101407_1')

Given the CVDR XML content document as an etree object, looks for the <source> tags (meta/owmsmantel/source), which are references to laws and other regulations (VERIFY)

This function

• extracts (only) the source tags that specify ...and ignores

in part to normalize what is in there a bit. Be aware this is more creative than a helper function probably should be.

Returns a list of :

  (type, origref, specref, parts, source_text)

where

• type: currently one of 'BWB' or 'CVDR'
• origref: URL-like reference
• specref: just the identifier
• parts: dict of parts parsed from URL, or None
• source_text: text (name and often reference to a part); seems to be more convention-based than standardized

For example (mostly to point out there is _plenty_ of variation in most parts) :

 ('BWB',
  '1.0:c:BWBR0015703&artikel=6&g=2014-11-13',
  'BWBR0015703',
  {'artikel': ['6'], 'g': ['2014-11-13']},
  'Participatiewet, art. 6')

or :

 ('BWB',
  'http://wetten.overheid.nl/BWBR0015703/geldigheidsdatum_15-05-2011#Hoofdstuk1_12',
  'BWBR0015703',
  {},
  'Wet werk en bijstand, art. 8, lid 1')

or :

 ('CVDR',
  'CVDR://103202_1',
  '103202_1',
  None,
  'Inspraakverordening Spijkenisse, art. 2')

or :

 ('CVDR',
  '1.1:CVDR229520-1',
  '229520',
  None,
  'Verordening voorzieningen maatschappelijke participatie 2012-A, artikel 4')

Given the XML content document as etree object, this is a quick and dirty 'give me mainly the plaintext in it', skipping any introductions and such.

Returns a single string. This is currently a best-effort formatting, where you should e.g. find that paragraphs are split with double newlines.

This is currently mostly copy-pasted from the bwb code TODO: unify, after I figure out all the varying structure

TODO: write functions that support "give me flat text for each article separately"

takes a CVDR id (with or without _version, i.e. expression id or work id), searches KOOP's CVDR repo, Returns: a list of all matching version expression ids

Keep in mind that this actively does requests, so preferably don't do this in bulk, and/or cache your results.

Takes the output of alineas_with_selective_path() puts text fragments together when their specified ['parts'] values are the same.

In other words, this lets you control just how flat to make the text, e.g.

• flatten all text within a lid (e.g. flattening lists),
• smush all lid text within an article together, etc.
• mostly flatten out the text, but still group it by hoofdstuk if those are present

...etc.

CONSIDER: returning a meta dict for each such grouped text (instead of the raw key)

Some of the XML files presented to us as XML data are actually devoid of content text. This determines if it's such a case. We actually parse it, so we can better distinguish between empty and near-empty.

Parses two different metadata-only XML styles found in KOOP's Offiele Publicaties repositories

• the one that looks like `<metadata_gegevens>` with a set of `<metadata name="DC.title" scheme="" content="...`
• the one that (after a namespace strip) looks like `<owms-metadata><owmskern>` with e.g. `<dcterms:identifier>gmb-...`

could also raise, e.g. a XMLSyntaxError

NOT TO BE CONFUSED with parse_op_searchmeta

CONSIDER: TODO: a similar flatten parameter (probably defaulting False)

Tries to return them in the same style, e.g.

• taking off the name-based grouping from DC.title
• taking off the tag-based grouping (ignoring owmskern tag)

similar to cvdr_meta; CONSIDER: abstract most of that into one helper function

Note that

• the 'enriched' and 'manifestations' keys show equivalent information
• the 'enriched' and 'manifestations' keys are not affected by flattening.

TODO: see how well this holds up to all areas

    {'doctype': 'ag-ek',
    'group': '1995',
    'doc_id': 'ag-ek-1995-02-08',
    'mn': '1',
    'exprtype': 'xml',
    'bn': 'ag-ek-1995-02-08.xml',
    'url': 'https://repository.overheid.nl/frbr/officielepublicaties/ag-ek/1995/ag-ek-1995-02-08/1/xml/ag-ek-1995-02-08.xml'}

Given a bunch of document types such as:

    ['metadata', 'metadataowms', 'pdf','odt', 'jpg', 'coordinaten', 'ocr', 'html', 'xml']

return only a subset of types, for defaults:

    ['metadata', 'metadataowms', 'xml', 'html']

...because those defaults prefer smaller, smaller, more data-like/simpler-to-parse variants and avoid content redundancy that we probably won't end up reading (e.g. why prse odt, pdf when you have html, xml).

DECISIONS/ASSUMPTIONS MADE:

• things like 'don't care about HTML when we have XML' which might not fit your needs exactly.
• Assumes xml is content, and it can be other things. You probably want to be sure about your specific purpose.
• Consider that if you e.g. hand in `['metadata', 'metadataowms', 'xml', 'pdf']` it will return _both_ xml and PDF. If you specifically wanted only one content thing, wanted only one, xml, then you should probably say first_of=('xml, 'html', 'html.zip', 'pdf', 'odt')