wetsuite.helpers.patterns

In case you have a lot of data, you can get cleaner (but reduced!) results by reporting how many distinct documents report the same specific explanation

    { 'AE' : {
        ['Abbreviation', 'Explanation']: 3,  
        ['Abbreviation', 'Erroneous']: 1 
    } }

Finds abbreviations with explanations next to them.

Looks for patterns like

• "Word Combination (WC)"
• "Wet Oven Overheid (Woo)"
• "With Periods (W.P.)"
• "(EA) Explained After" (probably rare)
• "BT (Bracketed terms)"
• "(Bracketed terms) BT" (probably rare)

Will both over- and under-accept, so if you want clean results, consider e.g. reporting only things present in multiple documents. see e.g. merge_results()

CONSIDER:

• how permissive to be with capitalization. Maybe make that a parameter?

• allow and ignore words like 'of', 'the'

• rewrite to deal with cases like

  • Autoriteit Consument en Markt (ACM)
  • De Regeling werving, reclame en verslavingspreventie kansspelen (hierna: Rwrvk)
  • Nationale Postcode Loterij N.V. (hierna: NPL)
  • Edelmetaal Waarborg Nederland B.V. (EWN)
  • College voor Toetsen en Examens (CvTE)
  • (and maybe:)
  • Pensioen- en Uitkeringsraad (PUR)
  • Nederlandse Loodsencorporatie (NLC)
  • Nederlandse Emissieautoriteit (NEa)
  • Kamer voor de Binnenvisserij (Kabivi)
  • (and maybe not:)
  • College van toezicht collectieve beheersorganisaties auteurs- en naburige rechten (College van Toezicht Auteursrechten (CvTA))
  • Keurmerkinstituut jeugdzorg (KMI)

• listening to 'hierna: ', e.g.

  • "Wet Bevordering Integriteitbeoordelingen door het Openbaar Bestuur (hierna: Wet BIBOB)"

  • "Drank- en horecawet (hierna: DHW)"

  • "Algemene wet bestuursrecht (hierna: Awb)"

  • "het Verdrag betreffende de werking van de Europese Unie (hierna: VWEU)"

  • "de Subsidieregeling OPZuid 2021-2027 (hierna: Subsidieregeling OPZuid)"

  • "de Wet werk en bijstand (hierna: WWB)"

  • "de Wet werk en inkomen naar arbeidsvermogen (hierna: WIA)"

  • "de Wet maatschappelijke ondersteuning (hierna: Wmo)"

    These seem to be more structured, in particular when you use (de|het) as a delimiter This seems overly specific, but works well to extract a bunch of these

Attempts to find references like :

    "artikel 5.1, tweede lid, aanhef en onder i, van de Woo"

and parse and resolve as much as it can.

This is a a separate function because it is more complex than most others, but if you want to look for more than just these, then you probably want to wield it via find_references.

These references are not a formalized format, and while the law ( https://wetten.overheid.nl/BWBR0005730/ ) that suggests the format of these should be succinct, and sometimes it looks like these have near-templates, that is not what real-world use looks like.

Another reasonable approach might be include each real-world variant format explicitly, as it lets you put stronger patterns first and fall back on fuzzier ones, it makes it clear what is being matched, and it's easier to see how common each is. However, that also easily leads to false negatives -- missing real references.

Instead, we

• start by finding some strong anchors
• keep accepting bits of adjacent string as long as they look like things we know "artikel 5.1," "tweede lid," "aanhef en onder i"
• then seeing what text is around it, which should be at least the law name

Neither will deal well with the briefest forms, e.g. "(81 WWB)" which is arguably only reasonable to recognize when you recognize either side (by known law name, which is harder for abbreviations in that it probably leads to false positives) ...and in that example, we might want to

• see if character context makes it reasonable - the parthentheses make it more reasonable than if you found the six characters '81 WWB' in any context
• check whether the estimated law (Wet werk en bijstand - BWBR0015703) has an article 81
• check, in some semantic way, whether Wet werk en bijstand makes any sense in context of the text

TODO: ...also so that we can return some estimation of

• how sure we are this is a reference,
• how complete a reference is, and/or
• how easy to resolve a reference is.

Looks for various different kinds of references in the given text, sorts the results.

Note that there is a gliding scale between 'is this and identifier and will we probably find most of them' and 'is this more textual, more varied, so more easily miss parts' (...and should this perhaps not be implemented with regexes as it currently is)

See also:

• Leidraad voor juridische auteurs

  Kamerstukken I 1995/96, 23700, nr. 188b, p. 3.
  Kamerstukken I 2014/15, 33802, C, p. 3.
  Kamerstukken II 1999/2000, 2000/2001, 2001/2002, 26 855.
  Kamerstukken I 2000/2001, 26 855 (250, 250a); 2001/2002, 26 855 (16, 16a, 16b, 16c).

    OJ L 69, 13.3.2013, p. 1
    OJ L 168, 30.6.2009, p. 41–47

    Council Directive 93/42/EEC of 14 June 1993
    Directive 93/42/EEC of 14 June 1993

    Council Regulation (EEC) No 2658/87

Takes a spacy Doc, and matches from you calling find_references, marks it as entities.

*Replaces* the currently marked entities, to avoid overlap. (CONSIDER: marking up in spans instead) (...also because char_span() with alignment_mode='expand' probably makes this easier.

Bases this on the plain text, and then trying to find all the tokens necessary to cover that (that code needs some double checking).

Quick and dirty splitter into words. Mainly used by abbrev_find