wetsuite.helpers.split

This module tries to wrangle distinct types of documents for you, from HTML to PDF, from varied specific sources, into plain text, so that you can consume it more easily.

It tries to give those into smallish chunks, ideally informed by the document structure.

Ideally, this module gets ongoing attention to cover all the documents that this project cares about, and to that end is a somwhat modular design.

You won't care about that until you want to add your own, but it does have some implications to how it should be used:

• decide(docbytes) will give you (score, splitter_object) tuples
  • you can avoid looser-structured splitters by testing for bad scores, _or_ by using decide's thresh to the same effect.
• each splitter object, when called, gives (metadata, intermediate, flat_text) tuples
  • flat_text is a string. Sometimes you care about only this
  • the other two are more source-specific
    • intermeiate tends to be the structure that that flat_text is from, in case that
    • metadata i

So if you want more control, dealing with the structure hints it also spits out, your code might look somehing like:

    for score, splitter in wetsuite.helpers.split.decide( docbytes ):
        for metadata, intermediate, text in splitter.fragments():
            print( text )
            print( '--------------------' )

**If you just want text**, with little control over the details:

    string_list = feeling_lucky( docbytes )

TODO:

• parameters, like "OCR that PDF if you have to"

CONSIDER:

• think about separating the "read document at lower level" code into functions you can use without having to rip it out these classes here. Right now, even if you know the class you want, you need to do:

      f = Fragments_XML_BWB()
      f.accepts()
      f.suitableness()
      for frag in f.fragments():
          print( frag )
  

• in particular the HTML code can probably be made rather faster

  • using lxml.html (or specifically iterparse?) instad of bs4, because bs4 is currently the slowest part of this ...bit of a rewrite, though.
  • having one class contain all of these - mostly to _share_ state like 'we tried to parse with bs4'

• always have a reasonable fallback for each document type (TODO: XML?)

• settling the intermediate format more?

  • Maybe we shouldn't make too many decisions for you, and merely yield suggestions you can easily ignore, e.g.
    • 'seems like same section, <TEXT>'
    • 'seems like new section, <TEXT>'
    • 'seems like page header/footer, <TEXT>'
    • 'seems like page footnotes, <TEXT>'
  • ...so that you, or a helper function, can group into different-sized chunks in the way you need it.

• maybe rename feeling_lucky, maybe move it to lazy?

• we were trying to be inspired by LaTeX hyphenation, which has a simple-but-pretty-great "this is the cost of breaking off here" the analogue of which wo7uld make "Hey can you break this to different degrees (sections, paragraphs)" possible

• There was an earlier idea to specify pattern matchers entirely as data, like:

        presets = {
            'beleidsregels': {
                'body': { 'name': 'div', 'id': 'PaginaContainer' },
                'keep_headers_html': True,
                'keep_tables_html':  True,
            },
        }
    - with a 'this is how to select for the document' and 'this is how to get the body from it', etc.
    - which is worth re-visiting, because 
      - it may be more easily altered in the long run, 
      - it would probably be more portable than this code currently is.
    - a limitation might be expressing more complex selection