This project is closed source for confidentiality reasons because the codebase has come into contact with experimental models. I plan on reusing this library, so I might do a clean-room reimplementation and open-source it.
For the project that motivated this library, see here.
Section Extraction
The primary purpose of gdocs_sections is section extraction. The full pipeline is supported, from authentication, fetching the document, parsing, content extraction, to final output formatting.
A "section" consists of a section header and body text. Typically, the section header corresponds to Google Docs' headers, which come in six levels, "Heading 1" to "Heading 6" in Docs UI, or HEADER_1 to HEADER_6 in Docs JSON output. By default, gdocs_sections captures sections headed with a Heading 3 and groups them by the Header 2 supersection which they appear under, but can be configured to capture or group sections at any heading level, including at multiple levels. Other Google Docs paragraph styles can be manually provided as section breaks.
Text is captured until the end of the section, which may consist of multiple paragraphs or subsections. The section ends at the start of the next section header, the start of a larger supersection, the end of the document, or another custom ending criterion.
Handled formatting includes header level, bold and italics, and text colour. Horizontal rules are handled and can be used as section boundaries but are not captured in the resulting sections and cannot be pasted into Google Docs. For simplicity and to avoid reproducing Google Docs' entire internal text model, gdocs_sections ignores formatting such as font, font size, and line spacing. Generally this is not an issue as pasting into a Google Doc will match most of these settings to the surrounding document. Parsing also ignores images and tables, though it is capable of extracting text within tables.
Output Formats
Internally, gdocs_sections works with Google Docs' JSON format which it converts to a simplified internal representation, but can convert this to a variety of output representations. Four output formats are supported. Most simply, the contents can be output as plain text, stripped of all formatting. To preserve some formatting while still being human-readable, gdocs_sections can output Markdown, preserving headings and other Markdown-compatible formatting such as bold and italics, while discarding incompatible formatting such as colours.
The complete representation with all formatting can be output in two HTML forms. The first format, html-web, is designed to be embedded in a web page, to display the extracted document contents with the same appearance as the original document. The second format, html-docs, is designed to be pasted into Google Docs. It will preserve all captured formatting, including header levels, which will show correctly in the outliner/table of contents.
To paste this, it is necessary to store it in the clipboard's text/html MIME type. The CLI for gdocs_sections provides an option to send this directly to the clipboard using xclip, if supported. When used as an API, it is recommended to simultaneously store html-docs in text/html while storing either plain text or Markdown in text/plain, so the same copy operation can be pasted into programs besides Docs.
Authentication
To access Google Docs via Google's API, it is necessary to authenticate, even if the document is public. To help with this, gdocs_sections comes with a small helper that authenticates with Application Default Credentials (ADC) if you have them locally on your system, and another simpler auth option that can be used if you are running inside Google Colab. The library can also work with user-provided credentials, if you want to hook it up to OAuth or some other Google credentials provider.
CLI
usage: python -m gdoc_sections.cli [-h] [--section-style SECTION_STYLE]
[--group-styles [GROUP_STYLES ...]] [--reset-styles [RESET_STYLES ...]]
[--body-styles [BODY_STYLES ...]] [--format {html-docs,html-web,markdown,plain}]
[--output OUTPUT] [--clipboard] [--list-groups] [--dump-json] urlThe library can also be accessed through a bundled stand-alone CLI tool.
The positional argument url is the Google Docs URL.
-h or --help shows the help message and exits.
--section-style SECTION_STYLE sets the paragraph style that starts a new section. The default is HEADING_3.
--group-styles [GROUP_STYLES ...] sets the paragraph style or styles tracked as each section's supersection, for lookup and --list-groups purposes. The default is HEADING_2. Pass it with no values to disable grouping.
--reset-styles [RESET_STYLES ...] sets styles that end a section without starting a new one or being tracked as a group.
--body-styles [BODY_STYLES ...] sets styles whose paragraphs are collected as section body content.
--format {html-docs,html-web,markdown,plain} sets the output format. The default is plain.
--output or -o OUTPUT writes output to a file instead of stdout.
--clipboard also copies the rendered output to the X11 clipboard as HTML via xclip. This requires --format to be html-web or html-docs.
--list-groups prints the group structure instead of rendering content: each group heading followed by the section headings under it.
--dump-json prints the raw Google Docs API JSON for the document instead of extracting and rendering. This is useful for inspecting document structure to find styles.