GoForge — Build | Test | Ship | Collaborate GoForge — Build | Test | Ship | Collaborate
v1.0.1

jsoup

HTML parsing — the DOM as a GADT

$ go get goforge.dev/gpsoup@v1.0.1
Source repository pkg.go.dev

Released — v1.0.0 (full feature parity). goforge.dev/gpsoup is a Go+ rewrite of jhy/jsoup — the HTML parser, DOM, sanitizer, and HTTP client — that models the DOM as a GADT and consumes std/parsec for its CSS-selector grammar. Every jsoup subsystem is ported and differentially tested against jsoup’s own JUnit oracle cases; the parity ledger is the null set. Install with go get goforge.dev/gpsoup@v1.0.1.

GoForge jsoup targets jhy/jsoup jsoup-1.22.2 (pinned). Requires the goplus v0.140.1 toolchain to regenerate. As of v1.0.1 the buffered rune scanner behind the tokenizer is std/text/scan and byte-level charset detection/decoding is std/charset (stdlib v0.208.0) — both extracted from this port for the wider portfolio, consumed with no functional change.

The DOM is a GADT

jsoup’s DOM is a mutable, cyclic node graph — the canonical case for goplus’s //goplus:box pointer enums. Node is a pointer-boxed enum whose variants (Document, Element, Text, Comment, Data, CData, Doctype, XmlDecl) each carry a shared *nodeCore (parent / children / sibling index), and every shared operation — NodeName, AppendChild, serialization, text extraction — is an exhaustive match, so adding a node kind without handling it everywhere is a compile error. The tokenizer’s Token is a pointer-boxed enum too. gpsoup is a real, DOM-shaped consumer of the pointer-boxed-enum feature it helped motivate. Deep traversals use Go+ tail recursion (tail func + recur), so serialization, text, and selection are stack-safe on arbitrarily deep documents.

std/parsec

The tokenizer’s tag/attribute-name grammar and the whole CSS-selector grammar (including the :nth-child an+b expression) are std/parsec combinators.

Full parity — every subsystem

  • HTML5 tokenizer — jsoup’s faithful ~69-state machine (CharacterReader + Token + Tokeniser + TokeniserState, ~3000 LOC), verified against jsoup’s CharacterReaderTest. Malformed markup never fails.
  • HTML5 tree construction — the full algorithm (TreeBuilder + HtmlTreeBuilder + HtmlTreeBuilderState): the 24 insertion modes, document normalization, implicit <tbody>/<tr>, <p>/<li>/heading auto-closing, table foster parenting, active-formatting reconstruction, and the adoption agency algorithm (<b>bold<i>both</b>italic → <b>bold<i>both</i></b><i>italic</i>).
  • Serializer — jsoup’s pretty-printer: block-boundary whitespace normalization, indentation, void/self-close handling, boolean-attr collapse, and the base/extended/xhtml escape modes with charset-aware entity encoding.
  • Text — TextOf with block-boundary spacing, readable-boundary tag separators, &nbsp;/invisible normalization, <pre>/<code> preservation, and the complete 2125-entry entity table plus numeric references.
  • CSS selectors — the full Evaluator / QueryParser: attribute operators, the combinators > / + / ~ / descendant, structural pseudo-classes, :has, :not, :matchText (serialization-transparent), the ::text / ::comment / ::data / ::cdata / ::node node selectors, and namespaces.
  • Sanitizer — safety: Cleaner + Safelist (none / basic / basicWithImages / relaxed), base-URI resolution, external-only rel=nofollow, and IsValid / IsValidBodyHtml.
  • HTTP — Connection / HttpConnection, consuming std/http (HTTP/3 → HTTP/2 → HTTP/1.1): sessions, proxies, TLS options, request-body streams, and DataUtil charset detection (BOM / meta / WHATWG index) with gzip.
  • XML — XmlTreeBuilder with namespace scoping, case preservation, a configurable TagSet, DOCTYPE/CDATA/decl nodes, and a dedicated serializer.
  • Traversal, mutation & streaming — NodeVisitor / NodeTraversor / NodeFilter, deep Clone, FormElement association, source ranges, the DOM mutators, and a genuinely incremental StreamParser that steps the tree builder token-by-token and emits each element as it completes.

Differential parity

Each subsystem carries a *_diff_test.go that ports jsoup’s own JUnit cases (exact inputs and expected outputs) and enforces them with t.Errorf — HTML tree construction (HtmlParserTest), serialization, .text(), selectors (SelectorTest), the sanitizer (CleanerTest / SafelistTest), XML, and the stream parser (StreamParserTest). The parity ledger is empty: no t.Skips and no logged-but-unenforced divergences. Gen-check, go vet, and -race are clean.

Provenance

Upstream: jhy/jsoup, MIT (© Jonathan Hedley). Independent Go+ implementation, not vendored source.