1goldmark
2==========================================
3
4[](https://pkg.go.dev/github.com/yuin/goldmark)
5[](https://github.com/yuin/goldmark/actions?query=workflow:test)
6[](https://coveralls.io/github/yuin/goldmark)
7[](https://goreportcard.com/report/github.com/yuin/goldmark)
8
9> A Markdown parser written in Go. Easy to extend, standards-compliant, well-structured.
10
11goldmark is compliant with CommonMark 0.30.
12
13Motivation
14----------------------
15I needed a Markdown parser for Go that satisfies the following requirements:
16
17- Easy to extend.
18 - Markdown is poor in document expressions compared to other light markup languages such as reStructuredText.
19 - We have extensions to the Markdown syntax, e.g. PHP Markdown Extra, GitHub Flavored Markdown.
20- Standards-compliant.
21 - Markdown has many dialects.
22 - GitHub-Flavored Markdown is widely used and is based upon CommonMark, effectively mooting the question of whether or not CommonMark is an ideal specification.
23 - CommonMark is complicated and hard to implement.
24- Well-structured.
25 - AST-based; preserves source position of nodes.
26- Written in pure Go.
27
28[golang-commonmark](https://gitlab.com/golang-commonmark/markdown) may be a good choice, but it seems to be a copy of [markdown-it](https://github.com/markdown-it).
29
30[blackfriday.v2](https://github.com/russross/blackfriday/tree/v2) is a fast and widely-used implementation, but is not CommonMark-compliant and cannot be extended from outside of the package, since its AST uses structs instead of interfaces.
31
32Furthermore, its behavior differs from other implementations in some cases, especially regarding lists: [Deep nested lists don't output correctly #329](https://github.com/russross/blackfriday/issues/329), [List block cannot have a second line #244](https://github.com/russross/blackfriday/issues/244), etc.
33
34This behavior sometimes causes problems. If you migrate your Markdown text from GitHub to blackfriday-based wikis, many lists will immediately be broken.
35
36As mentioned above, CommonMark is complicated and hard to implement, so Markdown parsers based on CommonMark are few and far between.
37
38Features
39----------------------
40
41- **Standards-compliant.** goldmark is fully compliant with the latest [CommonMark](https://commonmark.org/) specification.
42- **Extensible.** Do you want to add a `@username` mention syntax to Markdown?
43 You can easily do so in goldmark. You can add your AST nodes,
44 parsers for block-level elements, parsers for inline-level elements,
45 transformers for paragraphs, transformers for the whole AST structure, and
46 renderers.
47- **Performance.** goldmark's performance is on par with that of cmark,
48 the CommonMark reference implementation written in C.
49- **Robust.** goldmark is tested with `go test --fuzz`.
50- **Built-in extensions.** goldmark ships with common extensions like tables, strikethrough,
51 task lists, and definition lists.
52- **Depends only on standard libraries.**
53
54Installation
55----------------------
56```bash
57$ go get github.com/yuin/goldmark
58```
59
60
61Usage
62----------------------
63Import packages:
64
65```go
66import (
67 "bytes"
68 "github.com/yuin/goldmark"
69)
70```
71
72
73Convert Markdown documents with the CommonMark-compliant mode:
74
75```go
76var buf bytes.Buffer
77if err := goldmark.Convert(source, &buf); err != nil {
78 panic(err)
79}
80```
81
82With options
83------------------------------
84
85```go
86var buf bytes.Buffer
87if err := goldmark.Convert(source, &buf, parser.WithContext(ctx)); err != nil {
88 panic(err)
89}
90```
91
92| Functional option | Type | Description |
93| ----------------- | ---- | ----------- |
94| `parser.WithContext` | A `parser.Context` | Context for the parsing phase. |
95
96Context options
97----------------------
98
99| Functional option | Type | Description |
100| ----------------- | ---- | ----------- |
101| `parser.WithIDs` | A `parser.IDs` | `IDs` allows you to change logics that are related to element id(ex: Auto heading id generation). |
102
103
104Custom parser and renderer
105--------------------------
106```go
107import (
108 "bytes"
109 "github.com/yuin/goldmark"
110 "github.com/yuin/goldmark/extension"
111 "github.com/yuin/goldmark/parser"
112 "github.com/yuin/goldmark/renderer/html"
113)
114
115md := goldmark.New(
116 goldmark.WithExtensions(extension.GFM),
117 goldmark.WithParserOptions(
118 parser.WithAutoHeadingID(),
119 ),
120 goldmark.WithRendererOptions(
121 html.WithHardWraps(),
122 html.WithXHTML(),
123 ),
124 )
125var buf bytes.Buffer
126if err := md.Convert(source, &buf); err != nil {
127 panic(err)
128}
129```
130
131| Functional option | Type | Description |
132| ----------------- | ---- | ----------- |
133| `goldmark.WithParser` | `parser.Parser` | This option must be passed before `goldmark.WithParserOptions` and `goldmark.WithExtensions` |
134| `goldmark.WithRenderer` | `renderer.Renderer` | This option must be passed before `goldmark.WithRendererOptions` and `goldmark.WithExtensions` |
135| `goldmark.WithParserOptions` | `...parser.Option` | |
136| `goldmark.WithRendererOptions` | `...renderer.Option` | |
137| `goldmark.WithExtensions` | `...goldmark.Extender` | |
138
139Parser and Renderer options
140------------------------------
141
142### Parser options
143
144| Functional option | Type | Description |
145| ----------------- | ---- | ----------- |
146| `parser.WithBlockParsers` | A `util.PrioritizedSlice` whose elements are `parser.BlockParser` | Parsers for parsing block level elements. |
147| `parser.WithInlineParsers` | A `util.PrioritizedSlice` whose elements are `parser.InlineParser` | Parsers for parsing inline level elements. |
148| `parser.WithParagraphTransformers` | A `util.PrioritizedSlice` whose elements are `parser.ParagraphTransformer` | Transformers for transforming paragraph nodes. |
149| `parser.WithASTTransformers` | A `util.PrioritizedSlice` whose elements are `parser.ASTTransformer` | Transformers for transforming an AST. |
150| `parser.WithAutoHeadingID` | `-` | Enables auto heading ids. |
151| `parser.WithAttribute` | `-` | Enables custom attributes. Currently only headings supports attributes. |
152
153### HTML Renderer options
154
155| Functional option | Type | Description |
156| ----------------- | ---- | ----------- |
157| `html.WithWriter` | `html.Writer` | `html.Writer` for writing contents to an `io.Writer`. |
158| `html.WithHardWraps` | `-` | Render newlines as `<br>`.|
159| `html.WithXHTML` | `-` | Render as XHTML. |
160| `html.WithUnsafe` | `-` | By default, goldmark does not render raw HTML or potentially dangerous links. With this option, goldmark renders such content as written. |
161
162### Built-in extensions
163
164- `extension.Table`
165 - [GitHub Flavored Markdown: Tables](https://github.github.com/gfm/#tables-extension-)
166- `extension.Strikethrough`
167 - [GitHub Flavored Markdown: Strikethrough](https://github.github.com/gfm/#strikethrough-extension-)
168- `extension.Linkify`
169 - [GitHub Flavored Markdown: Autolinks](https://github.github.com/gfm/#autolinks-extension-)
170- `extension.TaskList`
171 - [GitHub Flavored Markdown: Task list items](https://github.github.com/gfm/#task-list-items-extension-)
172- `extension.GFM`
173 - This extension enables Table, Strikethrough, Linkify and TaskList.
174 - This extension does not filter tags defined in [6.11: Disallowed Raw HTML (extension)](https://github.github.com/gfm/#disallowed-raw-html-extension-).
175 If you need to filter HTML tags, see [Security](#security).
176 - If you need to parse github emojis, you can use [goldmark-emoji](https://github.com/yuin/goldmark-emoji) extension.
177- `extension.DefinitionList`
178 - [PHP Markdown Extra: Definition lists](https://michelf.ca/projects/php-markdown/extra/#def-list)
179- `extension.Footnote`
180 - [PHP Markdown Extra: Footnotes](https://michelf.ca/projects/php-markdown/extra/#footnotes)
181- `extension.Typographer`
182 - This extension substitutes punctuations with typographic entities like [smartypants](https://daringfireball.net/projects/smartypants/).
183- `extension.CJK`
184 - This extension is a shortcut for CJK related functionalities.
185
186### Attributes
187The `parser.WithAttribute` option allows you to define attributes on some elements.
188
189Currently only headings support attributes.
190
191**Attributes are being discussed in the
192[CommonMark forum](https://talk.commonmark.org/t/consistent-attribute-syntax/272).
193This syntax may possibly change in the future.**
194
195
196#### Headings
197
198```
199## heading ## {#id .className attrName=attrValue class="class1 class2"}
200
201## heading {#id .className attrName=attrValue class="class1 class2"}
202```
203
204```
205heading {#id .className attrName=attrValue}
206============
207```
208
209### Table extension
210The Table extension implements [Table(extension)](https://github.github.com/gfm/#tables-extension-), as
211defined in [GitHub Flavored Markdown Spec](https://github.github.com/gfm/).
212
213Specs are defined for XHTML, so specs use some deprecated attributes for HTML5.
214
215You can override alignment rendering method via options.
216
217| Functional option | Type | Description |
218| ----------------- | ---- | ----------- |
219| `extension.WithTableCellAlignMethod` | `extension.TableCellAlignMethod` | Option indicates how are table cells aligned. |
220
221### Typographer extension
222
223The Typographer extension translates plain ASCII punctuation characters into typographic-punctuation HTML entities.
224
225Default substitutions are:
226
227| Punctuation | Default entity |
228| ------------ | ---------- |
229| `'` | `‘`, `’` |
230| `"` | `“`, `”` |
231| `--` | `–` |
232| `---` | `—` |
233| `...` | `…` |
234| `<<` | `«` |
235| `>>` | `»` |
236
237You can override the default substitutions via `extensions.WithTypographicSubstitutions`:
238
239```go
240markdown := goldmark.New(
241 goldmark.WithExtensions(
242 extension.NewTypographer(
243 extension.WithTypographicSubstitutions(extension.TypographicSubstitutions{
244 extension.LeftSingleQuote: []byte("‚"),
245 extension.RightSingleQuote: nil, // nil disables a substitution
246 }),
247 ),
248 ),
249)
250```
251
252### Linkify extension
253
254The Linkify extension implements [Autolinks(extension)](https://github.github.com/gfm/#autolinks-extension-), as
255defined in [GitHub Flavored Markdown Spec](https://github.github.com/gfm/).
256
257Since the spec does not define details about URLs, there are numerous ambiguous cases.
258
259You can override autolinking patterns via options.
260
261| Functional option | Type | Description |
262| ----------------- | ---- | ----------- |
263| `extension.WithLinkifyAllowedProtocols` | `[][]byte` | List of allowed protocols such as `[][]byte{ []byte("http:") }` |
264| `extension.WithLinkifyURLRegexp` | `*regexp.Regexp` | Regexp that defines URLs, including protocols |
265| `extension.WithLinkifyWWWRegexp` | `*regexp.Regexp` | Regexp that defines URL starting with `www.`. This pattern corresponds to [the extended www autolink](https://github.github.com/gfm/#extended-www-autolink) |
266| `extension.WithLinkifyEmailRegexp` | `*regexp.Regexp` | Regexp that defines email addresses` |
267
268Example, using [xurls](https://github.com/mvdan/xurls):
269
270```go
271import "mvdan.cc/xurls/v2"
272
273markdown := goldmark.New(
274 goldmark.WithRendererOptions(
275 html.WithXHTML(),
276 html.WithUnsafe(),
277 ),
278 goldmark.WithExtensions(
279 extension.NewLinkify(
280 extension.WithLinkifyAllowedProtocols([][]byte{
281 []byte("http:"),
282 []byte("https:"),
283 }),
284 extension.WithLinkifyURLRegexp(
285 xurls.Strict,
286 ),
287 ),
288 ),
289)
290```
291
292### Footnotes extension
293
294The Footnote extension implements [PHP Markdown Extra: Footnotes](https://michelf.ca/projects/php-markdown/extra/#footnotes).
295
296This extension has some options:
297
298| Functional option | Type | Description |
299| ----------------- | ---- | ----------- |
300| `extension.WithFootnoteIDPrefix` | `[]byte` | a prefix for the id attributes.|
301| `extension.WithFootnoteIDPrefixFunction` | `func(gast.Node) []byte` | a function that determines the id attribute for given Node.|
302| `extension.WithFootnoteLinkTitle` | `[]byte` | an optional title attribute for footnote links.|
303| `extension.WithFootnoteBacklinkTitle` | `[]byte` | an optional title attribute for footnote backlinks. |
304| `extension.WithFootnoteLinkClass` | `[]byte` | a class for footnote links. This defaults to `footnote-ref`. |
305| `extension.WithFootnoteBacklinkClass` | `[]byte` | a class for footnote backlinks. This defaults to `footnote-backref`. |
306| `extension.WithFootnoteBacklinkHTML` | `[]byte` | a class for footnote backlinks. This defaults to `↩︎`. |
307
308Some options can have special substitutions. Occurrences of “^^” in the string will be replaced by the corresponding footnote number in the HTML output. Occurrences of “%%” will be replaced by a number for the reference (footnotes can have multiple references).
309
310`extension.WithFootnoteIDPrefix` and `extension.WithFootnoteIDPrefixFunction` are useful if you have multiple Markdown documents displayed inside one HTML document to avoid footnote ids to clash each other.
311
312`extension.WithFootnoteIDPrefix` sets fixed id prefix, so you may write codes like the following:
313
314```go
315for _, path := range files {
316 source := readAll(path)
317 prefix := getPrefix(path)
318
319 markdown := goldmark.New(
320 goldmark.WithExtensions(
321 NewFootnote(
322 WithFootnoteIDPrefix([]byte(path)),
323 ),
324 ),
325 )
326 var b bytes.Buffer
327 err := markdown.Convert(source, &b)
328 if err != nil {
329 t.Error(err.Error())
330 }
331}
332```
333
334`extension.WithFootnoteIDPrefixFunction` determines an id prefix by calling given function, so you may write codes like the following:
335
336```go
337markdown := goldmark.New(
338 goldmark.WithExtensions(
339 NewFootnote(
340 WithFootnoteIDPrefixFunction(func(n gast.Node) []byte {
341 v, ok := n.OwnerDocument().Meta()["footnote-prefix"]
342 if ok {
343 return util.StringToReadOnlyBytes(v.(string))
344 }
345 return nil
346 }),
347 ),
348 ),
349)
350
351for _, path := range files {
352 source := readAll(path)
353 var b bytes.Buffer
354
355 doc := markdown.Parser().Parse(text.NewReader(source))
356 doc.Meta()["footnote-prefix"] = getPrefix(path)
357 err := markdown.Renderer().Render(&b, source, doc)
358}
359```
360
361You can use [goldmark-meta](https://github.com/yuin/goldmark-meta) to define a id prefix in the markdown document:
362
363
364```markdown
365---
366title: document title
367slug: article1
368footnote-prefix: article1
369---
370
371# My article
372
373```
374
375### CJK extension
376CommonMark gives compatibilities a high priority and original markdown was designed by westerners. So CommonMark lacks considerations for languages like CJK.
377
378This extension provides additional options for CJK users.
379
380| Functional option | Type | Description |
381| ----------------- | ---- | ----------- |
382| `extension.WithEastAsianLineBreaks` | `...extension.EastAsianLineBreaksStyle` | Soft line breaks are rendered as a newline. Some asian users will see it as an unnecessary space. With this option, soft line breaks between east asian wide characters will be ignored. |
383| `extension.WithEscapedSpace` | `-` | Without spaces around an emphasis started with east asian punctuations, it is not interpreted as an emphasis(as defined in CommonMark spec). With this option, you can avoid this inconvenient behavior by putting 'not rendered' spaces around an emphasis like `太郎は\ **「こんにちわ」**\ といった`. |
384
385#### Styles of Line Breaking
386
387| Style | Description |
388| ----- | ----------- |
389| `EastAsianLineBreaksStyleSimple` | Soft line breaks are ignored if both sides of the break are east asian wide character. This behavior is the same as [`east_asian_line_breaks`](https://pandoc.org/MANUAL.html#extension-east_asian_line_breaks) in Pandoc. |
390| `EastAsianLineBreaksCSS3Draft` | This option implements CSS text level3 [Segment Break Transformation Rules](https://drafts.csswg.org/css-text-3/#line-break-transform) with [some enhancements](https://github.com/w3c/csswg-drafts/issues/5086). |
391
392#### Example of `EastAsianLineBreaksStyleSimple`
393
394Input Markdown:
395
396```md
397私はプログラマーです。
398東京の会社に勤めています。
399GoでWebアプリケーションを開発しています。
400```
401
402Output:
403
404```html
405<p>私はプログラマーです。東京の会社に勤めています。\nGoでWebアプリケーションを開発しています。</p>
406```
407
408#### Example of `EastAsianLineBreaksCSS3Draft`
409
410Input Markdown:
411
412```md
413私はプログラマーです。
414東京の会社に勤めています。
415GoでWebアプリケーションを開発しています。
416```
417
418Output:
419
420```html
421<p>私はプログラマーです。東京の会社に勤めています。GoでWebアプリケーションを開発しています。</p>
422```
423
424Security
425--------------------
426By default, goldmark does not render raw HTML or potentially-dangerous URLs.
427If you need to gain more control over untrusted contents, it is recommended that you
428use an HTML sanitizer such as [bluemonday](https://github.com/microcosm-cc/bluemonday).
429
430Benchmark
431--------------------
432You can run this benchmark in the `_benchmark` directory.
433
434### against other golang libraries
435
436blackfriday v2 seems to be the fastest, but as it is not CommonMark compliant, its performance cannot be directly compared to that of the CommonMark-compliant libraries.
437
438goldmark, meanwhile, builds a clean, extensible AST structure, achieves full compliance with
439CommonMark, and consumes less memory, all while being reasonably fast.
440
441- MBP 2019 13″(i5, 16GB), Go1.17
442
443```
444BenchmarkMarkdown/Blackfriday-v2-8 302 3743747 ns/op 3290445 B/op 20050 allocs/op
445BenchmarkMarkdown/GoldMark-8 280 4200974 ns/op 2559738 B/op 13435 allocs/op
446BenchmarkMarkdown/CommonMark-8 226 5283686 ns/op 2702490 B/op 20792 allocs/op
447BenchmarkMarkdown/Lute-8 12 92652857 ns/op 10602649 B/op 40555 allocs/op
448BenchmarkMarkdown/GoMarkdown-8 13 81380167 ns/op 2245002 B/op 22889 allocs/op
449```
450
451### against cmark (CommonMark reference implementation written in C)
452
453- MBP 2019 13″(i5, 16GB), Go1.17
454
455```
456----------- cmark -----------
457file: _data.md
458iteration: 50
459average: 0.0044073057 sec
460------- goldmark -------
461file: _data.md
462iteration: 50
463average: 0.0041611990 sec
464```
465
466As you can see, goldmark's performance is on par with cmark's.
467
468Extensions
469--------------------
470
471- [goldmark-meta](https://github.com/yuin/goldmark-meta): A YAML metadata
472 extension for the goldmark Markdown parser.
473- [goldmark-highlighting](https://github.com/yuin/goldmark-highlighting): A syntax-highlighting extension
474 for the goldmark markdown parser.
475- [goldmark-emoji](https://github.com/yuin/goldmark-emoji): An emoji
476 extension for the goldmark Markdown parser.
477- [goldmark-mathjax](https://github.com/litao91/goldmark-mathjax): Mathjax support for the goldmark markdown parser
478- [goldmark-pdf](https://github.com/stephenafamo/goldmark-pdf): A PDF renderer that can be passed to `goldmark.WithRenderer()`.
479- [goldmark-hashtag](https://github.com/abhinav/goldmark-hashtag): Adds support for `#hashtag`-based tagging to goldmark.
480- [goldmark-wikilink](https://github.com/abhinav/goldmark-wikilink): Adds support for `[[wiki]]`-style links to goldmark.
481- [goldmark-anchor](https://github.com/abhinav/goldmark-anchor): Adds anchors (permalinks) next to all headers in a document.
482- [goldmark-figure](https://github.com/mangoumbrella/goldmark-figure): Adds support for rendering paragraphs starting with an image to `<figure>` elements.
483- [goldmark-frontmatter](https://github.com/abhinav/goldmark-frontmatter): Adds support for YAML, TOML, and custom front matter to documents.
484- [goldmark-toc](https://github.com/abhinav/goldmark-toc): Adds support for generating tables-of-contents for goldmark documents.
485- [goldmark-mermaid](https://github.com/abhinav/goldmark-mermaid): Adds support for rendering [Mermaid](https://mermaid-js.github.io/mermaid/) diagrams in goldmark documents.
486- [goldmark-pikchr](https://github.com/jchenry/goldmark-pikchr): Adds support for rendering [Pikchr](https://pikchr.org/home/doc/trunk/homepage.md) diagrams in goldmark documents.
487- [goldmark-embed](https://github.com/13rac1/goldmark-embed): Adds support for rendering embeds from YouTube links.
488- [goldmark-latex](https://github.com/soypat/goldmark-latex): A $\LaTeX$ renderer that can be passed to `goldmark.WithRenderer()`.
489- [goldmark-fences](https://github.com/stefanfritsch/goldmark-fences): Support for pandoc-style [fenced divs](https://pandoc.org/MANUAL.html#divs-and-spans) in goldmark.
490- [goldmark-d2](https://github.com/FurqanSoftware/goldmark-d2): Adds support for [D2](https://d2lang.com/) diagrams.
491- [goldmark-katex](https://github.com/FurqanSoftware/goldmark-katex): Adds support for [KaTeX](https://katex.org/) math and equations.
492- [goldmark-img64](https://github.com/tenkoh/goldmark-img64): Adds support for embedding images into the document as DataURL (base64 encoded).
493
494
495goldmark internal(for extension developers)
496----------------------------------------------
497### Overview
498goldmark's Markdown processing is outlined in the diagram below.
499
500```
501 <Markdown in []byte, parser.Context>
502 |
503 V
504 +-------- parser.Parser ---------------------------
505 | 1. Parse block elements into AST
506 | 1. If a parsed block is a paragraph, apply
507 | ast.ParagraphTransformer
508 | 2. Traverse AST and parse blocks.
509 | 1. Process delimiters(emphasis) at the end of
510 | block parsing
511 | 3. Apply parser.ASTTransformers to AST
512 |
513 V
514 <ast.Node>
515 |
516 V
517 +------- renderer.Renderer ------------------------
518 | 1. Traverse AST and apply renderer.NodeRenderer
519 | corespond to the node type
520
521 |
522 V
523 <Output>
524```
525
526### Parsing
527Markdown documents are read through `text.Reader` interface.
528
529AST nodes do not have concrete text. AST nodes have segment information of the documents, represented by `text.Segment` .
530
531`text.Segment` has 3 attributes: `Start`, `End`, `Padding` .
532
533(TBC)
534
535**TODO**
536
537See `extension` directory for examples of extensions.
538
539Summary:
540
5411. Define AST Node as a struct in which `ast.BaseBlock` or `ast.BaseInline` is embedded.
5422. Write a parser that implements `parser.BlockParser` or `parser.InlineParser`.
5433. Write a renderer that implements `renderer.NodeRenderer`.
5444. Define your goldmark extension that implements `goldmark.Extender`.
545
546
547Donation
548--------------------
549BTC: 1NEDSyUmo4SMTDP83JJQSWi1MvQUGGNMZB
550
551License
552--------------------
553MIT
554
555Author
556--------------------
557Yusuke Inuzuka
View as plain text