Package: text/scanner

package scanner Import Path text/scanner (on go.dev) Dependency Relation imports 6 packages, and imported by one package

Involved Source Files d scanner.go Package scanner provides a scanner and tokenizer for UTF-8-encoded text. It takes an io.Reader providing the source, which then can be tokenized through repeated calls to the Scan function. For compatibility with existing tools, the NUL character is not allowed. If the first character in the source is a UTF-8 encoded byte order mark (BOM), it is discarded. By default, a [Scanner] skips white space and Go comments and recognizes all literals as defined by the Go language specification. It may be customized to recognize only a subset of those literals and to recognize different identifier and white space characters.

Code Examples package main import ( "fmt" "strings" "text/scanner" ) func main() { const src = ` // This is scanned code. if a > 10 { someParsable = text }` var s scanner.Scanner s.Init(strings.NewReader(src)) s.Filename = "example" for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() { fmt.Printf("%s: %s\n", s.Position, s.TokenText()) } } _isIdentRune package main import ( "fmt" "strings" "text/scanner" "unicode" ) func main() { const src = "%var1 var2%" var s scanner.Scanner s.Init(strings.NewReader(src)) s.Filename = "default" for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() { fmt.Printf("%s: %s\n", s.Position, s.TokenText()) } fmt.Println() s.Init(strings.NewReader(src)) s.Filename = "percent" // treat leading '%' as part of an identifier s.IsIdentRune = func(ch rune, i int) bool { return ch == '%' && i == 0 || unicode.IsLetter(ch) || unicode.IsDigit(ch) && i > 0 } for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() { fmt.Printf("%s: %s\n", s.Position, s.TokenText()) } } _mode package main import ( "fmt" "strings" "text/scanner" ) func main() { const src = ` // Comment begins at column 5. This line should not be included in the output. /* This multiline comment should be extracted in its entirety. */ ` var s scanner.Scanner s.Init(strings.NewReader(src)) s.Filename = "comments" s.Mode ^= scanner.SkipComments // don't skip comments for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() { txt := s.TokenText() if strings.HasPrefix(txt, "//") || strings.HasPrefix(txt, "/*") { fmt.Printf("%s: %s\n", s.Position, txt) } } } _whitespace package main import ( "fmt" "strings" "text/scanner" ) func main() { // tab-separated values const src = `aa ab ac ad ba bb bc bd ca cb cc cd da db dc dd` var ( col, row int s scanner.Scanner tsv [4][4]string // large enough for example above ) s.Init(strings.NewReader(src)) s.Whitespace ^= 1<<'\t' | 1<<'\n' // don't skip tabs and new lines for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() { switch tok { case '\n': row++ col = 0 case '\t': col++ default: tsv[row][col] = s.TokenText() } } fmt.Print(tsv) }

Package-Level Functions (only one)

func TokenString(tok rune) string TokenString returns a printable string for a token or Unicode character.