...

Package token

import "cuelang.org/go/cue/token"
Overview
Index

Overview ▾

Package token defines constants representing the lexical tokens of the Go programming language and basic operations on tokens (printing, predicates).

Index ▾

Constants
Variables
type File
    func NewFile(filename string, deprecatedBase, size int) *File
    func (f *File) AddLine(offset int)
    func (f *File) AddLineInfo(offset int, filename string, line int)
    func (f *File) Base() int
    func (f *File) Line(p Pos) int
    func (f *File) LineCount() int
    func (f *File) MergeLine(line int)
    func (f *File) Name() string
    func (f *File) Offset(p Pos) int
    func (f *File) Pos(offset int, rel RelPos) Pos
    func (f *File) Position(p Pos) (pos Position)
    func (f *File) PositionFor(p Pos, adjusted bool) (pos Position)
    func (f *File) SetLines(lines []int) bool
    func (f *File) SetLinesForContent(content []byte)
    func (f *File) Size() int
type Pos
    func (p Pos) Add(n int) Pos
    func (p Pos) Before(q Pos) bool
    func (p Pos) Column() int
    func (p Pos) File() *File
    func (p Pos) Filename() string
    func (p Pos) HasRelPos() bool
    func (p Pos) IsNewline() bool
    func (p Pos) IsValid() bool
    func (p Pos) Line() int
    func (p Pos) Offset() int
    func (p Pos) Position() Position
    func (p Pos) RelPos() RelPos
    func (p Pos) String() string
    func (p Pos) WithRel(rel RelPos) Pos
type Position
    func (pos *Position) IsValid() bool
    func (pos Position) String() string
type RelPos
    func (p RelPos) Pos() Pos
    func (p RelPos) String() string
type Token
    func Lookup(ident string) Token
    func (tok Token) IsKeyword() bool
    func (tok Token) IsLiteral() bool
    func (tok Token) IsOperator() bool
    func (tok Token) Precedence() int
    func (tok Token) String() string

Package files

position.go token.go

Constants

A set of constants for precedence-based expression parsing. Non-operators have lowest precedence, followed by operators starting with precedence 1 up to unary operators. The highest precedence serves as "catch-all" precedence for selector, indexing, and other operator and delimiter tokens.

const (
    LowestPrec  = lowestPrec
    UnaryPrec   = unaryPrec
    HighestPrec = highestPrec
)

Variables

NoPos is the zero value for Pos; there is no file and line information associated with it, and NoPos().IsValid() is false. NoPos is always smaller than any other Pos value. The corresponding Position value for NoPos is the zero value for Position.

var NoPos = Pos{}

type File

A File has a name, size, and line offset table.

type File struct {
    // contains filtered or unexported fields
}

func NewFile

func NewFile(filename string, deprecatedBase, size int) *File

NewFile returns a new file with the given OS file name. The size provides the size of the whole file.

The second argument is deprecated. It has no effect.

func (*File) AddLine

func (f *File) AddLine(offset int)

AddLine adds the line offset for a new line. The line offset must be larger than the offset for the previous line and smaller than the file size; otherwise the line offset is ignored.

func (*File) AddLineInfo

func (f *File) AddLineInfo(offset int, filename string, line int)

AddLineInfo adds alternative file and line number information for a given file offset. The offset must be larger than the offset for the previously added alternative line info and smaller than the file size; otherwise the information is ignored.

AddLineInfo is typically used to register alternative position information for //line filename:line comments in source files.

func (*File) Base

func (f *File) Base() int

Base returns the base offset of file f as passed to NewFile.

Deprecated: this method just returns the (deprecated) second argument passed to NewFile.

func (*File) Line

func (f *File) Line(p Pos) int

Line returns the line number for the given file position p; p must be a Pos value in that file or NoPos.

func (*File) LineCount

func (f *File) LineCount() int

LineCount returns the number of lines in file f.

func (*File) MergeLine

func (f *File) MergeLine(line int)

MergeLine merges a line with the following line. It is akin to replacing the newline character at the end of the line with a space (to not change the remaining offsets). To obtain the line number, consult e.g. Position.Line. MergeLine will panic if given an invalid line number.

func (*File) Name

func (f *File) Name() string

Name returns the file name of file f as registered with AddFile.

func (*File) Offset

func (f *File) Offset(p Pos) int

Offset returns the offset for the given file position p; p must be a valid Pos value in that file. f.Offset(f.Pos(offset)) == offset.

func (*File) Pos

func (f *File) Pos(offset int, rel RelPos) Pos

Pos returns the Pos value for the given file offset; the offset must be <= f.Size(). f.Pos(f.Offset(p)) == p.

func (*File) Position

func (f *File) Position(p Pos) (pos Position)

Position returns the Position value for the given file position p. Calling f.Position(p) is equivalent to calling f.PositionFor(p, true).

func (*File) PositionFor

func (f *File) PositionFor(p Pos, adjusted bool) (pos Position)

PositionFor returns the Position value for the given file position p. If adjusted is set, the position may be adjusted by position-altering //line comments; otherwise those comments are ignored. p must be a Pos value in f or NoPos.

func (*File) SetLines

func (f *File) SetLines(lines []int) bool

SetLines sets the line offsets for a file and reports whether it succeeded. The line offsets are the offsets of the first character of each line; for instance for the content "ab\nc\n" the line offsets are {0, 3}. An empty file has an empty line offset table. Each line offset must be larger than the offset for the previous line and smaller than the file size; otherwise SetLines fails and returns false. Callers must not mutate the provided slice after SetLines returns.

func (*File) SetLinesForContent

func (f *File) SetLinesForContent(content []byte)

SetLinesForContent sets the line offsets for the given file content. It ignores position-altering //line comments.

func (*File) Size

func (f *File) Size() int

Size returns the size of file f as passed to NewFile.

type Pos

Pos is a compact encoding of a source position within a file, as well as relative positioning information. It can be converted into a Position for a more convenient, but much larger, representation.

type Pos struct {
    // contains filtered or unexported fields
}

func (Pos) Add

func (p Pos) Add(n int) Pos

Add creates a new position relative to the p offset by n.

func (Pos) Before

func (p Pos) Before(q Pos) bool

func (Pos) Column

func (p Pos) Column() int

func (Pos) File

func (p Pos) File() *File

File returns the file that contains the position p or nil if there is no such file (for instance for p == NoPos).

func (Pos) Filename

func (p Pos) Filename() string

func (Pos) HasRelPos

func (p Pos) HasRelPos() bool

HasRelPos repors whether p has a relative position.

func (Pos) IsNewline

func (p Pos) IsNewline() bool

IsNewline reports whether the relative information suggests this node should be printed on a new lien.

func (Pos) IsValid

func (p Pos) IsValid() bool

IsValid reports whether the position is valid.

func (Pos) Line

func (p Pos) Line() int

func (Pos) Offset

func (p Pos) Offset() int

Offset reports the byte offset relative to the file.

func (Pos) Position

func (p Pos) Position() Position

func (Pos) RelPos

func (p Pos) RelPos() RelPos

func (Pos) String

func (p Pos) String() string

func (Pos) WithRel

func (p Pos) WithRel(rel RelPos) Pos

type Position

Position describes an arbitrary source position including the file, line, and column location. A Position is valid if the line number is > 0.

type Position struct {
    Filename string // filename, if any
    Offset   int    // offset, starting at 0
    Line     int    // line number, starting at 1
    Column   int    // column number, starting at 1 (byte count)

}

func (*Position) IsValid

func (pos *Position) IsValid() bool

IsValid reports whether the position is valid.

func (Position) String

func (pos Position) String() string

String returns a string in one of several forms:

file:line:column    valid position with file name
line:column         valid position without file name
file                invalid position with file name
-                   invalid position without file name

type RelPos

RelPos indicates the relative position of token to the previous token.

type RelPos int
const (
    // NoRelPos indicates no relative position is specified.
    NoRelPos RelPos = iota

    // Elided indicates that the token for which this position is defined is
    // not rendered at all.
    Elided

    // NoSpace indicates there is no whitespace after this token.
    NoSpace

    // Blank means there is horizontal space after this token.
    Blank

    // Newline means there is a single newline after this token.
    Newline

    // NewSection means there are two or more newlines after this token.
    NewSection
)

func (RelPos) Pos

func (p RelPos) Pos() Pos

func (RelPos) String

func (p RelPos) String() string

type Token

Token is the set of lexical tokens of the CUE configuration language.

type Token int

The list of tokens.

const (
    // Special tokens
    ILLEGAL Token = iota
    EOF
    COMMENT
    ATTRIBUTE // @foo(bar,baz=4)

    // Identifiers and basic type literals
    // (these tokens stand for classes of literals)
    IDENT // main, _tmp
    INT   // 12_345Mi, 0700, 0xdeadbeef, 1.2M
    FLOAT // 123.45,
    // DURATION // 3m4s TODO
    STRING        // "abc"
    INTERPOLATION // a part of a template string, e.g. `"age: \(`
    BOTTOM        // _|_

    // Operators and delimiters
    ADD // +
    SUB // -
    MUL // *
    POW // ^
    QUO // /

    IQUO // quo
    IREM // rem
    IDIV // div
    IMOD // mod

    AND // &
    OR  // |

    LAND // &&
    LOR  // ||

    BIND  // =
    EQL   // ==
    LSS   // <
    GTR   // >
    NOT   // !
    ARROW // <-

    NEQ // !=
    LEQ // <=
    GEQ // >=

    MAT  // =~
    NMAT // !~

    LPAREN   // (
    LBRACK   // [
    LBRACE   // {
    COMMA    // ,
    PERIOD   // .
    ELLIPSIS // ...

    RPAREN    // )
    RBRACK    // ]
    RBRACE    // }
    SEMICOLON // ;
    COLON     // :
    OPTION    // ?

    IF
    FOR
    IN
    LET
    FUNC // experimental

    TRUE
    FALSE
    NULL
)

func Lookup

func Lookup(ident string) Token

Lookup maps an identifier to its keyword token or IDENT (if not a keyword).

func (Token) IsKeyword

func (tok Token) IsKeyword() bool

IsKeyword returns true for tokens corresponding to keywords; it returns false otherwise.

func (Token) IsLiteral

func (tok Token) IsLiteral() bool

IsLiteral returns true for tokens corresponding to identifiers and basic type literals; it returns false otherwise.

func (Token) IsOperator

func (tok Token) IsOperator() bool

IsOperator returns true for tokens corresponding to operators and delimiters; it returns false otherwise.

func (Token) Precedence

func (tok Token) Precedence() int

Precedence returns the operator precedence of the binary operator op. If op is not a binary operator, the result is LowestPrecedence.

func (Token) String

func (tok Token) String() string

String returns the string corresponding to the token tok. For operators, delimiters, and keywords the string is the actual token character sequence (e.g., for the token ADD, the string is "+"). For all other tokens the string corresponds to the token constant name (e.g. for the token IDENT, the string is "IDENT").