Begin overhauling semantic phases
This change further merges the new AST and MuPack/MuIL formats and
abstractions into the core of the compiler. A good amount of the old
code is gone now; I decided against ripping it all out in one fell
swoop so that I can methodically check that we are preserving all
relevant decisions and/or functionality we had in the old model.
The changes are too numerous to outline in this commit message,
however, here are the noteworthy ones:
* Split up the notion of symbols and tokens, resulting in:
- pkg/symbols for true compiler symbols (bound nodes)
- pkg/tokens for name-based tokens, identifiers, constants
* Several packages move underneath pkg/compiler:
- pkg/ast becomes pkg/compiler/ast
- pkg/errors becomes pkg/compiler/errors
- pkg/symbols becomes pkg/compiler/symbols
* pkg/ast/... becomes pkg/compiler/legacy/ast/...
* pkg/pack/ast becomes pkg/compiler/ast.
* pkg/options goes away, merged back into pkg/compiler.
* All binding functionality moves underneath a dedicated
package, pkg/compiler/binder. The legacy.go file contains
cruft that will eventually go away, while the other files
represent a halfway point between new and old, but are
expected to stay roughly in the current shape.
* All parsing functionality is moved underneath a new
pkg/compiler/metadata namespace, and we adopt new terminology
"metadata reading" since real parsing happens in the MetaMu
compilers. Hence, Parser has become metadata.Reader.
* In general phases of the compiler no longer share access to
the actual compiler.Compiler object. Instead, shared state is
moved to the core.Context object underneath pkg/compiler/core.
* Dependency resolution during binding has been rewritten to
the new model, including stashing bound package symbols in the
context object, and detecting import cycles.
* Compiler construction does not take a workspace object. Instead,
creation of a workspace is entirely hidden inside of the compiler's
constructor logic.
* There are three Compile* functions on the Compiler interface, to
support different styles of invoking compilation: Compile() auto-
detects a Mu package, based on the workspace; CompilePath(string)
loads the target as a Mu package and compiles it, regardless of
the workspace settings; and, CompilePackage(*pack.Package) will
compile a pre-loaded package AST, again regardless of workspace.
* Delete the _fe, _sema, and parsetree phases. They are no longer
relevant and the functionality is largely subsumed by the above.
...and so very much more. I'm surprised I ever got this to compile again!
2017-01-18 21:18:37 +01:00
|
|
|
// Copyright 2016 Marapongo, Inc. All rights reserved.
|
|
|
|
|
2017-01-28 00:42:39 +01:00
|
|
|
// Package tokens contains the core MuIL symbol and token types.
|
Begin overhauling semantic phases
This change further merges the new AST and MuPack/MuIL formats and
abstractions into the core of the compiler. A good amount of the old
code is gone now; I decided against ripping it all out in one fell
swoop so that I can methodically check that we are preserving all
relevant decisions and/or functionality we had in the old model.
The changes are too numerous to outline in this commit message,
however, here are the noteworthy ones:
* Split up the notion of symbols and tokens, resulting in:
- pkg/symbols for true compiler symbols (bound nodes)
- pkg/tokens for name-based tokens, identifiers, constants
* Several packages move underneath pkg/compiler:
- pkg/ast becomes pkg/compiler/ast
- pkg/errors becomes pkg/compiler/errors
- pkg/symbols becomes pkg/compiler/symbols
* pkg/ast/... becomes pkg/compiler/legacy/ast/...
* pkg/pack/ast becomes pkg/compiler/ast.
* pkg/options goes away, merged back into pkg/compiler.
* All binding functionality moves underneath a dedicated
package, pkg/compiler/binder. The legacy.go file contains
cruft that will eventually go away, while the other files
represent a halfway point between new and old, but are
expected to stay roughly in the current shape.
* All parsing functionality is moved underneath a new
pkg/compiler/metadata namespace, and we adopt new terminology
"metadata reading" since real parsing happens in the MetaMu
compilers. Hence, Parser has become metadata.Reader.
* In general phases of the compiler no longer share access to
the actual compiler.Compiler object. Instead, shared state is
moved to the core.Context object underneath pkg/compiler/core.
* Dependency resolution during binding has been rewritten to
the new model, including stashing bound package symbols in the
context object, and detecting import cycles.
* Compiler construction does not take a workspace object. Instead,
creation of a workspace is entirely hidden inside of the compiler's
constructor logic.
* There are three Compile* functions on the Compiler interface, to
support different styles of invoking compilation: Compile() auto-
detects a Mu package, based on the workspace; CompilePath(string)
loads the target as a Mu package and compiles it, regardless of
the workspace settings; and, CompilePackage(*pack.Package) will
compile a pre-loaded package AST, again regardless of workspace.
* Delete the _fe, _sema, and parsetree phases. They are no longer
relevant and the functionality is largely subsumed by the above.
...and so very much more. I'm surprised I ever got this to compile again!
2017-01-18 21:18:37 +01:00
|
|
|
package tokens
|
|
|
|
|
2017-01-21 18:08:35 +01:00
|
|
|
import (
|
|
|
|
"strings"
|
|
|
|
|
|
|
|
"github.com/marapongo/mu/pkg/util/contract"
|
|
|
|
)
|
|
|
|
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
// Token is a qualified name that is capable of resolving to a symbol entirely on its own. Most uses of tokens are
|
|
|
|
// typed based on the context, so that a subset of the token syntax is permissable (see the various typedefs below).
|
|
|
|
// However, in its full generality, a token can have a package part, a module part, a module-member part, and a
|
|
|
|
// class-member part. Obviously tokens that are meant to address just a module won't have the module-member part, and
|
|
|
|
// tokens addressing module members won't have the class-member part, etc.
|
|
|
|
//
|
|
|
|
// Token's grammar is as follows:
|
|
|
|
//
|
2017-01-23 23:48:55 +01:00
|
|
|
// Token = <Identifier> |
|
|
|
|
// <QualifiedToken> |
|
|
|
|
// <DecoratedType>
|
|
|
|
// Identifier = <Name>
|
|
|
|
// QualifiedToken = <PackageName> [ ":" <ModuleName> [ "/" <ModuleMemberName> [ "." <ClassMemberName> ] ] ]
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
// PackageName = <QName>
|
|
|
|
// ModuleName = <QName>
|
|
|
|
// ModuleMemberName = <Name>
|
|
|
|
// ClassMemberName = <Name>
|
|
|
|
//
|
2017-01-23 23:48:55 +01:00
|
|
|
// A token may be a simple identifier in the case that it refers to a built-in symbol, like a primitive type, or a
|
|
|
|
// variable in scope, rather than a qualified token that is to be bound to a symbol through package/module resolution.
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
//
|
|
|
|
// Notice that both package and module names may be qualified names (meaning they can have "/" delimiters; see QName's
|
|
|
|
// comments), and that module and class members must use unqualified, simple names (meaning they have no delimiters).
|
|
|
|
// The specialized token kinds differ only in what elements they require as part of the token string.
|
2017-01-23 23:48:55 +01:00
|
|
|
//
|
|
|
|
// Finally, a token may also be a decorated type. This is for built-in array, map, pointer, and function types:
|
|
|
|
//
|
|
|
|
// DecoratedType = "*" <Token> |
|
|
|
|
// "[]" <Token> |
|
|
|
|
// "map[" <Token> "]" <Token> |
|
|
|
|
// "(" [ <Token> [ "," <Token> ]* ] ")" <Token>?
|
|
|
|
//
|
|
|
|
// Notice that a recursive parsing process is required to extract elements from a <DecoratedType> token.
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
type Token string
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func (tok Token) String() string { return string(tok) }
|
|
|
|
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
const ModuleDelimiter string = ":" // the character following a package (before a module).
|
|
|
|
const ModuleMemberDelimiter string = "/" // the character following a module (before a module member).
|
|
|
|
const ClassMemberDelimiter string = "." // the character following a class name (before a class member).
|
|
|
|
|
2017-01-25 19:51:04 +01:00
|
|
|
func (tok Token) Simple() bool { return !tok.HasModule() }
|
2017-01-22 18:45:58 +01:00
|
|
|
func (tok Token) HasModule() bool { return strings.Index(string(tok), ModuleDelimiter) != -1 }
|
|
|
|
func (tok Token) HasModuleMember() bool {
|
|
|
|
return strings.Index(string(tok), ModuleMemberDelimiter) != -1
|
|
|
|
}
|
|
|
|
func (tok Token) HasClassMember() bool { return strings.Index(string(tok), ClassMemberDelimiter) != -1 }
|
|
|
|
|
2017-01-25 19:51:04 +01:00
|
|
|
// Name returns the Token as a Name (and assumes it is a legal one).
|
|
|
|
func (tok Token) Name() Name {
|
|
|
|
contract.Requiref(tok.Simple(), "tok", "Simple")
|
|
|
|
contract.Requiref(IsName(tok.String()), "tok", "IsName")
|
|
|
|
return Name(tok.String())
|
|
|
|
}
|
|
|
|
|
2017-01-27 01:49:38 +01:00
|
|
|
// Names returns as many names as the Token has, each in a return slot, or "" for those that don't exist.
|
|
|
|
func (tok Token) Names() (PackageName, ModuleName, ModuleMemberName, ClassMemberName) {
|
|
|
|
// Check the token for the most specific to least specific cases.
|
2017-01-25 18:29:34 +01:00
|
|
|
if tok.HasClassMember() {
|
|
|
|
clmtok := ClassMember(tok)
|
2017-01-27 01:49:38 +01:00
|
|
|
return clmtok.Package().Name(), clmtok.Module().Name(),
|
|
|
|
ModuleMemberName(clmtok.Class().Name()), clmtok.Name()
|
2017-01-25 18:29:34 +01:00
|
|
|
} else if tok.HasModuleMember() {
|
|
|
|
memtok := ModuleMember(tok)
|
2017-01-27 01:49:38 +01:00
|
|
|
return memtok.Package().Name(), memtok.Module().Name(), memtok.Name(), ""
|
2017-01-25 18:29:34 +01:00
|
|
|
} else if tok.HasModule() {
|
|
|
|
modtok := Module(tok)
|
2017-01-27 01:49:38 +01:00
|
|
|
return modtok.Package().Name(), modtok.Name(), "", ""
|
|
|
|
}
|
2017-01-28 00:42:39 +01:00
|
|
|
return Package(tok).Name(), "", "", ""
|
2017-01-27 01:49:38 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
// Tokens returns as many tokens as the Token has, each in a return slot, or "" for those that don't exist.
|
|
|
|
func (tok Token) Tokens() (Package, Module, ModuleMember, ClassMember) {
|
|
|
|
// Check the token for the most specific to least specific cases.
|
|
|
|
if tok.HasClassMember() {
|
|
|
|
clmtok := ClassMember(tok)
|
|
|
|
return clmtok.Package(), clmtok.Module(), ModuleMember(clmtok.Class()), clmtok
|
|
|
|
} else if tok.HasModuleMember() {
|
|
|
|
memtok := ModuleMember(tok)
|
|
|
|
return memtok.Package(), memtok.Module(), memtok, ""
|
|
|
|
} else if tok.HasModule() {
|
|
|
|
modtok := Module(tok)
|
|
|
|
return modtok.Package(), modtok, "", ""
|
2017-01-25 18:29:34 +01:00
|
|
|
}
|
2017-01-28 00:42:39 +01:00
|
|
|
return Package(tok), "", "", ""
|
2017-01-25 18:29:34 +01:00
|
|
|
}
|
|
|
|
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
// Package is a token representing just a package. It uses a much simpler grammar:
|
|
|
|
// Package = <PackageName>
|
|
|
|
// Note that a package name of "." means "current package", to simplify emission and lookups.
|
|
|
|
type Package Token
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func NewPackageToken(nm PackageName) Package {
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(IsQName(string(nm)), "Package name '%v' is not a legal qualified name")
|
2017-01-21 20:04:03 +01:00
|
|
|
return Package(nm)
|
|
|
|
}
|
|
|
|
|
2017-01-21 18:08:35 +01:00
|
|
|
func (tok Package) Name() PackageName {
|
|
|
|
return PackageName(tok)
|
|
|
|
}
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func (tok Package) String() string { return string(tok) }
|
|
|
|
|
|
|
|
// PackageName is a qualified name referring to an imported package.
|
|
|
|
type PackageName QName
|
|
|
|
|
|
|
|
func (nm PackageName) String() string { return string(nm) }
|
|
|
|
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
// Module is a token representing a module. It uses the following subset of the token grammar:
|
|
|
|
// Module = <Package> ":" <ModuleName>
|
|
|
|
// Note that a module name of "." means "current module", to simplify emission and lookups.
|
|
|
|
type Module Token
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func NewModuleToken(pkg Package, nm ModuleName) Module {
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(IsQName(string(nm)), "Package '%v' module name '%v' is not a legal qualified name", pkg, nm)
|
2017-01-21 20:04:03 +01:00
|
|
|
return Module(string(pkg) + ModuleDelimiter + string(nm))
|
|
|
|
}
|
|
|
|
|
2017-01-21 18:08:35 +01:00
|
|
|
func (tok Module) Package() Package {
|
|
|
|
ix := strings.LastIndex(string(tok), ModuleDelimiter)
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(ix != -1, "Module token '%v' missing module delimiter", tok)
|
2017-01-21 18:08:35 +01:00
|
|
|
return Package(string(tok)[:ix])
|
|
|
|
}
|
|
|
|
|
|
|
|
func (tok Module) Name() ModuleName {
|
|
|
|
ix := strings.LastIndex(string(tok), ModuleDelimiter)
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(ix != -1, "Module token '%v' missing module delimiter", tok)
|
2017-01-21 18:08:35 +01:00
|
|
|
return ModuleName(string(tok)[ix+1:])
|
|
|
|
}
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func (tok Module) String() string { return string(tok) }
|
|
|
|
|
|
|
|
// ModuleName is a qualified name referring to an imported module from a package.
|
|
|
|
type ModuleName QName
|
|
|
|
|
|
|
|
func (nm ModuleName) String() string { return string(nm) }
|
|
|
|
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
// ModuleMember is a token representing a module's member. It uses the following grammar. Note that this is not
|
|
|
|
// ambiguous because member names cannot contain slashes, and so the "last" slash in a name delimits the member:
|
|
|
|
// ModuleMember = <Module> "/" <ModuleMemberName>
|
|
|
|
type ModuleMember Token
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func NewModuleMemberToken(mod Module, nm ModuleMemberName) ModuleMember {
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(IsName(string(nm)), "Module '%v' member name '%v' is not a legal name", mod, nm)
|
2017-01-21 20:04:03 +01:00
|
|
|
return ModuleMember(string(mod) + ModuleMemberDelimiter + string(nm))
|
|
|
|
}
|
|
|
|
|
2017-01-21 18:08:35 +01:00
|
|
|
func (tok ModuleMember) Package() Package {
|
|
|
|
return tok.Module().Package()
|
|
|
|
}
|
|
|
|
|
|
|
|
func (tok ModuleMember) Module() Module {
|
|
|
|
ix := strings.LastIndex(string(tok), ModuleMemberDelimiter)
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(ix != -1, "Module member token '%v' missing module member delimiter", tok)
|
2017-01-21 18:08:35 +01:00
|
|
|
return Module(string(tok)[:ix])
|
|
|
|
}
|
|
|
|
|
|
|
|
func (tok ModuleMember) Name() ModuleMemberName {
|
|
|
|
ix := strings.LastIndex(string(tok), ModuleMemberDelimiter)
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(ix != -1, "Module member token '%v' missing module member delimiter", tok)
|
2017-01-21 18:08:35 +01:00
|
|
|
return ModuleMemberName(string(tok)[ix+1:])
|
|
|
|
}
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func (tok ModuleMember) String() string { return string(tok) }
|
|
|
|
|
|
|
|
// ModuleMemberName is a simple name representing the module member's identifier.
|
|
|
|
type ModuleMemberName Name
|
|
|
|
|
|
|
|
func (nm ModuleMemberName) String() string { return string(nm) }
|
|
|
|
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
// ClassMember is a token representing a class's member. It uses the following grammar. Unlike ModuleMember, this
|
|
|
|
// cannot use a slash for delimiting names, because we use often ClassMember and ModuleMember interchangably:
|
2017-01-21 18:08:35 +01:00
|
|
|
// ClassMember = <ModuleMember> "." <ClassMemberName>
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
type ClassMember Token
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func NewClassMemberToken(class Type, nm ClassMemberName) ClassMember {
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(IsName(string(nm)), "Class '%v' member name '%v' is not a legal name", class, nm)
|
2017-01-21 20:04:03 +01:00
|
|
|
return ClassMember(string(class) + ClassMemberDelimiter + string(nm))
|
|
|
|
}
|
|
|
|
|
2017-01-21 18:08:35 +01:00
|
|
|
func (tok ClassMember) Package() Package {
|
|
|
|
return tok.Module().Package()
|
|
|
|
}
|
|
|
|
|
|
|
|
func (tok ClassMember) Module() Module {
|
|
|
|
return tok.Class().Module()
|
|
|
|
}
|
|
|
|
|
|
|
|
func (tok ClassMember) Class() Type {
|
|
|
|
ix := strings.LastIndex(string(tok), ClassMemberDelimiter)
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(ix != -1, "Class member token '%v' missing class member delimiter", tok)
|
2017-01-21 18:08:35 +01:00
|
|
|
return Type(string(tok)[:ix])
|
|
|
|
}
|
|
|
|
|
|
|
|
func (tok ClassMember) Name() ClassMemberName {
|
|
|
|
ix := strings.LastIndex(string(tok), ClassMemberDelimiter)
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(ix != -1, "Class member token '%v' missing class member delimiter", tok)
|
2017-01-21 18:08:35 +01:00
|
|
|
return ClassMemberName(string(tok)[ix+1:])
|
|
|
|
}
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func (tok ClassMember) String() string { return string(tok) }
|
|
|
|
|
|
|
|
// ClassMemberName is a simple name representing the class member's identifier.
|
|
|
|
type ClassMemberName Name
|
|
|
|
|
2017-01-25 23:08:49 +01:00
|
|
|
func (nm ClassMemberName) Name() Name { return Name(nm) }
|
2017-01-21 21:25:59 +01:00
|
|
|
func (nm ClassMemberName) String() string { return string(nm) }
|
|
|
|
|
2017-01-23 23:48:55 +01:00
|
|
|
// Type is a token representing a type. It is either a primitive type name, reference to a module class, or decorated:
|
|
|
|
// Type = <Name> | <ModuleMember> | <DecoratedType>
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
type Type Token
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func NewTypeToken(mod Module, nm TypeName) Type {
|
2017-02-08 18:12:09 +01:00
|
|
|
contract.Assertf(IsName(string(nm)), "Module '%v' type name '%v' is not a legal name", mod, nm)
|
2017-01-21 20:04:03 +01:00
|
|
|
return Type(string(mod) + ModuleMemberDelimiter + string(nm))
|
|
|
|
}
|
|
|
|
|
2017-01-21 18:08:35 +01:00
|
|
|
func (tok Type) Package() Package {
|
2017-01-23 23:48:55 +01:00
|
|
|
if tok.Primitive() || tok.Decorated() {
|
2017-01-21 18:08:35 +01:00
|
|
|
return Package("")
|
|
|
|
}
|
2017-01-28 00:42:39 +01:00
|
|
|
return ModuleMember(tok).Package()
|
2017-01-21 18:08:35 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
func (tok Type) Module() Module {
|
2017-01-23 23:48:55 +01:00
|
|
|
if tok.Primitive() || tok.Decorated() {
|
2017-01-21 18:08:35 +01:00
|
|
|
return Module("")
|
|
|
|
}
|
2017-01-28 00:42:39 +01:00
|
|
|
return ModuleMember(tok).Module()
|
2017-01-21 18:08:35 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
func (tok Type) Name() TypeName {
|
2017-01-23 23:48:55 +01:00
|
|
|
if tok.Primitive() || tok.Decorated() {
|
2017-01-21 18:08:35 +01:00
|
|
|
return TypeName(tok)
|
|
|
|
}
|
2017-01-28 00:42:39 +01:00
|
|
|
return TypeName(ModuleMember(tok).Name())
|
2017-01-21 18:08:35 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
func (tok Type) Member() ModuleMember {
|
|
|
|
return ModuleMember(tok)
|
|
|
|
}
|
|
|
|
|
2017-01-23 23:48:55 +01:00
|
|
|
// Decorated indicates whether this token represents a decorated type.
|
|
|
|
func (tok Type) Decorated() bool {
|
|
|
|
return tok.Pointer() || tok.Array() || tok.Map() || tok.Function()
|
|
|
|
}
|
|
|
|
|
|
|
|
func (tok Type) Pointer() bool { return IsPointerType(tok) }
|
|
|
|
func (tok Type) Array() bool { return IsArrayType(tok) }
|
|
|
|
func (tok Type) Map() bool { return IsMapType(tok) }
|
|
|
|
func (tok Type) Function() bool { return IsFunctionType(tok) }
|
|
|
|
|
2017-01-21 18:08:35 +01:00
|
|
|
// Primitive indicates whether this type is a primitive type name (i.e., not qualified with a module, etc).
|
|
|
|
func (tok Type) Primitive() bool {
|
2017-01-23 23:48:55 +01:00
|
|
|
return !tok.Decorated() && !Token(tok).HasModule()
|
2017-01-21 18:08:35 +01:00
|
|
|
}
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func (tok Type) String() string { return string(tok) }
|
|
|
|
|
|
|
|
// TypeName is a simple name representing the type's name, without any package/module qualifiers.
|
|
|
|
type TypeName Name
|
|
|
|
|
|
|
|
func (nm TypeName) String() string { return string(nm) }
|
|
|
|
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
// Variable is a token representing a variable (module property, class property, or local variable (including
|
|
|
|
// parameters)). It can be a simple name for the local cases, or a true token for others:
|
|
|
|
// Variable = <Name> | <ModuleMember> | <ClassMember>
|
|
|
|
type Variable Token
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func (tok Variable) String() string { return string(tok) }
|
|
|
|
|
Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise. In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure. As such, names are much simpler than tokens.
As explained in the comments, tokens.Names are simple identifiers:
Name = [A-Za-z_][A-Za-z0-9_]*
and tokens.QNames are fully qualified identifiers delimited by "/":
QName = [ <Name> "/" ]* <Name>
The legal grammar for a token depends on the subset of symbols that
token is meant to represent. However, the most general case, that
accepts all specializations of tokens, is roughly as follows:
Token = <Name> |
<PackageName>
[ ":" <ModuleName>
[ "/" <ModuleMemberName>
[ "." <Class MemberName> ]
]
]
where:
PackageName = <QName>
ModuleName = <QName>
ModuleMemberName = <Name>
ClassMemberName = <Name>
Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-20 02:57:20 +01:00
|
|
|
// Function is a token representing a variable (module method or class method). Its grammar is as follows:
|
|
|
|
// Variable = <ModuleMember> | <ClassMember>
|
|
|
|
type Function Token
|
|
|
|
|
2017-01-21 21:25:59 +01:00
|
|
|
func (tok Function) String() string { return string(tok) }
|