maxmustermann/TypeScript
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
TypeScript/tests/cases/unittests/services/colorization.ts

449 lines
20 KiB
TypeScript
Raw Normal View History

Fix runtests for unittests
2014-10-17 03:13:26 +02:00
/// <reference path="..\..\..\..\src\harness\external\mocha.d.ts" />
/// <reference path="..\..\..\..\src\harness\harnessLanguageService.ts" />
clean up tests
2014-08-09 01:45:10 +02:00
interface ClassificationEntry {
value: any;
Move unit tests to use the new adaptors
2015-02-04 02:28:33 +01:00
classification: ts.TokenClass;
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
position?: number;
clean up tests
2014-08-09 01:45:10 +02:00
}
Enable clorizer unit tests
2014-08-08 07:21:12 +02:00
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
describe("Colorization", function () {
Respond to code review comments
2015-02-09 18:19:50 +01:00
// Use the shim adapter to ensure test coverage of the shim layer for the classifier
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
const languageServiceAdapter = new Harness.LanguageService.ShimLanguageServiceAdapter(/*preprocessToResolve*/ false);
const classifier = languageServiceAdapter.getClassifier();
Move unit tests to use the new adaptors
2015-02-04 02:28:33 +01:00
spelling fixes for src Fixes: * actual * algorithm * analyze * applies * collapse * contrast * definition * diagnostic * different * displayed * display * documentation * finite * function * highlight * initialize * intentional * interface * invariants * items * keystroke * language * literal * original * output * position * receive * recorder * response * sequence * simplicity * statement Changing a \ to a /
2016-02-25 21:08:44 +01:00
function getEntryAtPosition(result: ts.ClassificationResult, position: number) {
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
let entryPosition = 0;
for (let i = 0, n = result.entries.length; i < n; i++) {
const entry = result.entries[i];
Move unit tests to use the new adaptors
2015-02-04 02:28:33 +01:00
if (entryPosition === position) {
return entry;
}
entryPosition += entry.length;
clean up tests
2014-08-09 01:45:10 +02:00
}
return undefined;
}
Add snapshot of compiler sources
2014-07-13 01:04:16 +02:00
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
function punctuation(text: string, position?: number) { return createClassification(text, ts.TokenClass.Punctuation, position); }
function keyword(text: string, position?: number) { return createClassification(text, ts.TokenClass.Keyword, position); }
function operator(text: string, position?: number) { return createClassification(text, ts.TokenClass.Operator, position); }
function comment(text: string, position?: number) { return createClassification(text, ts.TokenClass.Comment, position); }
function whitespace(text: string, position?: number) { return createClassification(text, ts.TokenClass.Whitespace, position); }
function identifier(text: string, position?: number) { return createClassification(text, ts.TokenClass.Identifier, position); }
function numberLiteral(text: string, position?: number) { return createClassification(text, ts.TokenClass.NumberLiteral, position); }
function stringLiteral(text: string, position?: number) { return createClassification(text, ts.TokenClass.StringLiteral, position); }
function finalEndOfLineState(value: number): ClassificationEntry { return { value: value, classification: undefined, position: 0 }; }
function createClassification(text: string, tokenClass: ts.TokenClass, position?: number): ClassificationEntry {
return {
value: text,
classification: tokenClass,
position: position,
};
}
clean up tests
2014-08-09 01:45:10 +02:00
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
function testLexicalClassification(text: string, initialEndOfLineState: ts.EndOfLineState, ...expectedEntries: ClassificationEntry[]): void {
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
const result = classifier.getClassificationsForLine(text, initialEndOfLineState, /*syntacticClassifierAbsent*/ false);
clean up tests
2014-08-09 01:45:10 +02:00
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
for (let i = 0, n = expectedEntries.length; i < n; i++) {
const expectedEntry = expectedEntries[i];
clean up tests
2014-08-09 01:45:10 +02:00
Move unit tests to use the new adaptors
2015-02-04 02:28:33 +01:00
if (expectedEntry.classification === undefined) {
assert.equal(result.finalLexState, expectedEntry.value, "final endOfLineState does not match expected.");
clean up tests
2014-08-09 01:45:10 +02:00
}
else {
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
const actualEntryPosition = expectedEntry.position !== undefined ? expectedEntry.position : text.indexOf(expectedEntry.value);
clean up tests
2014-08-09 01:45:10 +02:00
assert(actualEntryPosition >= 0, "token: '" + expectedEntry.value + "' does not exit in text: '" + text + "'.");
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
const actualEntry = getEntryAtPosition(result, actualEntryPosition);
clean up tests
2014-08-09 01:45:10 +02:00
assert(actualEntry, "Could not find classification entry for '" + expectedEntry.value + "' at position: " + actualEntryPosition);
Move unit tests to use the new adaptors
2015-02-04 02:28:33 +01:00
assert.equal(actualEntry.classification, expectedEntry.classification, "Classification class does not match expected. Expected: " + ts.TokenClass[expectedEntry.classification] + ", Actual: " + ts.TokenClass[actualEntry.classification]);
Fixed up colorization tests.
2014-09-23 21:23:33 +02:00
assert.equal(actualEntry.length, expectedEntry.value.length, "Classification length does not match expected. Expected: " + ts.TokenClass[expectedEntry.value.length] + ", Actual: " + ts.TokenClass[actualEntry.length]);
clean up tests
2014-08-09 01:45:10 +02:00
}
}
}
Add snapshot of compiler sources
2014-07-13 01:04:16 +02:00
clean up tests
2014-08-09 01:45:10 +02:00
describe("test getClassifications", function () {
it("Returns correct token classes", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("var x: string = \"foo\"; //Hello",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
clean up tests
2014-08-09 01:45:10 +02:00
keyword("var"),
whitespace(" "),
identifier("x"),
punctuation(":"),
keyword("string"),
operator("="),
stringLiteral("\"foo\""),
comment("//Hello"),
punctuation(";"));
Add snapshot of compiler sources
2014-07-13 01:04:16 +02:00
});
Changed wording for unit tests.
2014-09-23 23:14:27 +02:00
it("correctly classifies a comment after a divide operator", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("1 / 2 // comment",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
clean up tests
2014-08-09 01:45:10 +02:00
numberLiteral("1"),
whitespace(" "),
operator("/"),
numberLiteral("2"),
comment("// comment"));
Add snapshot of compiler sources
2014-07-13 01:04:16 +02:00
});
Changed wording for unit tests.
2014-09-23 23:14:27 +02:00
it("correctly classifies a literal after a divide operator", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("1 / 2, 3 / 4",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
clean up tests
2014-08-09 01:45:10 +02:00
numberLiteral("1"),
whitespace(" "),
operator("/"),
numberLiteral("2"),
numberLiteral("3"),
numberLiteral("4"),
operator(","));
Add snapshot of compiler sources
2014-07-13 01:04:16 +02:00
});
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
it("correctly classifies a multi-line string with one backslash", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("'line1\\",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
clean up tests
2014-08-09 01:45:10 +02:00
stringLiteral("'line1\\"),
finalEndOfLineState(ts.EndOfLineState.InSingleQuoteStringLiteral));
Add snapshot of compiler sources
2014-07-13 01:04:16 +02:00
});
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
it("correctly classifies a multi-line string with three backslashes", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("'line1\\\\\\",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
stringLiteral("'line1\\\\\\"),
finalEndOfLineState(ts.EndOfLineState.InSingleQuoteStringLiteral));
});
it("correctly classifies an unterminated single-line string with no backslashes", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("'line1",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
stringLiteral("'line1"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
});
it("correctly classifies an unterminated single-line string with two backslashes", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("'line1\\\\",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
stringLiteral("'line1\\\\"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
});
it("correctly classifies an unterminated single-line string with four backslashes", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("'line1\\\\\\\\",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
stringLiteral("'line1\\\\\\\\"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
});
it("correctly classifies the continuing line of a multi-line string ending in one backslash", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("\\",
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
ts.EndOfLineState.InDoubleQuoteStringLiteral,
stringLiteral("\\"),
finalEndOfLineState(ts.EndOfLineState.InDoubleQuoteStringLiteral));
});
it("correctly classifies the continuing line of a multi-line string ending in three backslashes", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("\\",
clean up tests
2014-08-09 01:45:10 +02:00
ts.EndOfLineState.InDoubleQuoteStringLiteral,
stringLiteral("\\"),
finalEndOfLineState(ts.EndOfLineState.InDoubleQuoteStringLiteral));
Add snapshot of compiler sources
2014-07-13 01:04:16 +02:00
});
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
it("correctly classifies the last line of an unterminated multi-line string ending in no backslashes", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification(" ",
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
ts.EndOfLineState.InDoubleQuoteStringLiteral,
stringLiteral(" "),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
});
it("correctly classifies the last line of an unterminated multi-line string ending in two backslashes", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("\\\\",
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
ts.EndOfLineState.InDoubleQuoteStringLiteral,
stringLiteral("\\\\"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
});
it("correctly classifies the last line of an unterminated multi-line string ending in four backslashes", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("\\\\\\\\",
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
ts.EndOfLineState.InDoubleQuoteStringLiteral,
stringLiteral("\\\\\\\\"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Fixed issue where classifier didn't check for backslash-newline.
2014-11-27 01:32:30 +01:00
});
Changed wording for unit tests.
2014-09-23 23:14:27 +02:00
it("correctly classifies the last line of a multi-line string", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("'",
clean up tests
2014-08-09 01:45:10 +02:00
ts.EndOfLineState.InSingleQuoteStringLiteral,
stringLiteral("'"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Add snapshot of compiler sources
2014-07-13 01:04:16 +02:00
});
add unterminated multiline comment classification tests
2014-08-08 08:51:15 +02:00
Changed wording for unit tests.
2014-09-23 23:14:27 +02:00
it("correctly classifies an unterminated multiline comment", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("/*",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
clean up tests
2014-08-09 01:45:10 +02:00
comment("/*"),
finalEndOfLineState(ts.EndOfLineState.InMultiLineCommentTrivia));
add unterminated multiline comment classification tests
2014-08-08 08:51:15 +02:00
});
Added test cases for multiline comments in line classification.
2014-09-23 23:13:43 +02:00
it("correctly classifies the termination of a multiline comment", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification(" */ ",
Added test cases for multiline comments in line classification.
2014-09-23 23:13:43 +02:00
ts.EndOfLineState.InMultiLineCommentTrivia,
comment(" */"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Added test cases for multiline comments in line classification.
2014-09-23 23:13:43 +02:00
});
it("correctly classifies the continuation of a multiline comment", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("LOREM IPSUM DOLOR ",
Added test cases for multiline comments in line classification.
2014-09-23 23:13:43 +02:00
ts.EndOfLineState.InMultiLineCommentTrivia,
comment("LOREM IPSUM DOLOR "),
finalEndOfLineState(ts.EndOfLineState.InMultiLineCommentTrivia));
});
it("correctly classifies an unterminated multiline comment on a line ending in '/*/'", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification(" /*/",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Added test cases for multiline comments in line classification.
2014-09-23 23:13:43 +02:00
comment("/*/"),
finalEndOfLineState(ts.EndOfLineState.InMultiLineCommentTrivia));
});
add unterminated multiline comment classification tests
2014-08-08 08:51:15 +02:00
Changed wording for unit tests.
2014-09-23 23:14:27 +02:00
it("correctly classifies an unterminated multiline comment with trailing space", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("/* ",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
clean up tests
2014-08-09 01:45:10 +02:00
comment("/* "),
finalEndOfLineState(ts.EndOfLineState.InMultiLineCommentTrivia));
add unterminated multiline comment classification tests
2014-08-08 08:51:15 +02:00
});
Add some huristic optimization to not colorize a keyword if precceded by a dot or a keyword. this should handel cases for "a.var" or "module string { }"
2014-08-08 19:37:10 +02:00
Changed wording for unit tests.
2014-09-23 23:14:27 +02:00
it("correctly classifies a keyword after a dot", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("a.var",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
clean up tests
2014-08-09 01:45:10 +02:00
identifier("var"));
Add some huristic optimization to not colorize a keyword if precceded by a dot or a keyword. this should handel cases for "a.var" or "module string { }"
2014-08-08 19:37:10 +02:00
});
Limited identifying identifiers after dots to keywords.
2014-09-26 18:46:14 +02:00
it("correctly classifies a string literal after a dot", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("a.\"var\"",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Limited identifying identifiers after dots to keywords.
2014-09-26 18:46:14 +02:00
stringLiteral("\"var\""));
});
Fix issue where trivia is incorrectly identified as an identifier following a dot. Fixes #753
2014-09-26 07:28:02 +02:00
it("correctly classifies a keyword after a dot separated by comment trivia", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("a./*hello world*/ var",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Fix issue where trivia is incorrectly identified as an identifier following a dot. Fixes #753
2014-09-26 07:28:02 +02:00
identifier("a"),
punctuation("."),
comment("/*hello world*/"),
identifier("var"));
});
Added unit test for property access colorization.
2014-09-23 22:59:47 +02:00
it("classifies a property access with whitespace around the dot", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification(" x .\tfoo ()",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Added unit test for property access colorization.
2014-09-23 22:59:47 +02:00
identifier("x"),
identifier("foo"));
});
it("classifies a keyword after a dot on previous line", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("var",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Do not colorize a keyword following an identifier.<newLine> as it is likely an error scenario as the user is typing
2014-09-13 01:49:45 +02:00
keyword("var"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Add some huristic optimization to not colorize a keyword if precceded by a dot or a keyword. this should handel cases for "a.var" or "module string { }"
2014-08-08 19:37:10 +02:00
});
Tweak classification so it does not classify things as keywords that could not actually be keywords according to the grammar.
2014-09-30 23:35:07 +02:00
it("classifies multiple keywords properly", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("public static",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Tweak classification so it does not classify things as keywords that could not actually be keywords according to the grammar.
2014-09-30 23:35:07 +02:00
keyword("public"),
keyword("static"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Tweak classification so it does not classify things as keywords that could not actually be keywords according to the grammar.
2014-09-30 23:35:07 +02:00
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("public var",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Tweak classification so it does not classify things as keywords that could not actually be keywords according to the grammar.
2014-09-30 23:35:07 +02:00
keyword("public"),
identifier("var"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Tweak classification so it does not classify things as keywords that could not actually be keywords according to the grammar.
2014-09-30 23:35:07 +02:00
});
Add some heuristics in the lexical classifier to make it play better with the syntactic classifier when classifying expressions involving generics.
2014-10-03 10:54:43 +02:00
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
it("classifies a single line no substitution template string correctly", () => {
testLexicalClassification("`number number public string`",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
stringLiteral("`number number public string`"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
});
it("classifies substitution parts of a template string correctly", () => {
testLexicalClassification("`number '${ 1 + 1 }' string '${ 'hello' }'`",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
stringLiteral("`number '${"),
numberLiteral("1"),
operator("+"),
numberLiteral("1"),
stringLiteral("}' string '${"),
stringLiteral("'hello'"),
stringLiteral("}'`"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
});
it("classifies an unterminated no substitution template string correctly", () => {
testLexicalClassification("`hello world",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
stringLiteral("`hello world"),
finalEndOfLineState(ts.EndOfLineState.InTemplateHeadOrNoSubstitutionTemplate));
});
it("classifies the entire line of an unterminated multiline no-substitution/head template", () => {
testLexicalClassification("...",
ts.EndOfLineState.InTemplateHeadOrNoSubstitutionTemplate,
stringLiteral("..."),
finalEndOfLineState(ts.EndOfLineState.InTemplateHeadOrNoSubstitutionTemplate));
});
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
it("classifies the entire line of an unterminated multiline template middle/end", () => {
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
testLexicalClassification("...",
ts.EndOfLineState.InTemplateMiddleOrTail,
stringLiteral("..."),
finalEndOfLineState(ts.EndOfLineState.InTemplateMiddleOrTail));
});
it("classifies a termination of a multiline template head", () => {
testLexicalClassification("...${",
ts.EndOfLineState.InTemplateHeadOrNoSubstitutionTemplate,
stringLiteral("...${"),
finalEndOfLineState(ts.EndOfLineState.InTemplateSubstitutionPosition));
});
it("classifies the termination of a multiline no substitution template", () => {
testLexicalClassification("...`",
ts.EndOfLineState.InTemplateHeadOrNoSubstitutionTemplate,
stringLiteral("...`"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
});
it("classifies the substitution parts and middle/tail of a multiline template string", () => {
testLexicalClassification("${ 1 + 1 }...`",
ts.EndOfLineState.InTemplateHeadOrNoSubstitutionTemplate,
stringLiteral("${"),
numberLiteral("1"),
operator("+"),
numberLiteral("1"),
stringLiteral("}...`"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
});
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
it("classifies a template middle and propagates the end of line state", () => {
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
testLexicalClassification("${ 1 + 1 }...`",
ts.EndOfLineState.InTemplateHeadOrNoSubstitutionTemplate,
stringLiteral("${"),
numberLiteral("1"),
operator("+"),
numberLiteral("1"),
stringLiteral("}...`"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
});
it("classifies substitution expressions with curly braces appropriately", () => {
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
let pos = 0;
let lastLength = 0;
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
testLexicalClassification("...${ () => { } } ${ { x: `1` } }...`",
ts.EndOfLineState.InTemplateHeadOrNoSubstitutionTemplate,
stringLiteral(track("...${"), pos),
punctuation(track(" ", "("), pos),
punctuation(track(")"), pos),
punctuation(track(" ", "=>"), pos),
punctuation(track(" ", "{"), pos),
punctuation(track(" ", "}"), pos),
stringLiteral(track(" ", "} ${"), pos),
punctuation(track(" ", "{"), pos),
identifier(track(" ", "x"), pos),
punctuation(track(":"), pos),
stringLiteral(track(" ", "`1`"), pos),
punctuation(track(" ", "}"), pos),
stringLiteral(track(" ", "}...`"), pos),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
// Adjusts 'pos' by accounting for the length of each portion of the string,
// but only return the last given string
function track(...vals: string[]): string {
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
for (let i = 0, n = vals.length; i < n; i++) {
Added tests, fixed order of emptying templateStack, unconditionally perform template classification.
2015-01-27 00:41:01 +01:00
pos += lastLength;
lastLength = vals[i].length;
}
return ts.lastOrUndefined(vals);
}
});
Add some heuristics in the lexical classifier to make it play better with the syntactic classifier when classifying expressions involving generics.
2014-10-03 10:54:43 +02:00
it("classifies partially written generics correctly.", function () {
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("Foo<number",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Add some heuristics in the lexical classifier to make it play better with the syntactic classifier when classifying expressions involving generics.
2014-10-03 10:54:43 +02:00
identifier("Foo"),
operator("<"),
identifier("number"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Add some heuristics in the lexical classifier to make it play better with the syntactic classifier when classifying expressions involving generics.
2014-10-03 10:54:43 +02:00
// Looks like a cast, should get classified as a keyword.
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("<number",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Add some heuristics in the lexical classifier to make it play better with the syntactic classifier when classifying expressions involving generics.
2014-10-03 10:54:43 +02:00
operator("<"),
keyword("number"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Add some heuristics in the lexical classifier to make it play better with the syntactic classifier when classifying expressions involving generics.
2014-10-03 10:54:43 +02:00
// handle nesting properly.
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification("Foo<Foo,Foo<number",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Add some heuristics in the lexical classifier to make it play better with the syntactic classifier when classifying expressions involving generics.
2014-10-03 10:54:43 +02:00
identifier("Foo"),
operator("<"),
identifier("Foo"),
Fix colorization unittest
2014-10-20 22:01:21 +02:00
operator(","),
Add some heuristics in the lexical classifier to make it play better with the syntactic classifier when classifying expressions involving generics.
2014-10-03 10:54:43 +02:00
identifier("Foo"),
operator("<"),
identifier("number"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Adding classification test.
2014-12-12 02:04:21 +01:00
});
Add some heuristics in the lexical classifier to make it play better with the syntactic classifier when classifying expressions involving generics.
2014-10-03 10:54:43 +02:00
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
it("LexicallyClassifiesConflictTokens", () => {
Adding classification test.
2014-12-12 02:04:21 +01:00
// Test conflict markers.
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification(
Adding classification test.
2014-12-12 02:04:21 +01:00
"class C {\r\n\
<<<<<<< HEAD\r\n\
v = 1;\r\n\
=======\r\n\
v = 2;\r\n\
>>>>>>> Branch - a\r\n\
}",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Adding classification test.
2014-12-12 02:04:21 +01:00
keyword("class"),
identifier("C"),
punctuation("{"),
comment("<<<<<<< HEAD"),
identifier("v"),
operator("="),
numberLiteral("1"),
punctuation(";"),
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
comment("=======\r\n v = 2;\r\n"),
Adding classification test.
2014-12-12 02:04:21 +01:00
comment(">>>>>>> Branch - a"),
punctuation("}"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
testLexicalClassification(
"<<<<<<< HEAD\r\n\
class C { }\r\n\
=======\r\n\
class D { }\r\n\
>>>>>>> Branch - a\r\n",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
Provide better error recovery when we encounter merge markers in the source. Previously we would just treat each merge marker as trivia and then continue scanning and parsing like normal. This worked well in some scenarios, but fell down in others like: ``` class C { public foo() { <<<<<<< HEAD this.bar(); } ======= this.baz(); } >>>>>>> Branch public bar() { } } ``` The problem stems from the previous approach trying to incorporate both branches of the merge into the final tree. In a case like this, that approach breaks down entirely. The the parser ends up seeing the close curly in both included sections, and it considers the class finished. Then, it starts erroring when it encounters "public bar()". The fix is to only incorporate one of these sections into the tree. Specifically, we only include the first section. The second sectoin is treated like trivia and does not affect the parse at all. To make the experience more pleasant we do *lexically* classify the second section. That way it does not appear as just plain black text in the editor. Instead, it will have appropriate lexicla classifications for keywords, literals, comments, operators, punctuation, etc. However, any syntactic or semantic feature will not work in the second block due to this being trivia as far as any feature is concerned. This experience is still much better than what we had originally (where merge markers would absolutely) destroy the parse tree. And it is better than what we checked in last week, which could easily create a borked tree for many types of merges. Now, almost all merges should still leave the tree in good shape. All LS features will work in the first section, and lexical classification will work in the second.
2014-12-19 04:18:13 +01:00
comment("<<<<<<< HEAD"),
keyword("class"),
identifier("C"),
punctuation("{"),
punctuation("}"),
comment("=======\r\nclass D { }\r\n"),
comment(">>>>>>> Branch - a"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
Add some heuristics in the lexical classifier to make it play better with the syntactic classifier when classifying expressions involving generics.
2014-10-03 10:54:43 +02:00
});
More tests
2015-02-20 21:15:37 +01:00
it("'of' keyword", function () {
testLexicalClassification("for (var of of of) { }",
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
ts.EndOfLineState.None,
More tests
2015-02-20 21:15:37 +01:00
keyword("for"),
punctuation("("),
keyword("var"),
keyword("of"),
keyword("of"),
keyword("of"),
punctuation(")"),
punctuation("{"),
punctuation("}"),
Add a common, dense, format for classification operations to lower cost of processing on the host side. We now just return an array of triples to represent classified results. The triple contains: 1) the start of the classification. 2) the length of the classification. 3) the type of the clasification. We also encode this into a comma separated string when passing over to the managed side (as opposed to an JSON array). That way we don't pay such a high JSON parsing cost. Instead, we can just do a string.split(",") on the encoded triples and process each element ourselves.
2015-04-30 23:26:46 +02:00
finalEndOfLineState(ts.EndOfLineState.None));
More tests
2015-02-20 21:15:37 +01:00
});
Add some huristic optimization to not colorize a keyword if precceded by a dot or a keyword. this should handel cases for "a.var" or "module string { }"
2014-08-08 19:37:10 +02:00
});
Lint unit tests via harnessSources in Jakefile.js
2016-05-24 19:17:16 +02:00
});
Reference in a new issue Copy permalink