Skip to content

Comments

Ensure @rematch and nextEmbedded can be used together in Monarch grammar#95742

Merged
alexdima merged 3 commits intomicrosoft:masterfrom
bolinfest:rematch-with-grammar-change
Apr 27, 2020
Merged

Ensure @rematch and nextEmbedded can be used together in Monarch grammar#95742
alexdima merged 3 commits intomicrosoft:masterfrom
bolinfest:rematch-with-grammar-change

Conversation

@bolinfest
Copy link
Contributor

@bolinfest bolinfest commented Apr 21, 2020

Similar to the repro case I created for #90266, I created a local
playground with the following:

const DEMO_LANG_ID = "demo";
const SQL_QUERY_START = "(SELECT|INSERT|UPDATE|DELETE|CREATE|REPLACE|ALTER|WITH)";

const languageConfiguration = {
	tokenizer: {
		root: [
			[
				`(\"\"\")${SQL_QUERY_START}`,
				[
					{
						"token": "string.quote",
					},
					{
						token: "@rematch",
						next: "@endStringWithSQL",
						nextEmbedded: "sql",
					},
				]
			],
			[
				/(""")$/,
				[
					{
						token: "string.quote",
						next: "@maybeStringIsSQL",
					},
				]
			],
		],
		maybeStringIsSQL: [
			[
				/(.*)/,
				{
					cases: {
						[`${SQL_QUERY_START}\\b.*`]: {
							token: "@rematch",
							next: "@endStringWithSQL",
							nextEmbedded: "sql",
						},
						"@default": {
							token: "@rematch",
							switchTo: "@endDblDocString",
						},
					}
				}
			],
		],
		endDblDocString: [
			[
				"[^\"]+",
				"string"
			],
			[
				"\\\\\"",
				"string"
			],
			[
				"\"\"\"",
				"string",
				"@popall"
			],
			[
				"\"",
				"string"
			]
		],
		endStringWithSQL: [
			[
				/"""/,
				{
					token: "string.quote",
					next: "@popall",
					nextEmbedded: "@pop",
				},
			]
		],
	},
};
monaco.languages.register({
	id: DEMO_LANG_ID,
	extensions: ['.example'],
});
monaco.languages.setMonarchTokensProvider(DEMO_LANG_ID, languageConfiguration);

const value = `\
mysql_query("""SELECT * FROM table_name WHERE ds = '<DATEID>'""")

mysql_query("""
SELECT *
FROM table_name
WHERE ds = '<DATEID>'
""")
`;

var editor = monaco.editor.create(document.getElementById("container"), {
	value,
	language: DEMO_LANG_ID,

	lineNumbers: "off",
	roundedSelection: false,
	scrollBeyondLastLine: false,
	readOnly: false,
	theme: "vs-dark",
});

Without my change to monarchLexer.ts, I get this error in the console:

errors.ts:26 Uncaught Error: demo: cannot pop embedded mode if not inside one

But with my change, I see a Python-esque multiline string literal with
SQL syntax highlighting within it.

@msftclas
Copy link

msftclas commented Apr 21, 2020

CLA assistant check
All CLA requirements met.

@bolinfest bolinfest force-pushed the rematch-with-grammar-change branch from cd256d5 to 4e52140 Compare April 21, 2020 05:14
@bolinfest
Copy link
Contributor Author

/cc @alexdima

…ammar.

Similar to the repro case I created for
microsoft#90266, I created a local
playground with the following:

```js
const DEMO_LANG_ID = "demo";
const SQL_QUERY_START = "(SELECT|INSERT|UPDATE|DELETE|CREATE|REPLACE|ALTER|WITH)";

const languageConfiguration = {
	tokenizer: {
		root: [
			[
				`(\"\"\")${SQL_QUERY_START}`,
				[
					{
						"token": "string.quote",
					},
					{
						token: "@rematch",
						next: "@endStringWithSQL",
						nextEmbedded: "sql",
					},
				]
			],
			[
				/(""")$/,
				[
					{
						token: "string.quote",
						next: "@maybeStringIsSQL",
					},
				]
			],
		],
		maybeStringIsSQL: [
			[
				/(.*)/,
				{
					cases: {
						[`${SQL_QUERY_START}\\b.*`]: {
							token: "@rematch",
							next: "@endStringWithSQL",
							nextEmbedded: "sql",
						},
						"@default": {
							token: "@rematch",
							switchTo: "@endDblDocString",
						},
					}
				}
			],
		],
		endDblDocString: [
			[
				"[^\"]+",
				"string"
			],
			[
				"\\\\\"",
				"string"
			],
			[
				"\"\"\"",
				"string",
				"@popall"
			],
			[
				"\"",
				"string"
			]
		],
		endStringWithSQL: [
			[
				/"""/,
				{
					token: "string.quote",
					next: "@popall",
					nextEmbedded: "@pop",
				},
			]
		],
	},
};
monaco.languages.register({
	id: DEMO_LANG_ID,
	extensions: ['.example'],
});
monaco.languages.setMonarchTokensProvider(DEMO_LANG_ID, languageConfiguration);

const value = `\
mysql_query("""SELECT * FROM table_name WHERE ds = '<DATEID>'""")

mysql_query("""
SELECT *
FROM table_name
WHERE ds = '<DATEID>'
""")
`;

var editor = monaco.editor.create(document.getElementById("container"), {
	value,
	language: DEMO_LANG_ID,

	lineNumbers: "off",
	roundedSelection: false,
	scrollBeyondLastLine: false,
	readOnly: false,
	theme: "vs-dark",
});
```

Without my change to monarchLexer.ts, I get this error in the console:

> errors.ts:26 Uncaught Error: demo: cannot pop embedded mode if not inside one

But with my change, I see a Python-esque multiline string literal with
SQL syntax highlighting within it.
@bolinfest bolinfest force-pushed the rematch-with-grammar-change branch from 4e52140 to b9f5b9e Compare April 21, 2020 05:49
@sbatten sbatten requested a review from alexdima April 21, 2020 18:44
@alexdima alexdima added this to the April 2020 milestone Apr 27, 2020
@alexdima
Copy link
Member

Thank you!

@alexdima alexdima merged commit 2249598 into microsoft:master Apr 27, 2020
@github-actions github-actions bot locked and limited conversation to collaborators Jun 11, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants