{"id":951,"date":"2025-07-27T13:33:29","date_gmt":"2025-07-27T20:33:29","guid":{"rendered":"https:\/\/sysprogs.com\/tutorials\/?p=951"},"modified":"2025-07-27T13:33:29","modified_gmt":"2025-07-27T20:33:29","slug":"creating-wrapper-classes","status":"publish","type":"post","link":"https:\/\/sysprogs.com\/CodeVROOM\/documentation\/examples\/structure\/wrappers\/","title":{"rendered":"Creating Wrapper Classes"},"content":{"rendered":"

This example is from our website backend. We had existing code for parsing a source file and generating HTML with syntax highlighting. Before adding some extra functionality like collapsible blocks, we wanted to split it into 2 parts: parsing part that produces a list of token objects, and formatting part that generates HTML. Here is the prompt:<\/p>\n

Create nested classes TokenWrapper and LineWrapper with int Number and List<> Tokens.\r\nTokenWrapper should privately wrap original token, have public string Value and nested enum DiffState{Normal, Added, Removed} State.\r\nDumpTokens() should first build a list of wrapped lines, then dump them all.<\/pre>\n
\t\t
\t\t\t
\t\t\t\tOriginal<\/span>Claude 3.7 Sonnet<\/span>LLaMA-3.3-70B<\/span>LLaMA-4-Scout<\/span>GPT-4.1-mini<\/span>\t\t\t<\/div>\t\t\t
\t\t\t\t
\r\n
Model:<\/td>Claude 3.7 Sonnet<\/td><\/tr>
Time:<\/td>21s planning + 15s coding<\/td><\/tr>
Prompt tokens:<\/td>5.1K<\/td><\/tr>
Response tokens:<\/td>893<\/td><\/tr><\/table>\r\n<\/div>\r\n
\r\n
Model:<\/td>LLaMA3.3-70B (Cerebras)<\/td><\/tr>
Time taken:<\/td>0.5s planning + 1s coding<\/td><\/tr>
Prompt tokens:<\/td>1.5K<\/td><\/tr>
Response tokens:<\/td>502<\/td><\/tr><\/table>\r\n<\/div>\r\n
\r\n
Model:<\/td>LLaMA-4-Scout-17B-16E-Instruct (Cerebras)<\/td><\/tr>
Time taken:<\/td>0.5s planning + 0.4s coding<\/td><\/tr>
Prompt tokens:<\/td>1.2K<\/td><\/tr>
Response tokens:<\/td>552<\/td><\/tr><\/table>\r\n<\/div>\r\n
\r\n
Model:<\/td>GPT-4.1-mini<\/td><\/tr>
Time taken:<\/td>7s planning + 16s coding<\/td><\/tr>
Prompt tokens:<\/td>1.1K<\/td><\/tr>
Response tokens:<\/td>648<\/td><\/tr><\/table>\r\n<\/div>\r\n\t\t\t\t
\t\t\t\t\t
\r\n
\t
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n
6<\/div>\n
7<\/div>\n
8<\/div>\n
9<\/div>\n
10<\/div>\n
11<\/div>\n
12<\/div>\n
13<\/div>\n
14<\/div>\n
15<\/div>\n
16<\/div>\n
17<\/div>\n
18<\/div>\n
19<\/div>\n
20<\/div>\n
21<\/div>\n
22<\/div>\n
23<\/div>\n
24<\/div>\n
25<\/div>\n
26<\/div>\n
27<\/div>\n
28<\/div>\n
29<\/div>\n
30<\/div>\n
31<\/div>\n
32<\/div>\n
33<\/div>\n
34<\/div>\n
35<\/div>\n
36<\/div>\n
37<\/div>\n
38<\/div>\n
39<\/div>\n
40<\/div>\n
41<\/div>\n
42<\/div>\n
43<\/div>\n
44<\/div>\n
45<\/div>\n
46<\/div>\n
47<\/div>\n
48<\/div>\n
49<\/div>\n
50<\/div>\n
51<\/div>\n
52<\/div>\n
53<\/div>\n
54<\/div>\n
55<\/div>\n
56<\/div>\n<\/div>\t
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>\t
using<\/span> SysprogsDevTools<\/span>.<\/span>Tokenizer<\/span>;<\/span>\nusing<\/span> System<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Collections<\/span>.<\/span>Generic<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Linq<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Text<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Threading<\/span>.<\/span>Tasks<\/span>;<\/span>\n\nnamespace<\/span> Sysprogs<\/span>.<\/span>IDE<\/span>.<\/span>Framework<\/span>.<\/span>Languages<\/span>.<\/span>DiffExport<\/span>\n{<\/span>\n    public<\/span> class<\/span> ExporterDemo<\/span>\n    {<\/span>\n        public<\/span> static<\/span> string<\/span> DumpTokens<\/span>(<\/span>string<\/span> text<\/span>,<\/span> UniversalTokenizer<\/span> tokenizer<\/span>)<\/span>\n        {<\/span>\n            var<\/span> lines<\/span> =<\/span> text<\/span>.<\/span>Split<\/span>(<\/span>'\\n'<\/span>)<\/span>;<\/span>\n            var<\/span> result<\/span> =<\/span> new<\/span> StringBuilder<\/span>(<\/span>)<\/span>;<\/span>\n            \n            UniversalTokenizer<\/span>.<\/span>TokenType<\/span> tokenType<\/span> =<\/span> 0<\/span>;<\/span>\n            int<\/span> contextFlags<\/span> =<\/span> 0<\/span>;<\/span>\n            \n            foreach<\/span> (<\/span>var<\/span> line<\/span> in<\/span> lines<\/span>)<\/span>\n            {<\/span>\n                var<\/span> tokens<\/span> =<\/span> tokenizer<\/span>.<\/span>TokenizeLine<\/span>(<\/span>line<\/span>,<\/span> ref<\/span> tokenType<\/span>,<\/span> contextFlags<\/span>)<\/span>;<\/span>\n                \n                foreach<\/span> (<\/span>var<\/span> token<\/span> in<\/span> tokens<\/span>)<\/span>\n                {<\/span>\n                    string<\/span> cssClass<\/span> =<\/span> TranslateTokenType<\/span>(<\/span>token<\/span>.<\/span>Type<\/span>)<\/span>;<\/span>\n                    if<\/span> (<\/span>cssClass<\/span> !=<\/span> null<\/span>)<\/span>\n                        result<\/span>.<\/span>Append<\/span>(<\/span>$"<span<\/span> class=\\"{cssClass}\\">{token.GetText(line)}<\/span>"<\/span>)<\/span>;<\/span>\n                    else<\/span>\n                        result<\/span>.<\/span>Append<\/span>(<\/span>token<\/span>.<\/span>GetText<\/span>(<\/span>line<\/span>)<\/span>)<\/span>;<\/span>\n                }<\/span>\n                \n                result<\/span>.<\/span>AppendLine<\/span>(<\/span>)<\/span>;<\/span>\n            }<\/span>\n            \n            return<\/span> result<\/span>.<\/span>ToString<\/span>(<\/span>)<\/span>;<\/span>\n        }<\/span>\n        \n        private<\/span> static<\/span> string<\/span> TranslateTokenType<\/span>(<\/span>UniversalTokenizer<\/span>.<\/span>TokenType<\/span> tokenType<\/span>)<\/span>\n        {<\/span>\n            switch<\/span> (<\/span>tokenType<\/span>)<\/span>\n            {<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>Keyword<\/span>:<\/span>\n                    return<\/span> "keyword"<\/span>;<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>CharacterLiteral<\/span>:<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>StringLiteral<\/span>:<\/span>\n                    return<\/span> "literal"<\/span>;<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>Comment<\/span>:<\/span>\n                    return<\/span> "comment"<\/span>;<\/span>\n                default<\/span>:<\/span>\n                    return<\/span> null<\/span>;<\/span>\n            }<\/span>\n        }<\/span>\n    }<\/span>\n}<\/span>\n\n<\/div><\/div>\t\t\t\r\n<\/div>\r\n
\r\n
\t
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n
6<\/div>\n
7<\/div>\n
8<\/div>\n
9<\/div>\n
10<\/div>\n
11<\/div>\n
12<\/div>\n
13<\/div>\n
14<\/div>\n
15<\/div>\n
16<\/div>\n
17<\/div>\n
18<\/div>\n
19<\/div>\n<\/div>
20<\/div>\n
21<\/div>\n
22<\/div>\n
23<\/div>\n
24<\/div>\n
25<\/div>\n
26<\/div>\n
27<\/div>\n
28<\/div>\n
29<\/div>\n
30<\/div>\n
31<\/div>\n
32<\/div>\n<\/div>
33<\/div>\n<\/div>
34<\/div>\n
35<\/div>\n
36<\/div>\n
37<\/div>\n
38<\/div>\n
39<\/div>\n
40<\/div>\n
41<\/div>\n
42<\/div>\n
43<\/div>\n
44<\/div>\n<\/div>
45<\/div>\n<\/div>
46<\/div>\n
47<\/div>\n
48<\/div>\n
49<\/div>\n
50<\/div>\n
51<\/div>\n
52<\/div>\n
53<\/div>\n
54<\/div>\n
55<\/div>\n
56<\/div>\n
57<\/div>\n
58<\/div>\n
59<\/div>\n
60<\/div>\n
61<\/div>\n
62<\/div>\n
63<\/div>\n
64<\/div>\n
65<\/div>\n
66<\/div>\n
67<\/div>\n
68<\/div>\n
69<\/div>\n
70<\/div>\n
71<\/div>\n
72<\/div>\n
73<\/div>\n
74<\/div>\n
75<\/div>\n
76<\/div>\n
77<\/div>\n
78<\/div>\n
79<\/div>\n
80<\/div>\n
81<\/div>\n
82<\/div>\n
83<\/div>\n
84<\/div>\n
85<\/div>\n
86<\/div>\n
87<\/div>\n
88<\/div>\n<\/div>
89<\/div>\n
90<\/div>\n
91<\/div>\n
92<\/div>\n
93<\/div>\n
94<\/div>\n
95<\/div>\n
96<\/div>\n
97<\/div>\n
98<\/div>\n
99<\/div>\n
100<\/div>\n
101<\/div>\n
102<\/div>\n
103<\/div>\n
104<\/div>\n<\/div>
105<\/div>\n<\/div>
106<\/div>\n<\/div>
107<\/div>\n<\/div>\t
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>\t
using<\/span> SysprogsDevTools<\/span>.<\/span>Tokenizer<\/span>;<\/span>\nusing<\/span> System<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Collections<\/span>.<\/span>Generic<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Linq<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Text<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Threading<\/span>.<\/span>Tasks<\/span>;<\/span>\n\nnamespace<\/span> Sysprogs<\/span>.<\/span>IDE<\/span>.<\/span>Framework<\/span>.<\/span>Languages<\/span>.<\/span>DiffExport<\/span>\n{<\/span>\n    public<\/span> class<\/span> ExporterDemo<\/span>\n    {<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>class<\/span> <\/span>TokenWrapper<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>enum<\/span> <\/span>DiffState<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Normal<\/span>,<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Added<\/span>,<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Removed<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>private<\/span> <\/span>readonly<\/span> <\/span>UniversalTokenizer<\/span>.<\/span>Token<\/span> <\/span>_token<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>private<\/span> <\/span>readonly<\/span> <\/span>string<\/span> <\/span>_text<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>string<\/span> <\/span>Value<\/span> <\/span>=><\/span> <\/span>_text<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>DiffState<\/span> <\/span>State<\/span> <\/span>{<\/span> <\/span>get<\/span>;<\/span> <\/span>set<\/span>;<\/span> <\/span>}<\/span> <\/span>=<\/span> <\/span>DiffState<\/span>.<\/span>Normal<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>UniversalTokenizer<\/span>.<\/span>TokenType<\/span> <\/span>Type<\/span> <\/span>=><\/span> <\/span>_token<\/span>.<\/span>Type<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>TokenWrapper<\/span>(<\/span>UniversalTokenizer<\/span>.<\/span>Token<\/span> <\/span>token<\/span>,<\/span> <\/span>string<\/span> <\/span>text<\/span>)<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>_token<\/span> <\/span>=<\/span> <\/span>token<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>_text<\/span> <\/span>=<\/span> <\/span>text<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>class<\/span> <\/span>LineWrapper<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>int<\/span> <\/span>Number<\/span> <\/span>{<\/span> <\/span>get<\/span>;<\/span> <\/span>set<\/span>;<\/span> <\/span>}<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>List<\/span><<\/span>TokenWrapper<\/span>><\/span> <\/span>Tokens<\/span> <\/span>{<\/span> <\/span>get<\/span>;<\/span> <\/span>set<\/span>;<\/span> <\/span>}<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>LineWrapper<\/span>(<\/span>int<\/span> <\/span>number<\/span>,<\/span> <\/span>List<\/span><<\/span>TokenWrapper<\/span>><\/span> <\/span>tokens<\/span>)<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Number<\/span> <\/span>=<\/span> <\/span>number<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Tokens<\/span> <\/span>=<\/span> <\/span>tokens<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span>\n        public<\/span> static<\/span> string<\/span> DumpTokens<\/span>(<\/span>string<\/span> text<\/span>,<\/span> UniversalTokenizer<\/span> tokenizer<\/span>)<\/span>\n        {<\/span>\n            var<\/span> lines<\/span> =<\/span> text<\/span>.<\/span>Split<\/span>(<\/span>'\\n'<\/span>)<\/span>;<\/span>\n            var<\/span> result<\/span> =<\/span> new<\/span> StringBuilder<\/span>(<\/span>)<\/span>;<\/span>\n            \n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\/\/<\/span> <\/span>First<\/span> <\/span>build<\/span> <\/span>a<\/span> <\/span>list<\/span> <\/span>of<\/span> <\/span>wrapped<\/span> <\/span>lines<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>wrappedLines<\/span> <\/span>=<\/span> <\/span>new<\/span> <\/span>List<\/span><<\/span>LineWrapper<\/span>><\/span>(<\/span>)<\/span>;<\/span> <\/span>\n            UniversalTokenizer<\/span>.<\/span>TokenType<\/span> tokenType<\/span> =<\/span> 0<\/span>;<\/span>\n            int<\/span> contextFlags<\/span> =<\/span> 0<\/span>;<\/span>\n            \n            for<\/span> (<\/span>int<\/span> line<\/span>Number<\/span> <\/span>=<\/span> <\/span>0<\/span>;<\/span> <\/span>lineNumber<\/span> <\/span><<\/span> lines<\/span>.<\/span>Length<\/span>;<\/span> <\/span>lineNumber<\/span>++<\/span>)<\/span>\n            {<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>line<\/span> <\/span>=<\/span> <\/span>lines<\/span>[<\/span>lineNumber<\/span>]<\/span>;<\/span> <\/span>\n                var<\/span> tokens<\/span> =<\/span> tokenizer<\/span>.<\/span>TokenizeLine<\/span>(<\/span>line<\/span>,<\/span> ref<\/span> tokenType<\/span>,<\/span> contextFlags<\/span>)<\/span>;<\/span>\n                \n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>wrappedTokens<\/span> <\/span>=<\/span> <\/span>new<\/span> <\/span>List<\/span><<\/span>TokenWrapper<\/span>><\/span>(<\/span>)<\/span>;<\/span> <\/span>\n                foreach<\/span> (<\/span>var<\/span> token<\/span> in<\/span> tokens<\/span>)<\/span>\n                {<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>wrappedToken<\/span> <\/span>=<\/span> <\/span>new<\/span> <\/span>TokenWrapper<\/span>(<\/span>token<\/span>,<\/span> <\/span>token<\/span>.<\/span>GetText<\/span>(<\/span>line<\/span>)<\/span>)<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>wrappedTokens<\/span>.<\/span>Add<\/span>(<\/span>wrappedToken<\/span>)<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>wrappedLines<\/span>.<\/span>Add<\/span>(<\/span>new<\/span> <\/span>LineWrapper<\/span>(<\/span>lineNumber<\/span> <\/span>+<\/span> <\/span>1<\/span>,<\/span> <\/span>wrappedTokens<\/span>)<\/span>)<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\/\/<\/span> <\/span>Now<\/span> <\/span>dump<\/span> <\/span>all<\/span> <\/span>the<\/span> <\/span>wrapped<\/span> <\/span>lines<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>foreach<\/span> <\/span>(<\/span>var<\/span> <\/span>wrappedLine<\/span> <\/span>in<\/span> <\/span>wrappedLines<\/span>)<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>foreach<\/span> <\/span>(<\/span>var<\/span> <\/span>token<\/span> <\/span>in<\/span> <\/span>wrappedLine<\/span>.<\/span>Tokens<\/span>)<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n                    string<\/span> cssClass<\/span> =<\/span> TranslateTokenType<\/span>(<\/span>token<\/span>.<\/span>Type<\/span>)<\/span>;<\/span>\n                    if<\/span> (<\/span>cssClass<\/span> !=<\/span> null<\/span>)<\/span>\n                        result<\/span>.<\/span>Append<\/span>(<\/span>$"<span<\/span> class=\\"{cssClass}\\">{token.<\/span>Value<\/span>}<\/span>"<\/span>)<\/span>;<\/span>\n                    else<\/span>\n                        result<\/span>.<\/span>Append<\/span>(<\/span>token<\/span>.<\/span>Value<\/span>)<\/span>;<\/span>\n                }<\/span>\n                \n                result<\/span>.<\/span>AppendLine<\/span>(<\/span>)<\/span>;<\/span>\n            }<\/span>\n            \n            return<\/span> result<\/span>.<\/span>ToString<\/span>(<\/span>)<\/span>;<\/span>\n        }<\/span><\/span>{...}<\/span>\n        \n        private<\/span> static<\/span> string<\/span> TranslateTokenType<\/span>(<\/span>UniversalTokenizer<\/span>.<\/span>TokenType<\/span> tokenType<\/span>)<\/span>\n        {<\/span>\n            switch<\/span> (<\/span>tokenType<\/span>)<\/span>\n            {<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>Keyword<\/span>:<\/span>\n                    return<\/span> "keyword"<\/span>;<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>CharacterLiteral<\/span>:<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>StringLiteral<\/span>:<\/span>\n                    return<\/span> "literal"<\/span>;<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>Comment<\/span>:<\/span>\n                    return<\/span> "comment"<\/span>;<\/span>\n                default<\/span>:<\/span>\n                    return<\/span> null<\/span>;<\/span>\n            }<\/span>\n        }<\/span><\/span>{...}<\/span>\n    }<\/span><\/span>{...}<\/span>\n}<\/span><\/span>{...}<\/span>\n\n<\/div><\/div>\t\t\t\r\n<\/div>\r\n
\r\n
\t
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n
6<\/div>\n
7<\/div>\n
8<\/div>\n
9<\/div>\n
10<\/div>\n
11<\/div>\n
12<\/div>\n
13<\/div>\n
14<\/div>\n
15<\/div>\n
16<\/div>\n
17<\/div>\n
18<\/div>\n
19<\/div>\n
20<\/div>\n
21<\/div>\n
22<\/div>\n<\/div>
23<\/div>\n<\/div>
24<\/div>\n
25<\/div>\n
26<\/div>\n
27<\/div>\n
28<\/div>\n
29<\/div>\n
30<\/div>\n
31<\/div>\n
32<\/div>\n
33<\/div>\n
34<\/div>\n<\/div>
35<\/div>\n<\/div>
36<\/div>\n
37<\/div>\n
38<\/div>\n
39<\/div>\n
40<\/div>\n
41<\/div>\n
42<\/div>\n
43<\/div>\n
44<\/div>\n
45<\/div>\n
46<\/div>\n
47<\/div>\n
48<\/div>\n
49<\/div>\n
50<\/div>\n
51<\/div>\n
52<\/div>\n
53<\/div>\n
54<\/div>\n
55<\/div>\n
 <\/div>\n
 <\/div>\n
56<\/div>\n
57<\/div>\n
58<\/div>\n
59<\/div>\n
60<\/div>\n
61<\/div>\n
62<\/div>\n
63<\/div>\n
64<\/div>\n
65<\/div>\n
66<\/div>\n
67<\/div>\n
68<\/div>\n<\/div>
69<\/div>\n
70<\/div>\n
71<\/div>\n
72<\/div>\n
73<\/div>\n
74<\/div>\n
75<\/div>\n
76<\/div>\n
77<\/div>\n
78<\/div>\n
79<\/div>\n
80<\/div>\n
81<\/div>\n
82<\/div>\n
83<\/div>\n
84<\/div>\n<\/div>
85<\/div>\n<\/div>
86<\/div>\n<\/div>
87<\/div>\n<\/div>\t
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>\t
using<\/span> SysprogsDevTools<\/span>.<\/span>Tokenizer<\/span>;<\/span>\nusing<\/span> System<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Collections<\/span>.<\/span>Generic<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Linq<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Text<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Threading<\/span>.<\/span>Tasks<\/span>;<\/span>\n\nnamespace<\/span> Sysprogs<\/span>.<\/span>IDE<\/span>.<\/span>Framework<\/span>.<\/span>Languages<\/span>.<\/span>DiffExport<\/span>\n{<\/span>\n    public<\/span> class<\/span> ExporterDemo<\/span>\n    {<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>class<\/span> <\/span>TokenWrapper<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>private<\/span> <\/span>UniversalTokenizer<\/span>.<\/span>Token<\/span> <\/span>_token<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>string<\/span> <\/span>Value<\/span> <\/span>=><\/span> <\/span>_token<\/span>.<\/span>GetText<\/span>(<\/span>""<\/span>)<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>enum<\/span> <\/span>DiffState<\/span> <\/span>{<\/span> <\/span>Normal<\/span>,<\/span> <\/span>Added<\/span>,<\/span> <\/span>Removed<\/span> <\/span>}<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>DiffState<\/span> <\/span>State<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>TokenWrapper<\/span>(<\/span>UniversalTokenizer<\/span>.<\/span>Token<\/span> <\/span>token<\/span>)<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>_token<\/span> <\/span>=<\/span> <\/span>token<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>class<\/span> <\/span>LineWrapper<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>int<\/span> <\/span>Number<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>List<\/span><<\/span>TokenWrapper<\/span>><\/span> <\/span>Tokens<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>LineWrapper<\/span>(<\/span>int<\/span> <\/span>number<\/span>,<\/span> <\/span>List<\/span><<\/span>TokenWrapper<\/span>><\/span> <\/span>tokens<\/span>)<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Number<\/span> <\/span>=<\/span> <\/span>number<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Tokens<\/span> <\/span>=<\/span> <\/span>tokens<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span>\n        public<\/span> static<\/span> string<\/span> DumpTokens<\/span>(<\/span>string<\/span> text<\/span>,<\/span> UniversalTokenizer<\/span> tokenizer<\/span>)<\/span>\n        {<\/span>\n            var<\/span> lines<\/span> =<\/span> text<\/span>.<\/span>Split<\/span>(<\/span>'\\n'<\/span>)<\/span>;<\/span>\n            var<\/span> wrappedLines<\/span> =<\/span> new<\/span> List<\/span><<\/span>LineWrapper<\/span>><\/span>(<\/span>)<\/span>;<\/span>\n <\/span>\n            for<\/span> <\/span>(<\/span>int<\/span> <\/span>i<\/span> <\/span>=<\/span> <\/span>0<\/span>;<\/span> <\/span>i<\/span> <\/span><<\/span> <\/span>lines<\/span>.<\/span>Length<\/span>;<\/span> <\/span>i<\/span>++<\/span>)<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n             <\/span> <\/span> <\/span> <\/span>UniversalTokenizer<\/span>.<\/span>TokenType<\/span> tokenType<\/span> =<\/span> 0<\/span>;<\/span>\n             <\/span> <\/span> <\/span> <\/span>int<\/span> contextFlags<\/span> =<\/span> 0<\/span>;<\/span>\n             <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>tokens<\/span> <\/span>=<\/span> <\/span>tokenizer<\/span>.<\/span>TokenizeLine<\/span>(<\/span>lines<\/span>[<\/span>i<\/span>]<\/span>,<\/span> <\/span>ref<\/span> <\/span>tokenType<\/span>,<\/span> <\/span>contextFlags<\/span>)<\/span>;<\/span>\n             <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>wrappedTokens<\/span> <\/span>=<\/span> <\/span>tokens<\/span>.<\/span>Select<\/span>(<\/span>token<\/span> <\/span>=><\/span> <\/span>new<\/span> <\/span>TokenWrapper<\/span>(<\/span>token<\/span>)<\/span>)<\/span>.<\/span>ToList<\/span>(<\/span>)<\/span>;<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>wrappedLines<\/span>.<\/span>Add<\/span>(<\/span>new<\/span> <\/span>LineWrapper<\/span>(<\/span>i<\/span> <\/span>+<\/span> <\/span>1<\/span>,<\/span> <\/span>wrappedTokens<\/span>)<\/span>)<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>new<\/span> <\/span>StringBuilder<\/span>(<\/span>)<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>foreach<\/span> <\/span>(<\/span>var<\/span> <\/span>wrappedLine<\/span> <\/span>in<\/span> <\/span>wrappedLines<\/span>)<\/span> <\/span>\n            {<\/span>\n                foreach<\/span> <\/span>(<\/span>var<\/span> token<\/span> <\/span>in<\/span> <\/span>wrappedLine<\/span>.<\/span>Tokens<\/span>)<\/span>;<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>foreach<\/span> <\/span>(<\/span>var<\/span> <\/span>token<\/span> <\/span>in<\/span> <\/span>tokens<\/span>)<\/span> <\/span>\n                {<\/span>\n                    string<\/span> cssClass<\/span> =<\/span> TranslateTokenType<\/span>(<\/span>token<\/span>.<\/span>_token<\/span>.<\/span>Type<\/span>)<\/span>;<\/span> <\/span>\/\/<\/span> <\/span>TODO:<\/span> <\/span>get<\/span> <\/span>Type<\/span> <\/span>from<\/span> <\/span>UniversalTokenizer.Token<\/span>\n                    if<\/span> (<\/span>cssClass<\/span> !=<\/span> null<\/span>)<\/span>\n                        result<\/span>.<\/span>Append<\/span>(<\/span>$"<span<\/span> class=\\"{cssClass}\\">{token.<\/span>Value<\/span>}<\/span>"<\/span>)<\/span>;<\/span>\n                    else<\/span>\n                        result<\/span>.<\/span>Append<\/span>(<\/span>token<\/span>.<\/span>Value<\/span>)<\/span>;<\/span>\n                }<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n                result<\/span>.<\/span>AppendLine<\/span>(<\/span>)<\/span>;<\/span>\n            }<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n            return<\/span> result<\/span>.<\/span>ToString<\/span>(<\/span>)<\/span>;<\/span>\n        }<\/span><\/span>{...}<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n        private<\/span> static<\/span> string<\/span> TranslateTokenType<\/span>(<\/span>UniversalTokenizer<\/span>.<\/span>TokenType<\/span> tokenType<\/span>)<\/span>\n        {<\/span>\n            switch<\/span> (<\/span>tokenType<\/span>)<\/span>\n            {<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>Keyword<\/span>:<\/span>\n                    return<\/span> "keyword"<\/span>;<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>CharacterLiteral<\/span>:<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>StringLiteral<\/span>:<\/span>\n                    return<\/span> "literal"<\/span>;<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>Comment<\/span>:<\/span>\n                    return<\/span> "comment"<\/span>;<\/span>\n                default<\/span>:<\/span>\n                    return<\/span> null<\/span>;<\/span>\n            }<\/span>\n        }<\/span><\/span>{...}<\/span>\n    }<\/span><\/span>{...}<\/span>\n}<\/span><\/span>{...}<\/span>\n\n<\/div><\/div>\t\t\t\r\n<\/div>\r\n
\r\n
\t
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n
6<\/div>\n
7<\/div>\n
8<\/div>\n
9<\/div>\n
10<\/div>\n
11<\/div>\n
12<\/div>\n
13<\/div>\n
14<\/div>\n
15<\/div>\n
16<\/div>\n
17<\/div>\n
18<\/div>\n
19<\/div>\n
20<\/div>\n
21<\/div>\n
22<\/div>\n
23<\/div>\n
24<\/div>\n
25<\/div>\n
26<\/div>\n
27<\/div>\n
28<\/div>\n
29<\/div>\n
30<\/div>\n
31<\/div>\n
32<\/div>\n
33<\/div>\n
34<\/div>\n
35<\/div>\n
36<\/div>\n
 <\/div>\n
37<\/div>\n
38<\/div>\n
39<\/div>\n
40<\/div>\n
41<\/div>\n<\/div>
42<\/div>\n
43<\/div>\n
44<\/div>\n
45<\/div>\n
46<\/div>\n
47<\/div>\n
48<\/div>\n
49<\/div>\n
50<\/div>\n
51<\/div>\n
52<\/div>\n
53<\/div>\n
54<\/div>\n
55<\/div>\n
56<\/div>\n
57<\/div>\n<\/div>
58<\/div>\n
59<\/div>\n
60<\/div>\n
61<\/div>\n
62<\/div>\n
63<\/div>\n
64<\/div>\n
65<\/div>\n
66<\/div>\n
67<\/div>\n
68<\/div>\n<\/div>
69<\/div>\n<\/div>
70<\/div>\n
71<\/div>\n
72<\/div>\n
73<\/div>\n
74<\/div>\n
75<\/div>\n
76<\/div>\n
77<\/div>\n
78<\/div>\n
79<\/div>\n
80<\/div>\n
81<\/div>\n
82<\/div>\n
83<\/div>\n
84<\/div>\n
85<\/div>\n
86<\/div>\n
87<\/div>\n
88<\/div>\n<\/div>
89<\/div>\n<\/div>
90<\/div>\n<\/div>
91<\/div>\n<\/div>
92<\/div>\n<\/div>\t
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>\t
using<\/span> SysprogsDevTools<\/span>.<\/span>Tokenizer<\/span>;<\/span>\nusing<\/span> System<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Collections<\/span>.<\/span>Generic<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Linq<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Text<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Threading<\/span>.<\/span>Tasks<\/span>;<\/span>\n\nnamespace<\/span> Sysprogs<\/span>.<\/span>IDE<\/span>.<\/span>Framework<\/span>.<\/span>Languages<\/span>.<\/span>DiffExport<\/span>\n{<\/span>\n    public<\/span> class<\/span> ExporterDemo<\/span>\n    {<\/span>\n        public<\/span> static<\/span> string<\/span> DumpTokens<\/span>(<\/span>string<\/span> text<\/span>,<\/span> UniversalTokenizer<\/span> tokenizer<\/span>)<\/span>\n        {<\/span>\n            var<\/span> lines<\/span> =<\/span> text<\/span>.<\/span>Split<\/span>(<\/span>'\\n'<\/span>)<\/span>;<\/span>\n            var<\/span> wrappedLines<\/span> =<\/span> new<\/span> List<\/span><<\/span>LineWrapper<\/span>><\/span>(<\/span>)<\/span>;<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n            UniversalTokenizer<\/span>.<\/span>TokenType<\/span> tokenType<\/span> =<\/span> 0<\/span>;<\/span>\n            int<\/span> contextFlags<\/span> =<\/span> 0<\/span>;<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n            for<\/span> (<\/span>int<\/span> <\/span>i<\/span> <\/span>=<\/span> <\/span>0<\/span>;<\/span> <\/span>i<\/span> <\/span><<\/span> lines<\/span>.<\/span>Length<\/span>;<\/span> <\/span>i<\/span>++<\/span>)<\/span>\n            {<\/span>\n                var<\/span> tokens<\/span> =<\/span> tokenizer<\/span>.<\/span>TokenizeLine<\/span>(<\/span>lines<\/span>[<\/span>i<\/span>]<\/span>,<\/span> ref<\/span> tokenType<\/span>,<\/span> contextFlags<\/span>)<\/span>;<\/span>\n                wrappedLines<\/span>.<\/span>Add<\/span>(<\/span>new<\/span> <\/span>LineWrapper<\/span>(<\/span>i<\/span> <\/span>+<\/span> <\/span>1<\/span>,<\/span> <\/span>tokens<\/span>.<\/span>Select<\/span>(<\/span>t<\/span> <\/span>=><\/span> <\/span>new<\/span> <\/span>TokenWrapper<\/span>(<\/span>t<\/span>,<\/span> <\/span>lines<\/span>[<\/span>i<\/span>]<\/span>)<\/span>)<\/span>.<\/span>ToList<\/span>(<\/span>)<\/span>)<\/span>)<\/span>;<\/span>\n            }<\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>new<\/span> <\/span>StringBuilder<\/span>(<\/span>)<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>foreach<\/span> <\/span>(<\/span>var<\/span> <\/span>line<\/span> <\/span>in<\/span> <\/span>wrappedLines<\/span>)<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>foreach<\/span> <\/span>(<\/span>var<\/span> <\/span>token<\/span> <\/span>in<\/span> <\/span>line<\/span>.<\/span>Tokens<\/span>)<\/span> <\/span>\n                {<\/span>\n                    string<\/span> cssClass<\/span> =<\/span> TranslateTokenType<\/span>(<\/span>token<\/span>.<\/span>WrappedToken<\/span>.<\/span>Type<\/span>)<\/span>;<\/span>\n                    if<\/span> (<\/span>cssClass<\/span> !=<\/span> null<\/span>)<\/span>\n                        result<\/span>.<\/span>Append<\/span>(<\/span>$"<span<\/span> class=\\"{cssClass}\\">{token.<\/span>Value<\/span>}<\/span>"<\/span>)<\/span>;<\/span>\n                    else<\/span>\n                        result<\/span>.<\/span>Append<\/span>(<\/span>token<\/span>.<\/span>Value<\/span>)<\/span>;<\/span>\n                }<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n                result<\/span>.<\/span>AppendLine<\/span>(<\/span>)<\/span>;<\/span>\n            }<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n            return<\/span> result<\/span>.<\/span>ToString<\/span>(<\/span>)<\/span>;<\/span>\n        }<\/span><\/span>{...}<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n        private<\/span> static<\/span> string<\/span> TranslateTokenType<\/span>(<\/span>UniversalTokenizer<\/span>.<\/span>TokenType<\/span> tokenType<\/span>)<\/span>\n        {<\/span>\n            switch<\/span> (<\/span>tokenType<\/span>)<\/span>\n            {<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>Keyword<\/span>:<\/span>\n                    return<\/span> "keyword"<\/span>;<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>CharacterLiteral<\/span>:<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>StringLiteral<\/span>:<\/span>\n                    return<\/span> "literal"<\/span>;<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>Comment<\/span>:<\/span>\n                    return<\/span> "comment"<\/span>;<\/span>\n                default<\/span>:<\/span>\n                    return<\/span> null<\/span>;<\/span>\n            }<\/span>\n        }<\/span><\/span>{...}<\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>class<\/span> <\/span>LineWrapper<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>int<\/span> <\/span>Number<\/span> <\/span>{<\/span> <\/span>get<\/span>;<\/span> <\/span>}<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>List<\/span><<\/span>TokenWrapper<\/span>><\/span> <\/span>Tokens<\/span> <\/span>{<\/span> <\/span>get<\/span>;<\/span> <\/span>}<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>LineWrapper<\/span>(<\/span>int<\/span> <\/span>number<\/span>,<\/span> <\/span>List<\/span><<\/span>TokenWrapper<\/span>><\/span> <\/span>tokens<\/span>)<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Number<\/span> <\/span>=<\/span> <\/span>number<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Tokens<\/span> <\/span>=<\/span> <\/span>tokens<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>class<\/span> <\/span>TokenWrapper<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>private<\/span> <\/span>UniversalTokenizer<\/span>.<\/span>Token<\/span> <\/span>_wrappedToken<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>private<\/span> <\/span>string<\/span> <\/span>_line<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>string<\/span> <\/span>Value<\/span> <\/span>=><\/span> <\/span>_wrappedToken<\/span>.<\/span>GetText<\/span>(<\/span>_line<\/span>)<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>enum<\/span> <\/span>DiffState<\/span> <\/span>{<\/span> <\/span>Normal<\/span>,<\/span> <\/span>Added<\/span>,<\/span> <\/span>Removed<\/span> <\/span>}<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>DiffState<\/span> <\/span>State<\/span> <\/span>{<\/span> <\/span>get<\/span>;<\/span> <\/span>set<\/span>;<\/span> <\/span>}<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>UniversalTokenizer<\/span>.<\/span>Token<\/span> <\/span>WrappedToken<\/span> <\/span>=><\/span> <\/span>_wrappedToken<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>TokenWrapper<\/span>(<\/span>UniversalTokenizer<\/span>.<\/span>Token<\/span> <\/span>wrappedToken<\/span>,<\/span> <\/span>string<\/span> <\/span>line<\/span>)<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>_wrappedToken<\/span> <\/span>=<\/span> <\/span>wrappedToken<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>_line<\/span> <\/span>=<\/span> <\/span>line<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>State<\/span> <\/span>=<\/span> <\/span>DiffState<\/span>.<\/span>Normal<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n    }<\/span><\/span>{...}<\/span>\n}<\/span><\/span>{...}<\/span>\n\n<\/div><\/div>\t\t\t\r\n<\/div>\r\n
\r\n
\t
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n
6<\/div>\n
7<\/div>\n
8<\/div>\n
9<\/div>\n
10<\/div>\n
11<\/div>\n
12<\/div>\n
13<\/div>\n
14<\/div>\n
15<\/div>\n
16<\/div>\n
17<\/div>\n
18<\/div>\n
19<\/div>\n
20<\/div>\n
21<\/div>\n
22<\/div>\n
23<\/div>\n
24<\/div>\n
25<\/div>\n
26<\/div>\n
27<\/div>\n
28<\/div>\n
29<\/div>\n
30<\/div>\n
31<\/div>\n
32<\/div>\n
33<\/div>\n
34<\/div>\n
35<\/div>\n
36<\/div>\n
37<\/div>\n
38<\/div>\n
39<\/div>\n
40<\/div>\n
41<\/div>\n
42<\/div>\n
43<\/div>\n
44<\/div>\n
45<\/div>\n
46<\/div>\n
47<\/div>\n
48<\/div>\n
49<\/div>\n<\/div>
50<\/div>\n
51<\/div>\n
52<\/div>\n
53<\/div>\n
54<\/div>\n
55<\/div>\n
56<\/div>\n
57<\/div>\n
58<\/div>\n
59<\/div>\n
60<\/div>\n
61<\/div>\n
62<\/div>\n
63<\/div>\n
64<\/div>\n
65<\/div>\n<\/div>
66<\/div>\n
67<\/div>\n
68<\/div>\n
69<\/div>\n
70<\/div>\n
71<\/div>\n
72<\/div>\n
73<\/div>\n
74<\/div>\n
75<\/div>\n
76<\/div>\n
77<\/div>\n<\/div>
78<\/div>\n
79<\/div>\n
80<\/div>\n
81<\/div>\n
82<\/div>\n
83<\/div>\n
84<\/div>\n
85<\/div>\n
86<\/div>\n<\/div>
87<\/div>\n<\/div>
88<\/div>\n
89<\/div>\n
90<\/div>\n
91<\/div>\n
92<\/div>\n
93<\/div>\n
94<\/div>\n
95<\/div>\n
96<\/div>\n
97<\/div>\n
98<\/div>\n<\/div>
99<\/div>\n<\/div>
100<\/div>\n<\/div>
101<\/div>\n<\/div>
102<\/div>\n<\/div>\t
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>
<\/div> <\/span><\/div>\n<\/div>\t
using<\/span> SysprogsDevTools<\/span>.<\/span>Tokenizer<\/span>;<\/span>\nusing<\/span> System<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Collections<\/span>.<\/span>Generic<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Linq<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Text<\/span>;<\/span>\nusing<\/span> System<\/span>.<\/span>Threading<\/span>.<\/span>Tasks<\/span>;<\/span>\n\nnamespace<\/span> Sysprogs<\/span>.<\/span>IDE<\/span>.<\/span>Framework<\/span>.<\/span>Languages<\/span>.<\/span>DiffExport<\/span>\n{<\/span>\n    public<\/span> class<\/span> ExporterDemo<\/span>\n    {<\/span>\n        public<\/span> static<\/span> string<\/span> DumpTokens<\/span>(<\/span>string<\/span> text<\/span>,<\/span> UniversalTokenizer<\/span> tokenizer<\/span>)<\/span>\n        {<\/span>\n            var<\/span> lines<\/span> =<\/span> text<\/span>.<\/span>Split<\/span>(<\/span>'\\n'<\/span>)<\/span>;<\/span>\n            var<\/span> wrappedLines<\/span> =<\/span> new<\/span> List<\/span><<\/span>LineWrapper<\/span>><\/span>(<\/span>)<\/span>;<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n            UniversalTokenizer<\/span>.<\/span>TokenType<\/span> tokenType<\/span> =<\/span> 0<\/span>;<\/span>\n            int<\/span> contextFlags<\/span> =<\/span> 0<\/span>;<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n            for<\/span> (<\/span>int<\/span> line<\/span>Index<\/span> <\/span>=<\/span> <\/span>0<\/span>;<\/span> <\/span>lineIndex<\/span> <\/span><<\/span> lines<\/span>.<\/span>Length<\/span>;<\/span> <\/span>lineIndex<\/span>++<\/span>)<\/span>\n            {<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>line<\/span> <\/span>=<\/span> <\/span>lines<\/span>[<\/span>lineIndex<\/span>]<\/span>;<\/span> <\/span>\n                var<\/span> tokens<\/span> =<\/span> tokenizer<\/span>.<\/span>TokenizeLine<\/span>(<\/span>line<\/span>,<\/span> ref<\/span> tokenType<\/span>,<\/span> contextFlags<\/span>)<\/span>;<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>tokenWrappers<\/span> <\/span>=<\/span> <\/span>new<\/span> <\/span>List<\/span><<\/span>TokenWrapper<\/span>><\/span>(<\/span>)<\/span>;<\/span> <\/span>\n                foreach<\/span> (<\/span>var<\/span> token<\/span> in<\/span> tokens<\/span>)<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>tokenWrappers<\/span>.<\/span>Add<\/span>(<\/span>new<\/span> <\/span>TokenWrapper<\/span>(<\/span>token<\/span>,<\/span> <\/span>line<\/span>)<\/span>)<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>wrappedLines<\/span>.<\/span>Add<\/span>(<\/span>new<\/span> <\/span>LineWrapper<\/span>(<\/span>lineIndex<\/span>,<\/span> <\/span>tokenWrappers<\/span>)<\/span>)<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>var<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>new<\/span> <\/span>StringBuilder<\/span>(<\/span>)<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>foreach<\/span> <\/span>(<\/span>var<\/span> <\/span>wrappedLine<\/span> <\/span>in<\/span> <\/span>wrappedLines<\/span>)<\/span> <\/span>\n             <\/span> <\/span> <\/span> <\/span>{<\/span>\n                foreach<\/span> <\/span>(<\/span>var<\/span> <\/span>token<\/span> <\/span>in<\/span> <\/span>wrappedLine<\/span>.<\/span>Tokens<\/span>)<\/span>;<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>string<\/span> <\/span>cssClass<\/span> <\/span>=<\/span> <\/span>TranslateTokenType<\/span>(<\/span>token<\/span>.<\/span>InternalToken<\/span>.<\/span>Type<\/span>)<\/span>;<\/span> <\/span>\n                    if<\/span> (<\/span>cssClass<\/span> !=<\/span> null<\/span>)<\/span>\n                        result<\/span>.<\/span>Append<\/span>(<\/span>$"<span<\/span> class=\\"{cssClass}\\">{token.<\/span>Value<\/span>}<\/span>"<\/span>)<\/span>;<\/span>\n                    else<\/span>\n                        result<\/span>.<\/span>Append<\/span>(<\/span>token<\/span>.<\/span>Value<\/span>)<\/span>;<\/span>\n                }<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n                result<\/span>.<\/span>AppendLine<\/span>(<\/span>)<\/span>;<\/span>\n            }<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n            return<\/span> result<\/span>.<\/span>ToString<\/span>(<\/span>)<\/span>;<\/span>\n        }<\/span><\/span>{...}<\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>\n        private<\/span> static<\/span> string<\/span> TranslateTokenType<\/span>(<\/span>UniversalTokenizer<\/span>.<\/span>TokenType<\/span> tokenType<\/span>)<\/span>\n        {<\/span>\n            switch<\/span> (<\/span>tokenType<\/span>)<\/span>\n            {<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>Keyword<\/span>:<\/span>\n                    return<\/span> "keyword"<\/span>;<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>CharacterLiteral<\/span>:<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>StringLiteral<\/span>:<\/span>\n                    return<\/span> "literal"<\/span>;<\/span>\n                case<\/span> UniversalTokenizer<\/span>.<\/span>TokenType<\/span>.<\/span>Comment<\/span>:<\/span>\n                    return<\/span> "comment"<\/span>;<\/span>\n                default<\/span>:<\/span>\n                    return<\/span> null<\/span>;<\/span>\n            }<\/span>\n        }<\/span><\/span>{...}<\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>class<\/span> <\/span>TokenWrapper<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>internal<\/span> <\/span>readonly<\/span> <\/span>UniversalTokenizer<\/span>.<\/span>Token<\/span> <\/span>InternalToken<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>readonly<\/span> <\/span>string<\/span> <\/span>Value<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>enum<\/span> <\/span>DiffState<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Normal<\/span>,<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Added<\/span>,<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Removed<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>DiffState<\/span> <\/span>State<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>TokenWrapper<\/span>(<\/span>UniversalTokenizer<\/span>.<\/span>Token<\/span> <\/span>token<\/span>,<\/span> <\/span>string<\/span> <\/span>line<\/span>)<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>InternalToken<\/span> <\/span>=<\/span> <\/span>token<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Value<\/span> <\/span>=<\/span> <\/span>token<\/span>.<\/span>GetText<\/span>(<\/span>line<\/span>)<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>State<\/span> <\/span>=<\/span> <\/span>DiffState<\/span>.<\/span>Normal<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>class<\/span> <\/span>LineWrapper<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>readonly<\/span> <\/span>int<\/span> <\/span>Number<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>readonly<\/span> <\/span>List<\/span><<\/span>TokenWrapper<\/span>><\/span> <\/span>Tokens<\/span>;<\/span> <\/span>\n <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>public<\/span> <\/span>LineWrapper<\/span>(<\/span>int<\/span> <\/span>number<\/span>,<\/span> <\/span>List<\/span><<\/span>TokenWrapper<\/span>><\/span> <\/span>tokens<\/span>)<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>{<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Number<\/span> <\/span>=<\/span> <\/span>number<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>Tokens<\/span> <\/span>=<\/span> <\/span>tokens<\/span>;<\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span> <\/span>}<\/span><\/span>{...}<\/span><\/span> <\/span>\n    }<\/span><\/span>{...}<\/span>\n}<\/span><\/span>{...}<\/span>\n\n<\/div><\/div>\t\t\t\r\n<\/div>\r\n\t\t\t\t<\/div>\t\t\t<\/div>\t\t\t
\t\t\tPowered by CodeVROOM<\/a><\/span>\t\t\tOpen in a new window<\/a><\/span>\t\t\t<\/div>\t\t<\/div>\t\t<\/p>\n
Here we used the planning mode<\/a> to automatically ask each model what other code it wanted to see, then pull it into the context window, and re-run the original request with it.<\/p>\n
Here comes the interesting part: on some of the runs, the models would put the newly generated code before the modified method, and the actually confused some models.<\/p>\n
Here’s how it works: models generate new text based on prior text by assigning different importance to different parts of it. By the time LLaMA-3 produced “public string Value =><\/strong>“, it was facing a choice:<\/p>\n