todparsekit - Project Hosting on Google Code

Last 7 days Dec 31, 2009 issue 16 (Parser gives ambiguous error for malformed grammar file (mis...) reported by joshua.kifer - What steps will reproduce the problem? 1. Load a grammar with a line that doesn't terminate with a semicolon. What is the expected output? What do you see instead? "Semicolon terminator missing on line N in grammar source" Instead I'm seeing:* Assertion failure in -[PKParserFactory didMatchStatement:], /Users/joshua/Workspace/XE/Contrib/ParseKit/PKParserFactory.m:605 What steps will reproduce the problem? 1. Load a grammar with a line that doesn't terminate with a semicolon. What is the expected output? What do you see instead? "Semicolon terminator missing on line N in grammar source" Instead I'm seeing:* Assertion failure in -[PKParserFactory didMatchStatement:], /Users/joshua/Workspace/XE/Contrib/ParseKit/PKParserFactory.m:605 Older Nov 13, 2009 r1411 (adding a macro. not using it yet) committed by todd.ditchendorf - adding a macro. not using it yet adding a macro. not using it yet Nov 06, 2009 issue 15 (Incorrent matches for Lua style comments) Status changed by todd.ditchendorf - owensd, thank you very much for the detailed bug reports. sorry for the delay in responding. i have fixed both issues (very similar to the solution you proposed) and included the fixes in HEAD of trunk. I will be doing a new tag with these fixes soon. thanks again. Todd Ditchendorf Status: Fixed owensd, thank you very much for the detailed bug reports. sorry for the delay in responding. i have fixed both issues (very similar to the solution you proposed) and included the fixes in HEAD of trunk. I will be doing a new tag with these fixes soon. thanks again. Todd Ditchendorf Status: Fixed Nov 05, 2009 r1410 (fix for 2nd issue in "Issue 15 in todparsekit: Incorrent mat...) committed by todd.ditchendorf - fix for 2nd issue in " Issue 15 in todparsekit: Incorrent matches for Lua style comments" reported by owensd http://code.google.com/p/todparsekit/issues/detail?id=15 fix for 2nd issue in " Issue 15 in todparsekit: Incorrent matches for Lua style comments" reported by owensd http://code.google.com/p/todparsekit/issues/detail?id=15 Nov 05, 2009 r1409 (fix for "Issue 15 in todparsekit: Incorrent matches for Lua ...) committed by todd.ditchendorf - fix for " Issue 15 in todparsekit: Incorrent matches for Lua style comments" reported by owensd http://code.google.com/p/todparsekit/issues/detail?id=15 fix for " Issue 15 in todparsekit: Incorrent matches for Lua style comments" reported by owensd http://code.google.com/p/todparsekit/issues/detail?id=15 Oct 14, 2009 issue 15 (Incorrent matches for Lua style comments) commented on by owensd - I replaced the PKTokenizer implementation of nextTokenFromReader:startingWith:tokenizer with this implementation: - (PKToken )nextTokenFromReader:(PKReader )r startingWith:(PKUniChar)cin tokenizer:(PKTokenizer )t { NSParameterAssert(r); NSParameterAssert(t); [self resetWithReader:r]; NSString symbol = [rootNode nextSymbol:r startingWith:cin]; while (symbol.length > 1) { if ([multiLineState.startMarkers containsObject:symbol]) { multiLineState.currentStartMarker = symbol; PKToken tok = [multiLineState nextTokenFromReader:r startingWith:cin tokenizer:t]; if (tok.isComment) { tok.offset = offset; } return tok; } else if ([singleLineState.startMarkers containsObject:symbol]) { singleLineState.currentStartMarker = symbol; PKToken tok = [singleLineState nextTokenFromReader:r startingWith:cin tokenizer:t]; if (tok.isComment) { tok.offset = offset; } return tok; } symbol = [symbol substringToIndex:(symbol.length - 1)]; [r unread:1]; } [r unread:symbol.length - 1]; return [[self nextTokenizerStateFor:cin tokenizer:t] nextTokenFromReader:r startingWith:cin tokenizer:t]; } This fixed my commenting issue. Of course, it doesn't fix the general symbol issue, though that might need to be controlled via a property/option anyway as it might not always be valid. Thanks for the great work on the parser code! =) I replaced the PKTokenizer implementation of nextTokenFromReader:startingWith:tokenizer with this implementation: - (PKToken )nextTokenFromReader:(PKReader )r startingWith:(PKUniChar)cin tokenizer:(PKTokenizer )t { NSParameterAssert(r); NSParameterAssert(t); [self resetWithReader:r]; NSString symbol = [rootNode nextSymbol:r startingWith:cin]; while (symbol.length > 1) { if ([multiLineState.startMarkers containsObject:symbol]) { multiLineState.currentStartMarker = symbol; PKToken tok = [multiLineState nextTokenFromReader:r startingWith:cin tokenizer:t]; if (tok.isComment) { tok.offset = offset; } return tok; } else if ([singleLineState.startMarkers containsObject:symbol]) { singleLineState.currentStartMarker = symbol; PKToken tok = [singleLineState nextTokenFromReader:r startingWith:cin tokenizer:t]; if (tok.isComment) { tok.offset = offset; } return tok; } symbol = [symbol substringToIndex:(symbol.length - 1)]; [r unread:1]; } [r unread:symbol.length - 1]; return [[self nextTokenizerStateFor:cin tokenizer:t] nextTokenFromReader:r startingWith:cin tokenizer:t]; } This fixed my commenting issue. Of course, it doesn't fix the general symbol issue, though that might need to be controlled via a property/option anyway as it might not always be valid. Thanks for the great work on the parser code! =) Oct 14, 2009 issue 15 (Incorrent matches for Lua style comments) commented on by owensd - This is really just a general problem with symbol recognition and falling back. Take this example: NSString s = @"+++\n+++-\n+++-+"; PKTokenizer t = [PKTokenizer tokenizerWithString:s]; [t.symbolState add:@"+++"]; [t.symbolState add:@"+++-+"]; The token output is: (+++) (+) (+) (+) (-) (+++-+) Instead of: (+++) (+++) (-) (+++-+) This is really just a general problem with symbol recognition and falling back. Take this example: NSString s = @"+++\n+++-\n+++-+"; PKTokenizer t = [PKTokenizer tokenizerWithString:s]; [t.symbolState add:@"+++"]; [t.symbolState add:@"+++-+"]; The token output is: (+++) (+) (+) (+) (-) (+++-+) Instead of: (+++) (+++) (-) (+++-+) Oct 14, 2009 issue 15 (Incorrent matches for Lua style comments) reported by owensd - I'm setting up the parser with the following code to generate multi-line Lua-style comments: NSString s = @"--[comment"; PKTokenizer t = [PKTokenizer tokenizerWithString:s]; [t setTokenizerState:t.symbolState from:'/' to:'/']; [t setTokenizerState:t.commentState from:'-' to:'-']; [t.commentState addSingleLineStartMarker:@"--"]; [t.commentState addMultiLineStartMarker:@"--[[" endMarker:@"]]"]; t.commentState.reportsCommentTokens = YES; PKToken eof = [PKToken EOFToken]; PKToken tok = nil; The token output is: (-) (-) ([) (comment) However, since '--' is actually a line comment, I would have expected the token output to be: (--[comment) Am I setting things up incorrectly? I tried to step through the code but I couldn't quite figure out the issue as of yet. I'm setting up the parser with the following code to generate multi-line Lua-style comments: NSString s = @"--[comment"; PKTokenizer t = [PKTokenizer tokenizerWithString:s]; [t setTokenizerState:t.symbolState from:'/' to:'/']; [t setTokenizerState:t.commentState from:'-' to:'-']; [t.commentState addSingleLineStartMarker:@"--"]; [t.commentState addMultiLineStartMarker:@"--[[" endMarker:@"]]"]; t.commentState.reportsCommentTokens = YES; PKToken eof = [PKToken EOFToken]; PKToken tok = nil; The token output is: (-) (-) ([) (comment) However, since '--' is actually a line comment, I would have expected the token output to be: (--[comment) Am I setting things up incorrectly? I tried to step through the code but I couldn't quite figure out the issue as of yet. Oct 11, 2009 r1408 (turning off GC for 1.5-fluid tag) committed by todd.ditchendorf - turning off GC for 1.5-fluid tag turning off GC for 1.5-fluid tag Oct 11, 2009 r1407 (turning off GC for 1.5-fluid tag) committed by todd.ditchendorf - turning off GC for 1.5-fluid tag turning off GC for 1.5-fluid tag Oct 11, 2009 r1406 (creating ParseKit 1.5 tag for use with Fluid/Cruz (so i can ...) committed by todd.ditchendorf - creating ParseKit 1.5 tag for use with Fluid/Cruz (so i can change things) creating ParseKit 1.5 tag for use with Fluid/Cruz (so i can change things) Oct 07, 2009 r1405 (no more a.count. use [a count] instead) committed by todd.ditchendorf - no more a.count. use [a count] instead no more a.count. use [a count] instead Oct 07, 2009 r1404 (no more string.length. use [string length] instead) committed by todd.ditchendorf - no more string.length. use [string length] instead no more string.length. use [string length] instead Sep 26, 2009 r1403 (making some of PKAssembly methods 'friend' methods ) committed by todd.ditchendorf - making some of PKAssembly methods 'friend' methods making some of PKAssembly methods 'friend' methods Sep 25, 2009 r1402 (proj) committed by todd.ditchendorf - proj proj Sep 25, 2009 r1401 (comments) committed by todd.ditchendorf - comments comments Sep 25, 2009 r1400 (cleaning up defines) committed by todd.ditchendorf - cleaning up defines cleaning up defines Sep 25, 2009 r1399 (adding some (unused) arithmetic parser stuff) committed by todd.ditchendorf - adding some (unused) arithmetic parser stuff adding some (unused) arithmetic parser stuff Sep 21, 2009 r1398 (some cleanup) committed by todd.ditchendorf - some cleanup some cleanup Sep 17, 2009 r1397 (tests) committed by todd.ditchendorf - tests tests Sep 17, 2009 r1396 (web) committed by todd.ditchendorf - web web Sep 17, 2009 r1395 (vers) committed by todd.ditchendorf - vers vers Sep 17, 2009 r1394 (Creating 1.5 release branch from -r1392) committed by todd.ditchendorf - Creating 1.5 release branch from -r1392 Creating 1.5 release branch from -r1392 Sep 17, 2009 r1393 (Creating 1.5 release tag from -r1392) committed by todd.ditchendorf - Creating 1.5 release tag from -r1392 Creating 1.5 release tag from -r1392 Sep 17, 2009 r1392 (docs) committed by todd.ditchendorf - docs docs Sep 17, 2009 r1391 (docs) committed by todd.ditchendorf - docs docs Sep 17, 2009 r1390 (web) committed by todd.ditchendorf - web web Sep 17, 2009 r1389 (keynote ) committed by todd.ditchendorf - keynote keynote Sep 17, 2009 r1388 (organizing website) committed by todd.ditchendorf - organizing website organizing website Sep 17, 2009 r1387 (docs) committed by todd.ditchendorf - docs docs Sep 17, 2009 r1386 (rev number) committed by todd.ditchendorf - rev number rev number Sep 17, 2009 r1385 (working on building for 64bit) committed by todd.ditchendorf - working on building for 64bit working on building for 64bit Sep 17, 2009 r1384 (working on building for 64bit) committed by todd.ditchendorf - working on building for 64bit working on building for 64bit Sep 17, 2009 r1383 (working on building for 64bit) committed by todd.ditchendorf - working on building for 64bit working on building for 64bit Sep 17, 2009 r1382 (working on building for 64bit) committed by todd.ditchendorf - working on building for 64bit working on building for 64bit Sep 17, 2009 r1381 (working on blocks api) committed by todd.ditchendorf - working on blocks api working on blocks api Sep 17, 2009 r1380 (working on blocks api) committed by todd.ditchendorf - working on blocks api working on blocks api Sep 17, 2009 r1379 (working on blocks api) committed by todd.ditchendorf - working on blocks api working on blocks api Sep 17, 2009 r1378 (working on blocks api) committed by todd.ditchendorf - working on blocks api working on blocks api Sep 17, 2009 r1377 (working on blocks api) committed by todd.ditchendorf - working on blocks api working on blocks api Sep 17, 2009 r1376 (working on blocks api) committed by todd.ditchendorf - working on blocks api working on blocks api Sep 16, 2009 r1375 ( - modernizing TDTestScaffold) committed by todd.ditchendorf - - modernizing TDTestScaffold - modernizing TDTestScaffold Sep 16, 2009 r1374 (- adding some blocks api to PKTokenizer & tests - modernizin...) committed by todd.ditchendorf - - adding some blocks api to PKTokenizer & tests - modernizing TDTestScaffold - adding some blocks api to PKTokenizer & tests - modernizing TDTestScaffold Sep 15, 2009 r1373 (initialization fixes) committed by todd.ditchendorf - initialization fixes initialization fixes Sep 15, 2009 r1372 (fixing 64bit build issues) committed by todd.ditchendorf - fixing 64bit build issues fixing 64bit build issues Sep 15, 2009 r1371 (fixing 64bit build issues) committed by todd.ditchendorf - fixing 64bit build issues fixing 64bit build issues Sep 15, 2009 r1370 (fixing 64bit build issues) committed by todd.ditchendorf - fixing 64bit build issues fixing 64bit build issues Sep 15, 2009 r1369 (fixing 64bit build issues) committed by todd.ditchendorf - fixing 64bit build issues fixing 64bit build issues Sep 15, 2009 r1368 (fixing 64bit build issues) committed by todd.ditchendorf - fixing 64bit build issues fixing 64bit build issues Sep 15, 2009 r1367 (fixing 64bit build issues) committed by todd.ditchendorf - fixing 64bit build issues fixing 64bit build issues

issue 15 (Incorrent matches for Lua style comments) commented on by owensd - I replaced the PKTokenizer implementation of nextTokenFromReader:startingWith:tokenizer with this implementation: - (PKToken *)nextTokenFromReader:(PKReader *)r startingWith:(PKUniChar)cin tokenizer:(PKTokenizer *)t { NSParameterAssert(r); NSParameterAssert(t); [self resetWithReader:r]; NSString *symbol = [rootNode nextSymbol:r startingWith:cin]; while (symbol.length > 1) { if ([multiLineState.startMarkers containsObject:symbol]) { multiLineState.currentStartMarker = symbol; PKToken *tok = [multiLineState nextTokenFromReader:r startingWith:cin tokenizer:t]; if (tok.isComment) { tok.offset = offset; } return tok; } else if ([singleLineState.startMarkers containsObject:symbol]) { singleLineState.currentStartMarker = symbol; PKToken *tok = [singleLineState nextTokenFromReader:r startingWith:cin tokenizer:t]; if (tok.isComment) { tok.offset = offset; } return tok; } symbol = [symbol substringToIndex:(symbol.length - 1)]; [r unread:1]; } [r unread:symbol.length - 1]; return [[self nextTokenizerStateFor:cin tokenizer:t] nextTokenFromReader:r startingWith:cin tokenizer:t]; } This fixed my commenting issue. Of course, it doesn't fix the general symbol issue, though that might need to be controlled via a property/option anyway as it might not always be valid. Thanks for the great work on the parser code! =)

I replaced the PKTokenizer implementation of nextTokenFromReader:startingWith:tokenizer with this implementation: - (PKToken *)nextTokenFromReader:(PKReader *)r startingWith:(PKUniChar)cin tokenizer:(PKTokenizer *)t { NSParameterAssert(r); NSParameterAssert(t); [self resetWithReader:r]; NSString *symbol = [rootNode nextSymbol:r startingWith:cin]; while (symbol.length > 1) { if ([multiLineState.startMarkers containsObject:symbol]) { multiLineState.currentStartMarker = symbol; PKToken *tok = [multiLineState nextTokenFromReader:r startingWith:cin tokenizer:t]; if (tok.isComment) { tok.offset = offset; } return tok; } else if ([singleLineState.startMarkers containsObject:symbol]) { singleLineState.currentStartMarker = symbol; PKToken *tok = [singleLineState nextTokenFromReader:r startingWith:cin tokenizer:t]; if (tok.isComment) { tok.offset = offset; } return tok; } symbol = [symbol substringToIndex:(symbol.length - 1)]; [r unread:1]; } [r unread:symbol.length - 1]; return [[self nextTokenizerStateFor:cin tokenizer:t] nextTokenFromReader:r startingWith:cin tokenizer:t]; } This fixed my commenting issue. Of course, it doesn't fix the general symbol issue, though that might need to be controlled via a property/option anyway as it might not always be valid. Thanks for the great work on the parser code! =)

Last 7 days

Older