cobol: Maintain lexer state after CDF.
Improves COBOL parsing by ensuring lexer state is preserved after handling directives, fixing implicit PICTURE recognition.
This commit enhances the COBOL parser’s ability to maintain lexer state when processing Compiler-Directing Facilities (CDF) lookahead tokens. Previously, if a CDF token was not a lookahead token, it was reprocessed, sometimes leading to incorrect NAME recognition. The changes ensure that the lexer state is correctly managed, fixing an issue where a CDF directive could prevent the recognition of an implicit PICTURE clause. Debug messages and internal logic have also been tidied.
In Details
When a CDF lookahead token in the COBOL frontend is not treated as a lookahead token itself, it is reprocessed by the main parser. This commit corrects the handling of such tokens, specifically when they are recognized as NAME and require lookup in the token-word list. It ensures that the lexer's state is preserved across this reprocessing. Additionally, a bug where a CDF directive interfered with implicit PICTURE clause recognition has been fixed. Changes are made in parse_ante.h, scan_post.h, symbols.cc, symbols.h, and util.cc.
- CDF
- Compiler-Directing Facility. In COBOL, these are statements that control how the compiler processes the source code, often used for conditional compilation or external data definitions.
- lexer state
- The internal information maintained by the lexer (tokenizer) that describes the current context of the input stream, such as the current token being processed, line number, and any active parsing modes.
- PICTURE clause
- A fundamental COBOL clause used to define the data type, size, and format of a data item, analogous to a type declaration in other languages.
- token-word list
- A lookup table or data structure within the COBOL compiler's parser that maps sequences of characters (tokens) to reserved words or identifiers.