c_lex.pl
The C lexer (milestone 4a): it turns preprocessed C text into a list of tokens, each with its file and line.
Input
The input is preprocessed text. A line marker # N "file" or #line N "file" sets the location of the next line; any other directive line, such as #pragma or #ident, is skipped. Comments are skipped too, so hand-written test input works without a preprocessor.
Tokens
A token is tokT, File:Line, with T one of:
| T | Is |
|---|---|
idAtom | an identifier |
kwAtom | a keyword; GNU spellings are folded, so __restrict__ is kwrestrict and __asm__ is kwasm |
intValue, Suffix−Radix | an integer constant: Suffix is none, u, l, ul, ll or ull; Radix is dec, or other for hex and octal, which type differently |
floatChars | a floating constant, its characters as written |
chrValue | a character constant, as an int |
strBytes | a string literal, its bytes without the terminator (u8"..." too, as UTF-8 bytes) |
wstrType, Codes | L"...", u"..." or U"...": Type is the element type, int (wchar_t), ushort (char16_t) or uint (char32_t), and Codes the elements without the 0 |
pAtom | a punctuator, such as p(->) or p<<= |
Lines, blanks and comments
Tokens are the tokens of the rest of the input, which starts at Loc. Where is bol at the beginning of a line, where # starts a directive, and mid elsewhere.
the rest of a comment after its /*; Loc is Loc0 advanced past the comment's newlines.
a directive line after its #; Loc is the location of the line that ends it. A line marker # N "file" flags or #line N "file" says the next line is line N of file. Any other directive is skipped.
Character classes as ASCII ranges: char_type/2's classes differ between Prolog hosts, and C and IR source only need these.
Tokens
one token, the T of a tok(T, Loc) term. Wide and u8 prefixes are tried before identifiers, which they would start.
Numbers
T is the int or float token of the pp-number that starts with the already-read characters Prefix.
T is int(Value, Suffix-Radix) or float(Chars) for the pp-number Chars; anything else is an error.
Value is the number Digits spell in Base.
Suffix names the rest of the pp-number, an integer suffix in either case and order (u, l, ll, ul, ull or none). It fails if the rest is not such a suffix.
Only the characters are checked here; the lowering rejects floats.
Character and string bodies: lists of byte values.
Bytes is the literal body up to the closing Quote, with escapes decoded and other characters encoded as UTF-8.
Byte is the value of an escape after its backslash, reduced to 8 bits.
Wide literals: elements are code points, not UTF-8 bytes.
Digits are exactly N hexadecimal digits.
Value is the single element of a wide character constant, as an int; wchar_t is int here.
like wide_char_value/4 for [V|Codes].
Value is the single byte of a character constant, as a (signed) char promoted to int. A multi-character constant is rejected.
Bytes is the UTF-8 encoding of Code.
Punctuators, longest first.
Spelling is one of C's punctuators of two or three characters (C11 6.4.6), digraphs aside; the one-character ones are punct//1's.
Spelling is a digraph (C11 6.4.6p3), a preprocessing token that stands for [, ], {, } or #.
Keywords, with GNU alternate spellings folded.
Keyword is the kw/1 name of the C or GNU keyword spelled Spelling.
| 511 | auto | auto |
| 512 | break | break |
| 513 | case | case |
| 514 | char | char |
| 515 | const | const |
| 516 | continue | continue |
| 517 | default | default |
| 518 | do | do |
| 519 | double | double |
| 520 | else | else |
| 521 | enum | enum |
| 522 | extern | extern |
| 523 | float | float |
| 524 | for | for |
| 525 | goto | goto |
| 526 | if | if |
| 527 | inline | inline |
| 528 | int | int |
| 529 | long | long |
| 530 | register | register |
| 531 | restrict | restrict |
| 532 | return | return |
| 533 | short | short |
| 534 | signed | signed |
| 535 | sizeof | sizeof |
| 536 | static | static |
| 537 | struct | struct |
| 538 | switch | switch |
| 539 | typedef | typedef |
| 540 | union | union |
| 541 | unsigned | unsigned |
| 542 | void | void |
| 543 | (volatile) | (volatile) |
| 544 | while | while |
| 545 | '_Alignas' | alignas |
| 546 | '_Alignof' | alignof |
| 547 | '_Atomic' | atomic |
| 548 | '_Bool' | bool |
| 549 | '_Complex' | complex |
| 550 | '_Generic' | generic |
| 551 | '_Imaginary' | imaginary |
| 552 | '_Noreturn' | noreturn |
| 553 | '_Static_assert' | static_assert |
| 554 | '_Thread_local' | (thread_local) |
| 555 | '__restrict' | restrict |
| 556 | '__restrict__' | restrict |
| 557 | '__inline' | inline |
| 558 | '__inline__' | inline |
| 559 | '__const' | const |
| 560 | '__const__' | const |
| 561 | '__volatile' | (volatile) |
| 562 | '__volatile__' | (volatile) |
| 563 | '__signed' | signed |
| 564 | '__signed__' | signed |
| 565 | asm | asm |
| 566 | '__asm' | asm |
| 567 | '__asm__' | asm |
| 568 | '__attribute' | attribute |
| 569 | '__attribute__' | attribute |
| 570 | '__extension__' | extension |
| 571 | '__typeof' | typeof |
| 572 | '__typeof__' | typeof |
| 573 | typeof | typeof |
| 574 | '__alignof' | alignof |
| 575 | '__alignof__' | alignof |
| 576 | '__int128' | int128 |
| 577 | '__thread' | (thread_local) |
throw a nanofil_error/1 located at Loc.