c_pp.pl
The C preprocessor (milestone 4e): it expands a C file's macros, includes and conditionals and hands the result to c_lex as tokens.
Preprocessing tokens
A preprocessing token is tSpelling, File:Line, Space, Hide:
| Part | Is |
|---|---|
| Spelling | the token's text, as an atom |
File:Line | its location |
| Space | yes if whitespace precedes the token, else no; # and #define need it |
| Hide | the hide set: the sorted macro names that must not expand this token again |
A newline is the token nl until the input is split into lines.
The environment
The environment is envMacros, IncludeDirs, Features, Once, File:
| Part | Is |
|---|---|
| Macros | an assoc from macro names to objBody or funParams, Body, and from savedName to the #pragma push_macro stack of Name |
| IncludeDirs | the include search path |
| Features | the target's answers to feature operators |
| Once | the files seen by #pragma once |
| File | the file whose directives are being processed |
Only the file and target-profile adapters do I/O. No C source or header is sent to clang.
Tokens are the c_lex tokens of File after preprocessing, each at the location of the preprocessing token it came from. Profile is profile(Macros, IncludeDirs, FeatureAnswers) from pp_profile/2. Throws nanofil_error/1 on a preprocessing or lexical error.
the c_lex tokens of PpTokens. Each preprocessing token is lexed on its own and keeps its location.
File's expanded tokens. Env carries the macros and the #pragma once set back to the includer.
PpTokens are File's preprocessing tokens, before line splitting. Impure file adapter. Keep the input buffer local so it can be reclaimed before processing nested includes; the macro environment stays explicit. A file's tokens do not depend on the macros, so each file is read once per pp_file/3: headers are included again and again (lvm.c reads 124 files 272 times), and reading was most of the preprocessor's time.
source_cache(?File, ?PpTokens): read_source/2's tokens of File, for the pp_file/3 call in progress.
Translation phases 2 and 3
PpTokens are the preprocessing tokens of Chars, which start at Loc. A comment is one whitespace character, even across physical newlines. A parallel location vector survives splicing even inside identifiers and literals. In particular __LINE__ after a continuation is the physical source line, not the number of newlines in the spliced buffer.
remove backslash-newline and give each remaining character its physical location.
Share one location term per physical line, not one per character.
Locs runs in step with the input characters; Space says whether whitespace precedes the next token.
the comment's characters, leaving the newline; Rest is Locs without their locations.
like line_comment//2, through the */.
Chars is the spelling of one preprocessing token, the longest that matches.
Lines and directives
Lines are PpTokens split at each nl, the nl tokens dropped.
Line is PpTokens up to the first nl, and Rest follows it.
like tokens_line_rest/3 for [T|Ts]; T is nl or a t/4 token.
Ordinary lines are collected in Pending (reversed) and expanded together, so a function invocation can span newlines. Directives are barriers; inactive groups never expand tokens.
for #line N ["file"] at HashLoc, the line after the directive is line N (of file), Target.
File is named by the optional "file" operand Tokens of #line, else File0.
the tokens of Lines, which follow the directive at Old, renumbered so that the first is at New.
Ts are the expanded tokens of the reversed list of lines Pending.
one directive line after its '#'. Conditionals are tracked even in inactive groups; other directives only run in active ones. Ts is Rest with an included file's tokens in front.
Stack is the conditional stack after the directive #Name Args.
Stack is Stack0 with the group that #Kind Args opens.
in an active parent, #elif Args is taken if no earlier group was and its condition holds.
run #Name Args in an active group; Ts is Rest with an included file's tokens in front. Name is any identifier in the source, so the last clause is a real default case: the cuts commit to a known directive.
#pragma once, push_macro and pop_macro; other pragmas are ignored. Macro save stacks use compound assoc keys, which cannot collide with C identifier atoms. Pragma operands are deliberately not macro-expanded.
Macros is Macros0 after #pragma Op("Name"). A saved entry is some(Def) or none (undefined); a pop with nothing saved changes nothing.
Macros is Macros0 with Name as #pragma push_macro saved it, some(Def) or none.
Macros is Macros0 without Name, which need not be defined.
Macros is Macros0 with the definition Tokens of #define. A macro is function-like if '(' follows its name without a space.
named(Name) for each name up to ')'; a final ... (or P...) is variadic(Name).
Includes
"name" (quote) or <name> (angle).
the header name of an #include or __has_include operand. The operand is expanded only if it is not already a header name.
Path is the first existing file for header Name included from File: an absolute Name, then (for quote) File's directory, then Dirs in order. Each cut commits to the first existing file, so at most one Path is found.
#include_next and __has_include_next search the directories after the one File is in. Kinds include and __has_include also match the last clause, so a retry yields a second Search (the directories after File's, if any).
Macro expansion
Macro replacement uses token hide sets rather than a global recursion stack: an unexpandable token stays unexpandable after argument prescan, and a replacement may consume a function-call '(' from the input tail. A function-like replacement's hide set is the intersection of the hide sets of the macro name and of the closing ')', plus the name. tokens_expansion(+Ts, +Env, -Out): Out is Ts fully macro-expanded.
BaseHide is the hide set Hide of a macro name, intersected with that of a function-like invocation's ')' if EndHide is some(Set).
Cs are the elements of As in Bs.
A replacement token moves to the invocation and gains its hide set.
EndHide is some(Set), Set the hide set of a function-like invocation's ')', or none for an object-like macro.
the tokens after '(' split at top-level commas, up to the matching ')' whose hide set is EndHide.
bind(P, Raw, Env) for each parameter; a variadic one takes the remaining arguments.
like params_bindings/5 for [Param|Params]. Only the last can be variadic.
Ts are the token lists Args joined by commas.
Ts is A, then each of Args after a comma.
AfterPaste is yes right after ##. # P stringifies the raw argument, an operand of ## is the raw argument, and any other use is the fully expanded argument; an empty argument becomes the placemarker ''.
A is the string literal spelling of Ts, as # makes it.
apply ## and drop placemarkers. A pasted spelling must be exactly one preprocessing token.
T, applied to a prefix of Ts that leaves Rest, is a builtin operator replaced by Repl. Builtin operators are not macros in clang's -dM output. Feature answers are discovered once with the target profile, not guessed from __GNUC__.
Conditions
condition(+Kind, +Args, +Env, -Value) for #if, #ifdef and #ifndef.
defined X and defined(X) become 1 or 0 before macro expansion.
the c_lex token of a #if operand or operator. A name left after expansion, keyword or not, is 0.
Target profile
Profile is profile(Macros, IncludeDirs, FeatureAnswers) for the target of the clang executable Clang: its predefined macros, its #include <...> search path and its answers to feature operators. Impure adapter: discover the selected target, never preprocess the input.
This avoids embedding Nix store paths or a second, stale copy of the pin's 400+ predefined macros. Pipes are drained before waiting for the child.
run Command with Args; Chars is its standard output. Fails unless it exits with status 0.
Paths are the directories listed after clang -v's "#include <...>" line; fails if there is none.
Batch the feature names used by glibc. The adapter uses a shell pipeline because this Trealla revision leaves the child's own stdin writer open with stdin(pipe(...)); closing the parent stream cannot deliver EOF.
A is the spelling of Op(Name), asking clang if the profile has no answer for it. This is target introspection only: pass an operator and a single name, never a source expression, file, declaration or include to clang.
Answers are clang's answer(Op, Name, A) for each Op-Name in Pairs.
Quoted is A in single quotes for /bin/sh.