PHP syntax analyzer: RE2C && BISON summary_PHP tutorial-PHP Tutorial-php.cn

Before this, I have tried a project that automatically generates so extensions for our PHP code,

Compiled into PHP, I call it phptoc.

However, due to various reasons, this project was suspended.

I wrote this article firstly because there is too little information in this area, and secondly to summarize what I have learned for future reference. If you can understand PHP syntax analysis

The research on PHP source code will go to a higher level ^.^…

I try to write it as plainly and understandably as possible.

This project idea originated from Facebook’s open source project HipHop.

Actually, I am skeptical about the 50%-60% performance improvement of this project. Fundamentally speaking, if PHP uses APC cache, will its performance be low?

As for HipHop, I haven’t tested it yet, so I can’t say for sure.

PHPtoc, I just want to liberate C programmers, hoping to achieve the goal of allowing PHPer to use PHP code to write an extension close to PHP extension performance,

The process is as follows: read the PHP file, parse the PHP code, perform a syntax analyzer on it, generate the corresponding ZendAPI, and compile it into an extension.

Get to the point

The most difficult thing here is the syntax analyzer. Everyone should know that PHP also has its own syntax analyzer. The current version uses re2c and Bison.

So, naturally I also used this combination.

It is not practical to use PHP's syntax analyzer, because you need to modify zend_language_parser.y and zend_language_scanner.l and recompile, which is not only difficult, but may also affect PHP itself.

So I decided to rewrite a set of my own syntax analysis rules. This function is equivalent to rewriting PHP's syntax analyzer, and of course some uncommon ones will be discarded.

re2c && yacc/bison, by referencing its corresponding files, then compile them into a *.c file, and finally compile it with gcc to generate

Into our own program. Therefore, they are not fundamentally a syntax analysis program. They just generate an independent C text based on our rules

file, this c file is the real syntax analysis program we need. I prefer to call it a syntax generator. As shown below:

Note: a.c in the picture is the final code generated by the scanner. .

re2c scanner, if the scanning rule file we write is called scanner.l, it will scan the content of the PHP file we wrote, and then scan it according to

The rules we wrote generate different tokens and pass them to parse.

The (f)lex grammar rule we wrote, for example, we call it Parse.y

It will be compiled into a parse.tab.h, parse.tab.c file through yacc/bison. Parse performs different operations according to different tokens

For example, our PHP code is “echo 1″;

Scan there is a rule:

"echo" {

return T_ECHO;
}
The scanner function scan will get the "echo 1" string, and it will loop through this piece of code. If it finds an echo string, it will return the token as a keyword: T_ECHO,

parse.y and scanner.l will generate two c files, scanner.c and parse.tab.c respectively, and compile them together with gcc.

I will talk about it in detail below

If you are interested, you can check it out. I have also translated a Chinese version,

It’s not over yet, I will post it later.

re2c provides some macro interfaces, which we use. I simply translated them. My English is not good and there may be errors. If you need the original text, you can go to the address above to view it.

Interface code:
Unlike other scanner programs, re2c does not generate a complete scanner: the user must provide some interface code. Users must define the following macros or other corresponding configurations.
YYCONDTYPE
With -c mode you can use the -to parameter to generate a file: use conditions containing enumeration types. Each value will be used as a condition in the rule set.
YYCTYPE
Used to maintain an input symbol. Usually char or unsigned char.
YYCTXMARKER
*For expressions of type YYCTYPE, the context of the generated code traceback information will be saved in YYCTXMARKER. The user needs to define this macro if the scanner rule requires the use of one or more regular expressions in the context.
YYCURSOR
The expression pointer of type *YYCTYPE points to the currently input symbol, and the generated code is matched as a symbol. At the beginning, YYCURSOR is assumed to point to the first character of the current token. At the end, YYCURSOR will point to the first character of the next token.
YYDEBUG(state,current)
This is only required when the -d flag is specified. It is very easy to debug the generated code when calling user-defined functions.
This function should have the following signature: void YYDEBUG(int state,char current). The first parameter accepts state , with a default value of -1 The second parameter accepts the current position of the input.
YYFILL(n)
When the buffer needs to be filled, the generated code will call YYFILL(n): providing at least n characters. YYFILL(n) will adjust YYCURSOR, YYLIMIT, YYMARKER and YYCTXMARKER as needed. Note that in typical programming languages, n is equal to the length of the longest keyword plus one. Users can define YYMAXFILL once in /*!max:re2c*/ to specify the maximum length. If -1 is used, YYMAXFILL will block once after /*!re2c*/.
YYGETCONDITION()
If -c mode is used, this definition will obtain the condition set before the scanner code. This value must be initialized to the type of the enumeration YYCONDTYPE.
YYGETSTATE()
If -f mode is specified, the user needs to define this macro. If so, in order to get the saved state at the beginning of the scanner, the generated code will call YYGETSTATE(). YYGETSTATE() must return a signed integer. If this value is -1, it tells the scanner that this is the first time. Execution, otherwise this value is equal to the state saved by previous YYSETSTATE(s). Otherwise, the scanner will call YYFILL(n) immediately after resuming operation.
YYLIMIT
The type of expression *YYCTYPE marks the end of the buffer (YYLIMIT(-1) is the last character of the buffer). The generated code will continuously compare YYCORSUR and YYLIMIT to determine when to fill the buffer.
YYSETCONDITION(c)
This macro is used to set conditions in conversion rules. It is only useful when -c mode is specified and conversion rules are used.
YYSETSTATE(s)
The user only needs to define this macro when specifying -f mode. If so, the generated code will call YYSETSTATE(s) before YYFILL(n). The parameter of YYSETSTATE is a signed integer called a unique identifier. A specific YYFILL(n) instance.
YYMARKER
An expression of type *YYCTYPE, the generated code saves the traceback information to YYMARKER. Some simple scanners may not be useful.
The scanner, as the name suggests, scans files to find key codes.

Scanner file structure:

/* #include file*/
/*Macro definition*/
//Scan function
int scan(char *p){
/*Scanner rules area*/
}
//Execute the scan function and return the token to yacc/bison.
int yylex(){
int token;
char *p=YYCURSOR;//YYCURSOR is a pointer pointing to our PHP text content
while(token=scan(p)){//The pointer p will be moved here to determine whether it is the scanner we defined above...
return token;
}
}
int main(int argc,char**argv){
BEGIN(INITIAL);//
YYCURSOR=argv[1];//YYCURSOR is a pointer pointing to our PHP text content,
yyparse();
}
BEGIN is a defined macro

#define YYCTYPE char //Type of input symbol
#define STATE(name) yyc##name
#define BEGIN(n) YYSETCONDITION(STATE(n))
#define LANG_SCNG(v) (sc_globals.v)
#define SCNG LANG_SCNG
#define YYGETCONDITION() SCNG(yy_state)
#define YYSETCONDITION(s) SCNG(yy_state)=s
The yyparse function is defined in yacc,

There is a key macro in it: YYLEX

#define YYLEX yylex()

It will execute the scanner's yylex

It may be a little twisted, so re-tie it:

In scanner.l, by calling parse.y parser function yyparse, this function calls yylex of scanner.l to generate the key code token, yylex

Return the scanner's

The token is returned to parse.y, and parse executes different codes based on different tokens.

Example:

scanner.l
#include "scanner.h"
#include "parse.tab.h"
int scan(char *p){
/*!re2c
" BEGIN(ST_IN_SCRIPTING);
return T_OPEN_TAG;
}
"echo" {

return T_ECHO;
}
[0-9]+ {
return T_LNUMBER;
}
*/
}
int yylex(){
int c;

//      return T_STRING;
         int token;
char *p=YYCURSOR;
while(token=scan(p)){
                    return token;
}
}

int main (int argc,char ** argv){
BEGIN(INITIAL);//Initialization
YYCURSOR=argv[1];//Put the string entered by the user into YYCURSOR
yyparse();//yyparse() -》yylex()-》yyparse()
return 0;
}
Such a simple scanner is made,

What about the parser?

The parsers I use are flex and bison. . .

About the file structure of flex:

%{
/*
The C code segment will be copied verbatim into the C source file generated after lex compilation
You can define some global variables, arrays, function routines, etc...
*/
#include
#include "scanner.h"
extern int yylex();//It is defined in scanner.l. .
void yyerror(char *);
# define YYPARSE_PARAM tsrm_ls
# define YYLEX_PARAM tsrm_ls
%}
{Definition section, where token is defined}
//This is the key. The token program does the switch based on this.
%token T_OPEN_TAG
%token T_ECHO
%token T_LNUMBER
%%
{Rule Section}
start:
T_OPEN_TAG{printf("startn"); }
|start statement
;
statement:
T_ECHO expr {printf("echo :%sn",$3)}
;
expr:
T_LNUMBER {$$=$1;}
%%
{User code snippet}
void yyerror(char *msg){
printf("error:%sn",msg);
}
In the rule section, start is the beginning. If scan recognizes the PHP start tag, it will return T_OPEN_TAG, then execute the code in the brackets and output start.

In scanner.l, the call to scan is a while loop, so it will check to the end of the php code,

yyparse will switch based on the tag returned by scan, and then goto to the corresponding code. For example, yyparse.y finds that the current token is T_OPEN_TAG,

It will be mapped to line 21 corresponding to parse.y through the macro #line, the position of T_OPEN_TAG, and then executed

So, what did TOKEN do after it was returned to yyparse?

In order to be more intuitive, I use gdb tracking:

PHP syntax analyzer: RE2C && BISON summary_PHP tutorial

At this time yychar is 258, what is 258?

PHP syntax analyzer: RE2C && BISON summary_PHP tutorial

258 is the enumeration type data automatically generated by bison.

Continue

YYTRANSLATE macro accepts yychar and returns the corresponding value

#define YYTRANSLATE(YYX) ((unsigned int) (YYX) <= YYMAXUTOK ? yytranslate[YYX] : YYUNDEFTOK)

/* YYTRANSLATE[YYLEX] -- Bison symbol number corresponding to YYLEX. */
static const yytype_uint8 yytranslate[] =
{
       0,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,    27,     2,
      22,    23,     2,     2,    28,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,    21,
       2,    26,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,    24,     2,    25,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     2,     2,     2,     2,
       2,     2,     2,     2,     2,     2,     1,     2,     3,     4,
       5,     6,     7,     8,     9,    10,    11,    12,    13,    14,
      15,    16,    17,    18,    19,    20
};
yyparse拿到这个值，不断地translate，