Branch: Tag:


2013-04-21 12:36:58 by Henrik Grubbström (Grubba) <>

Merge pike-modules/Tokenizer into grubba/tokenizer

This merges the old experimental Tokenizer module into current Pike 7.9.

* /home/grubba/src/cvsmigration/Tokenizer: (72 commits)


2006-10-03 16:47:34 by Henrik Grubbström (Grubba) <>

Implemented BreakNoLabel in C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.37
Rev: src/modules/Tokenizer/


2006-09-30 14:33:47 by Henrik Grubbström (Grubba) <>

Implemented more stuff in C. Removed some pike code that had already been implemented in C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.35
Rev: src/modules/Tokenizer/


2004-12-16 16:22:02 by Henrik Grubbström (Grubba) <>

Removed or replaced with "->is_group" all use of "->group_start" as appropriate.
Replaced all use of "->contents" with "->get_contents()".
Note: There are now lots of redundant ->is_group checks.
Note: May want to cache the return value from ->get_contents() is several places.

Rev: src/modules/Tokenizer/


2004-12-03 15:32:12 by Henrik Grubbström (Grubba) <>

Changed all ->`+=(1) to ->next().
Moved Scope class.
Separated parse_meta_identifier() to parse_meta_function().
Updated for the new *Group classes.
Added For and Sscanf meta programs.

Rev: src/modules/Tokenizer/


2004-11-28 19:39:24 by Henrik Grubbström (Grubba) <>

Updated to the new Group/EndGroup API.

Rev: src/modules/Tokenizer/


2004-09-08 19:20:58 by Henrik Grubbström (Grubba) <>

Added Case.

Rev: src/modules/Tokenizer/


2004-09-07 17:01:30 by Henrik Grubbström (Grubba) <>

Added meta programs for Dot and ColonColon.
Added support for adjacent string concat.

Rev: src/modules/Tokenizer/

2004-09-07 16:15:12 by Henrik Grubbström (Grubba) <>

Added a lot of symbols to Pike_0_1_Tokens.

Rev: src/modules/Tokenizer/


2004-09-04 19:10:35 by Henrik Grubbström (Grubba) <>

Added TokenList()->_sizeof().
Iterator(Tokenlist)->insert() no longer advances the current position.
Various parser fixes.
Added some debug code.
Splited parse_meta_program() into two, with the second being parse_meta_identifier().
Added support for parsing of soft casts, splice and return.
Improved several error messages.

Rev: src/modules/Tokenizer/


2004-09-03 12:04:03 by Henrik Grubbström (Grubba) <>

Various parser fixes.

Rev: src/modules/Tokenizer/


2003-12-28 15:50:03 by Henrik Grubbström (Grubba) <>

Updated TokenHandler type to confirm to the new report() API.
Removed Pike code for classes now implemented in the cmod.
Updated various metaprograms to know about the pos ==> context change.
Now compiles again.

Rev: src/modules/Tokenizer/


2003-12-01 14:33:03 by Henrik Grubbström (Grubba) <>

Various fixes due to new naming conventions.
Started implementing TokenizingCompiler in C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.11
Rev: src/modules/Tokenizer/


2003-11-28 16:49:38 by Henrik Grubbström (Grubba) <>

{Literal,String,Integer,Float}Token are now implemented in C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.9
Rev: src/modules/Tokenizer/

2003-11-28 14:33:10 by Henrik Grubbström (Grubba) <>

Moved some more stuff from Pike to C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.8
Rev: src/modules/Tokenizer/


2003-11-27 16:50:03 by Henrik Grubbström (Grubba) <>

Updated for the new basic token.

Rev: src/modules/Tokenizer/


2003-10-31 14:56:59 by Henrik Grubbström (Grubba) <>

Added {insert,append}() to the TokenList iterator.
Fixed typo.
Added parsing of various literal types.
Added Break and BreakNoLabel.

Rev: src/modules/Tokenizer/


2003-10-07 14:43:19 by Henrik Grubbström (Grubba) <>

Changed Scope interface.

Rev: src/modules/Tokenizer/


2003-09-05 16:44:02 by Henrik Grubbström (Grubba) <>

Now utilizes the new foreach behaviour.

Rev: src/modules/Tokenizer/

2003-09-05 16:25:51 by Henrik Grubbström (Grubba) <>

Added parsing of namespace directives.

Rev: src/modules/Tokenizer/

2003-09-05 12:41:47 by Henrik Grubbström (Grubba) <>

Added lambda.

Rev: src/modules/Tokenizer/


2003-07-23 16:00:42 by Henrik Grubbström (Grubba) <>

Added a few more @ignore.

Rev: src/modules/Tokenizer/


2003-07-03 14:57:47 by Henrik Grubbström (Grubba) <>

Added parsing of if and else.
Removed some obsolete code.
Added some provisions for code generation.

Rev: src/modules/Tokenizer/


2003-06-01 15:33:43 by Henrik Grubbström (Grubba) <>

Most operators are now mapped to calls of their predef variants.

Rev: src/modules/Tokenizer/


2003-05-31 17:10:49 by Henrik Grubbström (Grubba) <>

Added parsing of foreach.

Rev: src/modules/Tokenizer/

2003-05-31 15:55:23 by Henrik Grubbström (Grubba) <>

Added support for implicit lambdas.
Fixed parsing of the ..-operator.

Rev: src/modules/Tokenizer/


2003-05-30 14:42:34 by Henrik Grubbström (Grubba) <>

Added some support for scopes, and variable declarations.

Rev: src/modules/Tokenizer/


2003-04-27 15:44:01 by Henrik Grubbström (Grubba) <>

Changed semantics of parse_decl() somewhat.

Rev: src/modules/Tokenizer/


2003-04-26 16:23:27 by Henrik Grubbström (Grubba) <>

Some support for parsing of statements.

Rev: src/modules/Tokenizer/


2003-04-20 16:21:55 by Henrik Grubbström (Grubba) <>

Large parts of the expression parser seems to work.

Rev: src/modules/Tokenizer/


2003-04-17 15:19:34 by Henrik Grubbström (Grubba) <>

Begun support for parsing of expressions.

Rev: src/modules/Tokenizer/


2003-04-13 19:33:15 by Henrik Grubbström (Grubba) <>

More token lookup fixes.

Rev: src/modules/Tokenizer/


2003-04-12 15:06:52 by Henrik Grubbström (Grubba) <>

Now knows more about when reserved words showed up.

Rev: src/modules/Tokenizer/

2003-04-12 13:03:11 by Henrik Grubbström (Grubba) <>

Lots of changes. Now has the beginnings of a metaprogram based compiler.

Rev: src/modules/Tokenizer/


2003-01-18 15:07:52 by Henrik Grubbström (Grubba) <>

Now uses new features in precompile.pike.
Added report() in basic_tokenizer to allow for easier overloading.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.4
Rev: src/modules/Tokenizer/


2003-01-17 16:46:28 by Henrik Grubbström (Grubba) <>

Added extra level of indirection.
Now uses parent pointer to access the report function. Modified the API accordingly.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.2
Rev: src/modules/Tokenizer/
Rev: src/modules/Tokenizer/toktest.pike:1.2


2002-10-17 13:32:35 by Henrik Grubbström (Grubba) <>

Fixed type.

Rev: src/modules/Tokenizer/

2002-10-17 10:43:58 by Henrik Grubbström (Grubba) <>

First version.

Rev: src/modules/Tokenizer/.cvsignore:1.1
Rev: src/modules/Tokenizer/
Rev: src/modules/Tokenizer/MetaProg.yacc:1.1
Rev: src/modules/Tokenizer/ParseTree.c:1.1
Rev: src/modules/Tokenizer/Pike.yacc:1.1
Rev: src/modules/Tokenizer/Pike2.yacc:1.1
Rev: src/modules/Tokenizer/Tokenizer.cmod:1.1
Rev: src/modules/Tokenizer/acconfig.h:1.1
Rev: src/modules/Tokenizer/
Rev: src/modules/Tokenizer/misc.c:1.1
Rev: src/modules/Tokenizer/
Rev: src/modules/Tokenizer/
Rev: src/modules/Tokenizer/tokenizer.h:1.1
Rev: src/modules/Tokenizer/toktest.pike:1.1