Branch: Tag:


2013-04-21 12:36:58 by Henrik Grubbström (Grubba) <>

Merge pike-modules/Tokenizer into grubba/tokenizer

This merges the old experimental Tokenizer module into current Pike 7.9.

* /home/grubba/src/cvsmigration/Tokenizer: (72 commits)


2008-05-29 10:13:57 by Henrik Grubbström (Grubba) <>

Object subtype indexing mega patch.
Added inherit_number argument to the high-level object indexing functions.
Subtyped objects should now be usable in many cases in Pike code.
Updated several modules to the new API.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.41


2008-05-26 09:07:36 by Henrik Grubbström (Grubba) <>

Moved definition of yyerror macro so as to not conflict with the header files.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.40
Rev: src/modules/Tokenizer/acconfig.h:1.2


2008-05-17 14:51:06 by Henrik Grubbström (Grubba) <>

Updated to compile with low_sscanf() in Pike 7.7 taking flags.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.39


2008-04-26 13:45:06 by Henrik Grubbström (Grubba) <>

Updated to Pike 7.7's naming convention.
handle_pike_compat() ==> change_compiler_compatibility().

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.38


2006-10-03 16:47:34 by Henrik Grubbström (Grubba) <>

Implemented BreakNoLabel in C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.37
Rev: src/modules/Tokenizer/

2006-10-03 16:29:59 by Henrik Grubbström (Grubba) <>

Code cleanups and added diagnostics.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.36


2006-09-30 14:33:47 by Henrik Grubbström (Grubba) <>

Implemented more stuff in C. Removed some pike code that had already been implemented in C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.35
Rev: src/modules/Tokenizer/


2004-12-16 16:05:56 by Henrik Grubbström (Grubba) <>

Added get_{end_token,contents}().

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.34


2004-12-10 15:13:36 by Henrik Grubbström (Grubba) <>

Fixed typo in BraceGroup.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.33


2004-12-09 17:30:37 by Henrik Grubbström (Grubba) <>

Renamed scan() to next() and updated return value accordingly.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.32


2004-12-03 15:26:28 by Henrik Grubbström (Grubba) <>

Changed all simple_add_constant(X, Y) to add_constant(MK_STRING(X), Y).
Made Group()->terminator a constant.
Added {Paren,Brace,Bracket}Group.
Fixed lookup of grouping_compiler_inh_report.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.31

2004-12-03 13:47:49 by Henrik Grubbström (Grubba) <>

Group()->_sprintf() now shows the start and end token.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.30


2004-11-28 19:38:34 by Henrik Grubbström (Grubba) <>

Reduced verbosity of Token()->_sprintf().
Fixed initialization of Group().
GroupingCompiler()->group() now seems to work.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.29


2004-11-27 16:33:22 by Henrik Grubbström (Grubba) <>

Optimized GroupingCompiler()->group() somewhat by caching the lookup of the append function.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.28


2004-11-26 19:19:20 by Henrik Grubbström (Grubba) <>

Cleaned up grouping API.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.27


2004-11-25 16:30:42 by Henrik Grubbström (Grubba) <>

GroupingCompiler now implemented in C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.26


2004-10-29 15:51:39 by Henrik Grubbström (Grubba) <>

Added class Group.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.25


2004-09-07 15:43:55 by Henrik Grubbström (Grubba) <>


Rev: src/modules/Tokenizer/Tokenizer.cmod:1.24


2004-09-03 12:46:43 by Henrik Grubbström (Grubba) <>

Fixed bug in handle().

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.23


2004-09-02 14:35:28 by Henrik Grubbström (Grubba) <>

Cleanups due to new precompile.pike.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.22


2004-07-08 14:02:43 by Henrik Grubbström (Grubba) <>

Improved f_werror() replacement.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.21

2004-07-08 13:08:27 by Henrik Grubbström (Grubba) <>

f_werror() is no more.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.20


2004-04-01 12:37:06 by Henrik Grubbström (Grubba) <>

Fixed a few typos.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.19


2004-03-25 15:34:20 by Henrik Grubbström (Grubba) <>

Some OSes (WIN32) define ERROR, som the SL_CASE() macro doesn't work.
Fixed by expanding the macro.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.18


2003-12-28 15:51:22 by Henrik Grubbström (Grubba) <>

Now with some handling of pragmas.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.17


2003-12-21 17:17:29 by Henrik Grubbström (Grubba) <>

Various changes with regard to position and context handling.
* Joined Position and Context handling into a single mapping.
* As a consequence report() at all levels now takes a mapping
as the second argument.
* compiler()->basic_tokenizer()->line_no is no more.
* Added compiler()->basic_tokenizer()->context to hold context information.
* Added compiler()->basic_tokenizer()->{get,set}_context() to simplify
handling of the context mapping.
* Got rid of extraneous INIT and EXIT code.
* Shortened several variable names that were longer than 40 characters.
* Documented some inherits.
* TokenizingCompiler()->pike_tokenizer()->handle_pike_compat() is no longer
just a prototype. Allthough it currently just sets the context.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.16


2003-12-18 17:46:18 by Henrik Grubbström (Grubba) <>

Now uses MK_STRING().

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.15


2003-12-13 17:25:06 by Henrik Grubbström (Grubba) <>

Added class Position.
Added C implementations of handle() and scan() in TokenizingCompiler()->pike_tokenizer.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.14


2003-12-12 14:47:50 by Henrik Grubbström (Grubba) <>

Added convenience macro APPLY_SELF().
Implemented yyerror(), handle_pragma(), parse_string() and preprocess() in TokenizingCompiler()->pike_tokenizer in C.
Named the SeverityLevel type.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.13


2003-12-02 12:39:11 by Henrik Grubbström (Grubba) <>

Fixed inherit of program in parent.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.12


2003-12-01 14:33:03 by Henrik Grubbström (Grubba) <>

Various fixes due to new naming conventions.
Started implementing TokenizingCompiler in C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.11
Rev: src/modules/Tokenizer/


2003-11-29 13:23:23 by Henrik Grubbström (Grubba) <>

Now uses the new PIKE_MODULE{INIT,EXIT} syntax.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.10


2003-11-28 16:49:38 by Henrik Grubbström (Grubba) <>

{Literal,String,Integer,Float}Token are now implemented in C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.9
Rev: src/modules/Tokenizer/

2003-11-28 14:33:10 by Henrik Grubbström (Grubba) <>

Moved some more stuff from Pike to C.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.8
Rev: src/modules/Tokenizer/


2003-11-27 16:49:32 by Henrik Grubbström (Grubba) <>

The basic token nolonger holds a value.
The field "token" in a token is now a constant.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.7


2003-07-24 14:40:48 by Henrik Grubbström (Grubba) <>

Fixed AutoDoc markup for SeverityLevel.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.6


2003-04-12 13:02:08 by Henrik Grubbström (Grubba) <>

Tokenizer.Token()->_sprintf() now looks at its first argument.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.5


2003-01-18 15:07:52 by Henrik Grubbström (Grubba) <>

Now uses new features in precompile.pike.
Added report() in basic_tokenizer to allow for easier overloading.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.4
Rev: src/modules/Tokenizer/


2003-01-17 17:36:09 by Henrik Grubbström (Grubba) <>

Indentation fix.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.3

2003-01-17 16:46:28 by Henrik Grubbström (Grubba) <>

Added extra level of indirection.
Now uses parent pointer to access the report function. Modified the API accordingly.

Rev: src/modules/Tokenizer/Tokenizer.cmod:1.2
Rev: src/modules/Tokenizer/
Rev: src/modules/Tokenizer/toktest.pike:1.2


2002-10-17 10:43:58 by Henrik Grubbström (Grubba) <>

First version.

Rev: src/modules/Tokenizer/.cvsignore:1.1
Rev: src/modules/Tokenizer/
Rev: src/modules/Tokenizer/MetaProg.yacc:1.1
Rev: src/modules/Tokenizer/ParseTree.c:1.1
Rev: src/modules/Tokenizer/Pike.yacc:1.1
Rev: src/modules/Tokenizer/Pike2.yacc:1.1
Rev: src/modules/Tokenizer/Tokenizer.cmod:1.1
Rev: src/modules/Tokenizer/acconfig.h:1.1
Rev: src/modules/Tokenizer/
Rev: src/modules/Tokenizer/misc.c:1.1
Rev: src/modules/Tokenizer/
Rev: src/modules/Tokenizer/
Rev: src/modules/Tokenizer/tokenizer.h:1.1
Rev: src/modules/Tokenizer/toktest.pike:1.1