EFUNs: Added Pike 8.0 compat variant of sscanf and array_sscanf().
Stdio.Buffer: read_buffer() now propagates the error mode.
Build: Fix warning about signed/unsigned comparison.
Shuffler: PIKE_T_FREE not in PIKEVARs.
Stdio.Buffer: Fix docs.
Stdio.Buffer: Remove stale code.
Compiler [Typechecker]: Fixup the type for classes before end_first_pass().
Stdio.Buffer: Swap buffer pointers and improve docs.
Stdio.Buffer: Simplify code.
Revert "Stdio.Buffer: Reduce memory footprint some more." This reverts commit e366d20b4f3c11ffb01dc9ee2535b93517eacc1c.
Stdio.Buffer: Clarify code and slightly slim down struct.
Stdio.Buffer: Make checks on io->output robust.
Stdio.Buffer: Reduce memory footprint some more.
Stdio.Buffer: Do not optimise the buffer to empty if there are subbuffers.
Fix typos.
Stdio.Buffer: Allow appending to locked buffers.
Stdio.Buffer: Autocopy it when reallocing a locked buffer.
Stdio.Buffer: Decrease memory footprint for non PIKE_DEBUG.
Stdio.Buffer: Reduce run-time argument strictness in _search(). Searching for values that aren't possible isn't an error, so avoid the precompiler generating errors for such parameters. Fixes testsuite failure searching for character -1.
Return -1 when failing to find a hit with search(Stdio.Buffer).
Stdio.Buffer [search]: Support searching for substrings.
Stdio.Buffer: Added _search().
Stdio.Buffer [search]: Support specifying the end position. It is now possible to inform the Stdio.Buffer lfun::_search() that it shouldn't scan past a certain position. Also removes an unused variable.
Stdio.Buffer [search]: _search() now behaves as documented. Return UNDEFINED instead of -1 on failure (as -1 is a valid result).
Stdio.Buffer: added truncate()
Revert "Stdio.Buffer: Check for PIKE_T_FUNCTION before acting on objects." This reverts commit 01543a00e48294140995709915891d7f7745bdb2. This commit was part of a set of speculative fixes that introduced other issues (SIGSEGV in testsuite).
Revert "Stdio.Buffer: Protect against calls in destructed Buffer objects." This reverts commit 5884c9d61d11a8374e2feac0e4f211121468aea8. This commit was part of a set of speculative fixes that introduced other issues (SIGSEGV in testsuite).
Revert "Stdio.Buffer: Explicitly clear io->this to mark destruction." This reverts commit 730037056431cbf81006c690ee0a6fd0d7bd2b08. This commit was part of a set of speculative fixes that introduced other issues (SIGSEGV in testsuite).
Revert "Stdio.Buffer: Use PIKE_TYPEOF() macro." This reverts commit 2317b255de66f471eb1555165af276f51430015d. This commit was part of a set of speculative fixes that introduced other issues (SIGSEGV in testsuite).
Stdio.Buffer: Use PIKE_TYPEOF() macro.
Stdio.Buffer: Explicitly clear io->this to mark destruction.
Stdio.Buffer: Protect against calls in destructed Buffer objects.
Stdio.Buffer: Check for PIKE_T_FUNCTION before acting on objects.
Stdio.Buffer: Put null in a freed io->output.u.object.
Documentation [Stdio.Buffer]: Fixed markup typo.
Documentation [Stdio]: Document low-level classes.
Merge branch 'patches/bug10016' into 8.0 * patches/bug10016: Stdio.Buffer.RewindKey: LFUNs should not be private.
Merge branch 'patches/bug10016' * patches/bug10016: Stdio.Buffer.RewindKey: LFUNs should not be private.
Stdio.Buffer.RewindKey: LFUNs should not be private. Fixes #10016.
Merge branch 'patches/pike210' into 8.0 * patches/pike210: Stdio.Buffer: Fix type for _sprintf().
Merge branch 'patches/pike210-master' * patches/pike210-master: Stdio.Buffer: Fix type for _sprintf().
Merge branch 'patches/pike210' into patches/pike210-master * patches/pike210: Stdio.Buffer: Fix type for _sprintf().
Merge branch 'patches/pike210' into patches/pike210-new * patches/pike210: Stdio.Buffer: Fix type for _sprintf().
Stdio.Buffer: Fix type for _sprintf(). Fixes PIKE-210 (#8210).
Stdio.Buffer: Fixed input_from() with non-Stdio.File objects. The return value from input_from() was always -1 when it was called with non-Stdio.File objects (even if it successfully read data). This issue is triggered by the testsuite for the Filesystem module.
Stdio.Buffer: Stricter types. Use string(8bit) in more places.
Stdio.Buffer: Mark the class for immediate destruction. This is needed since the object may hold a lock on a different Stdio.Buffer. Potentially fixes some of [LysLysKOM 23391165].
Merge remote-tracking branch 'origin/master' into new_utf8
Merge commit '722771973bd' into patches/lyslyskom22891031 * commit '722771973bd': (6177 commits) Verify that callablep responses are aligned with reality. ...
Merge branch 'grubba/wop-local-variables-debug-info' into bill/debugger-concept
Stdio.Buffer: Let the GC know again about our on_write callback Removed in the buffer mode API change commit, but turned out to not do nothing.
Stdio.Buffer: Use Pike stack for storing on_write cb for call Gives normal error propagation while freeing us from having to handle errors ourselves.
Stdio: Buffer mode now uses ->write(Stdio.Buffer()) API As part of this change, the Stdio.Buffer()->__fd_set_output() API got replaced by Stdio.Buffer()->__set_on_write(), and users (File and SSL.File) have been adopted to use it.
Merge branch 'buffer_mode_proposal_wip'
Stdio.File()->read(): advance buffer if possible This commit also removed Stdio.Buffer()->advance() because it is a bad API. It allows one to 'validate' uninitialized memory inside of Stdio.Buffer objects. Instead, we now try to advance the write offset in Buffer objects in Stdio.File()->read(), instead. This is done using a new internal API which can be extended to advance other buffer types in the future.
Stdio.Buffer: added allocate() and advance()
Merge remote-tracking branch 'origin/8.1' into gobject-introspection
Stdio.Buffer: Added support for escapes to read_cstring().
Fix refdoc typo.
Punctuation in error messages.
Build [Stdio.Buffer]: Fixed a few warnings.
Stdio.Buffer: fixed some compile warnings
Stdio.Buffer: Eliminates race condition from read_cstring(). In addition it makes read_cstring(): - Restartable after buffer changes from range_error(). - Call range_error(0) just like sscanf(). - Faster. The race condition occurred after a pause due to range_error(), the subsequent io_rewind() at the end could not rewind far enough.
Stdio.Buffer: Decrease system call overhead.
Stdio.Buffer: Centralise bufferconstants.
Stdio.Buffer: Simplify code, centralise locked_move check.
Remove redundant includes.
Merge branch '8.1' into peter/travis
Use the ALLOC_STRUCT macro when possible.
Move pike_search.h include from pike_memory.h and made includes implicit where needed.
Merge remote-tracking branch 'origin/8.1' into peter/travis
Clena up stack in create
Stdio.Buffer: Support bignums in add_int{8,16,32}(). add_int8() and add_int16() now support getting called with a bignum. add_int32() with a bignum now ignores any bits outside the 32-bit range.
Merge branch 'grubba/rename_lfun_destroy' into 8.1 * grubba/rename_lfun_destroy: Modules: Fixed logts of warnings. Testsuite: Updated for LFUN::_destruct(). Compiler: Don't complain about LFUN::destroy() in compat mode. Fix multiple warnings. Runtime: LFUN::destroy() has been renamed to _destruct(). Compiler: Rename LFUN::destroy() to LFUN::_destruct().
Fix multiple warnings. LFUN::destroy() has been renamed to _destruct().
Stdio.Buffer: Some type and documentation updates.
Stdio.Buffer()->trim(): fixed possible memcpy on overlapping regions trim() will move data inside of the buffer regardless of the number of wasted bytes. In those cases src and dst of the memcpy() call may overlap.
Stdio.Buffer: fixed growth strategy Unless trim() was called manually or unless the buffer was empty, the growth strategy would never reuse empty space at the beginning of the buffer. Instead, it would always allocate more space. The code now works as described in the comment. When at least half of the buffer space in the beginning of the allocated region is unused, the data is moved to the front before new memory is being allocated.
MsgPack.encode_to(): call fd output function
Stdio.Buffer: added more missing io_trigger_output calls add_padding() and add_bignum() did not trigger the fd output callback.
Stdio.Buffer: trigger fd output in add_int*() and input_from() The add_int family of methods and input_from() did not trigger a call to the fd output function, which could result in write() never being called when using Stdio.File buffer mode.
Stdio.Buffer: fix unread() The documentation promises that unread() will always succeed until new data is added to the buffer. This contract was broken when the new growth strategy was added by commit 2c4241823. That commit added a possible trim to every read operation, which would try to shrink the buffer and make a future unread() fail. Also, the max_waste check contained a typo, which made it always succeed. This issue does not affect rewind keys.
Stdio.Buffer: return unsigned integers in _sizeof On 32bit systems we might encounter buffers larger than INT_MAX. Use push_ulongest() to make sure that we do not accidentally return negative length.
Stdio.Buffer: fixed integer overflow in read_json() If the decoded JSON is larger than INT_MAX, it could not be correctly decoded from a buffer.
Stdio.Buffer: fixed typo in previous commit
Linewrap
Storage is zeroed by calloc, so no need to memset it.
Call fast_clone_object instead of low_clone+call_c_initializers
Prevent SIGSEGV when attempting to create rewind keys on the buffers returned by read_buffer().
Stdio.Buffer: Fixed multiple issues with read_le_int(). The threshold for using io_read_number() instead of io_read_bignum() was off by one (on the safe side). Fixes broken use of mpz_import() in io_read_bignum(). Fixed typo in documentation.
Added basic little endian support to reading Stdio.Buffer.
Break out sprintf-related stuff from stralloc.h to sprintf.h. Minor header file cleanup.
Stdio.Buffer: Allocate a constant program id.
Stdio.Buffer: Support some bignums in add_int32(). FIXME: Ought not to care about the high order bits of the bignum. FIXME: Add similar support to the other add_int*() as well. Thanks to Per Cederqvist <ceder@lysator.liu.se> for the report. Fixes [bug 7712 (#7712)].
(0..255) -> 8bit and similar changes This means that buffer.cmod now needs a pike 8.0 (or newer) precompiler.
Stdio.Buffer: Some cosmetic changes.
LONGEST is just INT64, so use that instead.
SIMPLE_BAD_ARG_ERROR is only kept for compatibility, but we still had 500+ uses of it.
Stdio.Buffer: export a minimal C-API Make a minimal API available to be used from c modules. This allows writing encoders/decoders which directly operate on Buffer objects.
Allow read_cstring to have a different end sentinel than null.
Backported locked check fix for Stdio.Buffer from 8.1
Static:ified a lot of strings. This is more or less all instances of push_text(static data here). Left to do: make_shared_X_string(static data here). :)
Use the set_unaligned16/32 function
Stdio.Buffer: Avoid unaligned accesses. Fixes SIGBUS when adding integers to unaligned buffers on non-x86/amd64.
Stdio.Buffer: Modifying read-only buffers is NOT a good idea... Fixes SSL testsuite failure.
Stdio.Buffer: Added some locking to _sprintf(). The buffer could get trimmed during _sprintf() before it was rewound.
Stdio.Buffer: removed C99ism
Fixed an issue with output_to to non-files (eg, ssl). This bug was created by the new code that allows the buffer to shrink.
Rewrote buffer growth strategy It is now 1.625x old size instead of 2.0 x old size + bytes_added, with some exception. Also added a shrink strategy, by default it's close to the reverse: If more than 61.5% of the buffer is empty space, shrink it down (1/1.625). (cherry picked from commit 917184b64444098f8b84c79cdd1de89ccaa467f6) Conflicts: src/modules/_Stdio/buffer.cmod
Rewrote buffer growth strategy It is now 1.625x old size instead of 2.0 x old size + bytes_added, with some exception. Also added a shrink strategy, by default it's close to the reverse: If more than 61.5% of the buffer is empty space, shrink it down (1/1.625).
Normalized file ends.
Stdio.Buffer: io_ensure_malloced: use size_t type for 'bytes' input parameter. Also adds an overflow check.
Stdio.Buffer: use round_up*() functions to enlarge buffer. Fixes an infinity loop due to overflow. Also changed the round_up*() functions to return 0 on overflow and made 1 the next power of two after 0.
Stdio.Buffer: xrealloc() now throws on realloc() failure.
Stdio.Buffer: undo erroneous change of allocation behaviour
Stdio.Buffer: throw on realloc() failures
Stdio.Buffer(): add_padding() now adds padding.
Stdio.Buffer(): Don't allow negative padding to be added.
Stdio.Buffer: less leaking of rewind structs
Stdio.Buffer: Avoid shift by negative. Fixes [CID 1294563].
Stdio.Buffer: Fixed NULL-deref on UNDEFINED. Fixes [CID 1294611].
Stdio.Buffer: Fixed integer overflow. Fixes [CID 1294563].
[portability] Replaced use of MAX with MAXIMUM. The macro MAX is apparently available on Linux, but not on Solaris 10.
Build [NT]: <arpa/inet.h> is not available on all platforms. Restored some #ifdef guards.
Stdio.Buffer: trigger write cb when adding to empty buffer When the buffer was empty and a new string was added to it, the write callback was not triggered.
Stdio.Buffer: Use ptrdiff_t instead of ssize_t. ptrdiff_t is C89, while ssize_t is POSIX. As we support some non-POSIX architectures, use the more portable type.
build: help build on Windows
Stdio.Buffer: Fix alignment problem in io_append_{short,int}_uc
Make cast() protected.
Stdio.Buffer: LFUN::_sprintf() must not have side effects! sprintf("%s", buf) and sprintf("%q", buf) had the side effect of emptying the buffer (which is not what you expect when you use them for debug output...).
The buffer_error_program only takes one argument. This could probably be handled better.
Work around precompiler type issue.
Fix docs.
Stdio.Buffer: Support all types for add_hstring.
Stdio.Buffer: Support offset parameter for add_hstring/read_hstring.
Removed stray undef.
Don't include gmp.h directly. Use bignum.h
Simplify code.
Access bignum_program directly.
Use the feature flag instead of api_version.
Fixed the name of the rewind_on_error and rewind_key functions
Stdio.Buffer: Fixed some documentation typos.
Stdio.Buffer: Use the correct macro...
Stdio.Buffer: Fixed return type for sprintf().
Stdio.Buffer: Added try_output().
Stdio.Buffer: Handle UNDEFINED nbytes in input_from(). The nbytes argument may be UNDEFINED (typically when the once argument has been specified). Handle it as if nbytes hasn't been specified. Thanks to Stephen R. van den Berg <srb@cuci.nl> for the report. Fixes some of [LysLysKOM 20998397]/[Pike mailinglist 14064].
Fixed warning.
Removed compat flags from sprintf and sscanf.
Stdio.Buffer: output_to() et al now support functions. This reduces the need for proxy objects in higher level code using Stdio.Buffer internally. Also adds support for subtyped objects and multiple inheritance of Stdio.Fd et al.
Re-add early exit if not a full write was accepted. This is most often an optimazation.
While any data can be sent, continue sending.
Fixed output_to nbytes feature for non-file objects.
io_rewind is already applied in io_call_write. Don't do it again in output_to.
Stdio.Buffer now works on 32bit systems
Whitespace changes only
Stdio.IOBuffer -> Stdio.Buffer (part 2)
Stdio.IOBuffer -> Stdio.Buffer (part 1)
IOBuffer: Propagate read errors in input_from(). input_from() now returns -1 on read error. Also minor optimization to avoid extra read(2) call when reading a multiple of 4096 bytes.
Stricter types in buffer.cmod
Added rewind_key, much like rewind_on_error, but not automatic. Also speed up read_cstring about a factor of two.
Added read_hint and add_hint They read and add integers encoded like hstrings, first a length then the number.
Stdio.IOBuffer.UnwindKey: allow explicit ->rewind()
Added Stdio.IOBuffer::rewind_on_error. Use this function to cause the buffer to be rewound to the location it was at when the function was called if the object it returs goes out of scope.
Stdio.IOBuffer: bad return type of add_int32()
Removed debug.
Added read_sint for read signed integer. The reverse might be more obvious (read_int for signed, read_uint for unsigned), but at least it exists now. Also fixed so that add_int works for all negative numbers, and optimized it rather significantly for bignums (it no longer creates a string copy of the bignum value). Also added a simple add_padding() function to add a few bytes of data set to a specific value (basically memset)
IOBuffer: Use add_ref().
Fixed signed/unsigned warning
Allow adding chunks bigger than 2^32 at a time using add() and friends. Also, instead of saying you are trying to read too much when reading -<whatever> bytes using read or try_read you will now get an error indicating you are reading a negative number of bytes, which is hard. I guess it might unread(), but that would be highly unexpected.
No, actually, it is not reached. This saves a few bytes of binary size. :)
Do not trigger the write callback again when nothing is written This saves the user from checking sizeof(buffer) before doing buffer->output_to(...)
IOBuffer: Added try_read(). This is an analogue to Stdio.File()->read().
Simplify code, speed up fast path.
IOBuffer: Fixed fatal "Invalid io_rewind_on_error nesting". The value in io->locked_move was off by one.
Revert "IOBuffer: Optimise and fix range checks." This reverts commit 91460a41b2b68b9d745827a07b6762adc99e3ed9. This introduced several errors, including some that causes buffer overruns. It also made the code biger on average (just check using size). Please do not change the math unless you are very certain it will work. Also, please trust the compiler, both to unroll loops even if it does not make sense and to optimize obvious comparisons, yes? Obvious example: check_avail_mul suddenly is pointless. The whole idea with that function is to _safely_ check if A*B is bigger than the buffer. Consider A*B >being > NATIVE_INT_MAX.
IOBuffer.cstring to parse \0 terminated C-strings.
IOBuffer: Optimise and fix range checks. Eliminates some of the recursions.
Also removed add_short
Removed add_byte. We have add_int8.
Unused local variable.
Workaround for precompiler issue in IOBuffer. The precompiler generates invalid code and fails to compile when PRECOMPILE_API_VERSION < 5. Generated code ends up inside the #ifdef-clause.
Improved performance in output_to noticeably Also, do not crash if other threads use the buffer while output_to has been called. It was very very intentionally not threaded. If we _need_ thread support (say, we want to support writing to blocking files in the background) the buffer class has to be rewritten quite a lot. Most importantly it can not allow anyone to touch it while the other thread is using the memory, which in turn means that it has to be refcounted, and operations on it while refs > 1 will have to be a big no-no, or copy all data. It is rather hard to use the buffer if it behaves like that, mainly when it is used as an output buffer for a file (it is assumed that the add functions are called to add output then, and if the writing is done in a thread it will be likely to cause issues) Hence the total lack of THREADS_ALLOW in the buffer object. The alternative is to locking to all buffer functions. Or, I guess, we could just leave it as is and hope nobody does antything "mean". Alternatively, use the kernel async_write code on linux, that would make output_to nonblocking even for blocking files on that platform, and hopefully the memory will be copied when the system call returns, and it is thus safe to change it. The main issue with output_to + write + threads is that the threads are enabled before the system call has copied the data, and you will thus get a spurious error if the buffer is reallocated in another thread.
When we know the types we can optimize pop and friends. Use that.
IOBuffer: Improved support for threading in output_to().
As it turns out io_trigger_output is inlined in every location using io_append. Fixed that.
Saved 200 bytes binary size and made the code faster. One lesson learned: pop_n_elems is rather big.
Use the get-memobj-pointer function and some other cleanups. The compiled code is now 20% smaller.
Stdio.IOBuffer: wrong return type of read_int32
IOBuffer: Support file objects with overloaded write(). FIXME: Support subtyped file objects?
The compiler will do this implicitly.
IOBuffer: Use ptrdiff_t instead of ssize_t. ptrdiff_t is C89, while ssize_t is POSIX. As we support some non-POSIX architectures, use the more portable type.
Added a few read_int<size>() functions for common int sizes.
Moved some common code to functions. Also use the new "map variable from structures" support to remove two getters.
IOBuffer: Fixed string leak. Also fixes some potential related leaks.
IOBuffer: Propagate errno to the file object on error.
IOBuffer: Return -1 from output_to() on write error.
IOBuffer: Fixed some AutoDoc markup issues.
IOBuffer: Survive cyclic arrays.
IOBuffer: Fixed warning.
IOBuffer: Fixed compilation error --with-rtldebug.
Fixed autodoc error
IOBuffer: Added some more tests and fixed errors found At least almost all functions in IOBuffer have some basic tests now..
Fixed IOBuffer.add_int
IOBuffer: Cleanup of some code.
Some IOBuffer work o Made all (I think) functions atomic. Either they work or they do nothing (except throw an error or return 0). o One minor optimization to avoid copies when strings/system.memory object etc are added to an empty buffer. Unless you add more data the buffer is just moved to point to the object that was just added. The malloced buffer is still kept around, however. o Added a very very basic testsuite for IOBuffer. More to come.
IOBuffer: Some documentation fixes.
IOBuffer: Improved error recovery for read_h{string,buffer}(). Don't advance the buffer pointers if there's not enough data. NB: Not a 100% robust. It will fail if io_avail() throws errors, or if the holerith length is negative or too large.
Merge remote-tracking branch 'origin/8.0' into string_alloc Conflicts: src/stralloc.c
Allow "read only once" more for input_from in IOBuffer.
port.h is included by global.h
Added buffered asynchronous I/O mode to Stdio.File In this mode the input and output is kept in buffers in the file object. Adding data to the output buffer will write it to the filedescriptor, and the read callback will receive the input buffer as its argument. The write callback will have the output buffer as a second argument, but most of the time this can be ignored since you keep the buffer object around. The write callback will only be called if the buffer needs more data, so if you implement a range_error function in the buffer that refills the buffer it will only be called once that callback returns 0.
Correctly update the box events when reading/writing from files
Stdio.IOBuffer.intput/output from fd can now take the amount to process This is better for SSL files, where you can write any amount of data.
IOBuffer: Use apply_current() to call range_error(). This gets rid of a dynamic lookup and a reference on the string "range_error". It also improves the odds that IOBuffer will support being inherited multiple times, as well as private inherit.
Added add_ints() method to IOBuffer.
Some tweaks to Stdio.IOBuffer: o The various functions that add data now return the buffer o It is now possible to overload the range_error method. This can be used to fill the buffer on demand.
Added _encode and _decode to Stdio.IOBuffer
Some changes to Stdio.IOBuffer: o Set error mode now accepts a program to throw on error o Relaxed argument types (for now)
Correctly throw buffer error in error mode
Added some more functionality to Stdio.IOBuffer o input_from(file), output_to(file) -- async or sync file-IO. o add and crete now supports the other buffer objects
Added Stdio.IOBuffer.write_to(Stdio.Stream).
Added read_json method. It will read the first json value on the stream and return it.
The IOBuffer documentation now extracts
Moved __builtin.IOBuffer to Stdio.IOBuffer (for now)