Support bidi reordering of overlay and display strings.

Fix bugs #7616, #8133, #8867.

 src/xdisp.c (compute_display_string_pos)
 (compute_display_string_end): Accept additional argument STRING.
 (init_iterator, reseat_1): Initialize bidi_it->string.s to NULL.
 (reseat_to_string): Initialize bidi_it->string.s and
 bidi_it->string.schars.
 (Fcurrent_bidi_paragraph_direction): Initialize itb.string.s to
 NULL (avoids a crash in bidi_paragraph_init).  Initialize
 itb.string.lstring.
 (init_iterator): Call bidi_init_it only of a valid
 buffer position was specified.  Initialize paragraph_embedding to
 L2R.
 (reseat_to_string): Initialize the bidi iterator.
 (display_string): If we need to ignore text properties of
 LISP_STRING, set IT->stop_charpos to IT->end_charpos.  (The
 original value of -1 will not work with bidi.)
 (compute_display_string_pos): First arg is now struct
 `text_pos *'; all callers changed.  Support display properties on
 Lisp strings.
 (compute_display_string_end): Support display properties on Lisp
 strings.
 (init_iterator, reseat_1, reseat_to_string): Initialize the
 string.bufpos member to 0 (zero, for compatibility with IT_CHARPOS
 when iterating on a string not from display properties).
 (compute_display_string_pos, compute_display_string_end): Fix
 calculation of the object to scan.  Fixes an error when using
 arrow keys.
 (next_element_from_buffer): Don't abort when IT_CHARPOS is before
 base_level_stop; instead, set base_level_stop to BEGV.  Fixes
 crashes in vertical-motion.
 (next_element_from_buffer): Improve commentary for when
 the iterator is before prev_stop.
 (init_iterator): Initialize bidi_p from the default value of
 bidi-display-reordering, not from buffer-local value.  Use the
 buffer-local value only if initializing for buffer iteration.
 (handle_invisible_prop): Support invisible properties on strings
 that are being bidi-reordered.
 (set_iterator_to_next): Support bidi reordering of C strings and
 Lisp strings.
 (next_element_from_string): Support bidi reordering of Lisp
 strings.
 (handle_stop_backwards): Support Lisp strings as well.
 (display_string): Support display of R2L glyph rows.  Use
 IT_STRING_CHARPOS when displaying from a Lisp string.
 (init_iterator): Don't initialize it->bidi_p for strings
 here.
 (reseat_to_string): Initialize it->bidi_p for strings here.
 (next_element_from_string, next_element_from_c_string)
 (next_element_from_buffer): Add xassert's for correspondence
 between IT's object being iterated and it->bidi_it.string
 structure.
 (face_before_or_after_it_pos): Support bidi iteration.
 (next_element_from_c_string): Handle the case of the first string
 character that is not the first one in the visual order.
 (get_visually_first_element): New function, refactored from common
 parts of next_element_from_buffer, next_element_from_string, and
 next_element_from_c_string.
 (tool_bar_lines_needed, redisplay_tool_bar)
 (display_menu_bar): Force left-to-right direction.  Add a FIXME
 comment for making that be controlled by a user option.
 (push_it, pop_it): Save and restore the state of the
 bidi iterator.  Save and restore the bidi_p flag.
 (pop_it): Iterate out of display property for string iteration as
 well.
 (iterate_out_of_display_property): Support iteration over strings.
 (handle_single_display_spec): Set up it->bidi_it for iteration
 over a display string, and call bidi_init_it.
 (handle_single_display_spec, next_overlay_string)
 (get_overlay_strings_1, push_display_prop): Set up the bidi
 iterator for displaying display or overlay strings.
 (forward_to_next_line_start): Don't use the shortcut if
 bidi-iterating.
 (back_to_previous_visible_line_start): If handle_display_prop
 pushed the iterator stack, restore the internal state of the bidi
 iterator by calling bidi_pop_it same number of times.
 (reseat_at_next_visible_line_start): If ON_NEWLINE_P is non-zero,
 and we are bidi-iterating, don't decrement the iterator position;
 instead, set the first_elt flag in the bidi iterator, to produce
 the same effect.
 (reseat_1): Remove redundant setting of string_from_display_prop_p.
 (push_display_prop): xassert that we are iterating a buffer.
 (push_it, pop_it): Save and restore paragraph_embedding member.
 (handle_single_display_spec, next_overlay_string)
 (get_overlay_strings_1, reseat_1, reseat_to_string)
 (push_display_prop): Set up the `unibyte' member of bidi_it.string
 correctly.  Don't assume unibyte strings are not bidi-reordered.
 (compute_display_string_pos)
 (compute_display_string_end): Fix handling the case of C string.
 (push_it, pop_it): Save and restore from_disp_prop_p.
 (handle_single_display_spec, push_display_prop): Set the
 from_disp_prop_p flag.
 (get_overlay_strings_1): Reset the from_disp_prop_p flag.
 (pop_it): Call iterate_out_of_display_property only if we are
 popping after iteration over a string that came from a display
 property.  Fix a typo in popping stretch info.  Add an assertion
 for verifying that the iterator position is in sync with the bidi
 iterator.
 (handle_single_display_spec, get_overlay_strings_1)
 (push_display_prop): Fix initialization of paragraph direction for
 string when that of the parent object is not yet determined.
 (reseat_1): Call bidi_init_it to resync the bidi
 iterator with IT's position.  (Bug#7616)
 (find_row_edges): If ROW->start.pos gives position
 smaller than min_pos, use it as ROW->minpos.  (Bug#7616)
 (handle_stop, back_to_previous_visible_line_start, reseat_1):
 Reset the from_disp_prop_p flag.
 (SAVE_IT, RESTORE_IT): New macros.
 (pos_visible_p, face_before_or_after_it_pos)
 (back_to_previous_visible_line_start)
 (move_it_in_display_line_to, move_it_in_display_line)
 (move_it_to, move_it_vertically_backward, move_it_by_lines)
 (try_scrolling, redisplay_window, display_line): Use them when
 saving a temporary copy of the iterator and restoring it back.
 (back_to_previous_visible_line_start, reseat_1)
 (init_iterator): Empty the bidi cache "stack".
 (move_it_in_display_line_to): If iterator ended up at
 EOL, but we never saw any buffer positions smaller than
 to_charpos, return MOVE_POS_MATCH_OR_ZV.  Fixes vertical cursor
 motion in bidi-reordered lines.
 (move_it_in_display_line_to): Record prev_method and prev_pos
 immediately before the call to set_iterator_to_next.  Fixes cursor
 motion in bidi-reordered lines with stretch glyphs and strings
 displayed in margins.  (Bug#8133) (Bug#8867)
 Return MOVE_POS_MATCH_OR_ZV only if iterator position is past
 TO_CHARPOS.
 (pos_visible_p): Support positions in bidi-reordered lines.  Save
 and restore bidi cache.
 src/bidi.c (bidi_level_of_next_char): clen should be EMACS_NT, not int.
 (bidi_paragraph_info): Delete unused struct.
 (bidi_cache_idx, bidi_cache_last_idx): Declare EMACS_INT.
 (bidi_cache_start): New variable.
 (bidi_cache_reset): Reset bidi_cache_idx to bidi_cache_start, not
 to zero.
 (bidi_cache_fetch_state, bidi_cache_search)
 (bidi_cache_find_level_change, bidi_cache_iterator_state)
 (bidi_cache_find, bidi_peek_at_next_level)
 (bidi_level_of_next_char, bidi_find_other_level_edge)
 (bidi_move_to_visually_next): Compare cache index with
 bidi_cache_start rather than with zero.
 (bidi_fetch_char): Accept new argument STRING; all callers
 changed.  Support iteration over a string.  Support strings with
 display properties.  Support unibyte strings.  Fix the type of
 `len' according to what STRING_CHAR_AND_LENGTH expects.
 (bidi_paragraph_init, bidi_resolve_explicit_1)
 (bidi_resolve_explicit, bidi_resolve_weak)
 (bidi_level_of_next_char, bidi_move_to_visually_next): Support
 iteration over a string.
 (bidi_set_sor_type, bidi_resolve_explicit_1)
 (bidi_resolve_explicit, bidi_type_of_next_char): ignore_bn_limit
 can now be zero (for strings); special values 0 and -1 were
 changed to -1 and -2, respectively.
 (bidi_char_at_pos): New function.
 (bidi_paragraph_init, bidi_resolve_explicit, bidi_resolve_weak):
 Call it instead of FETCH_MULTIBYTE_CHAR.
 (bidi_move_to_visually_next): Abort if charpos or bytepos were not
 initialized to valid values.
 (bidi_init_it): Don't initialize charpos and bytepos with invalid
 values.
 (bidi_level_of_next_char): Allow the sentinel "position" to pass
 the test for valid cached positions.  Fix the logic for looking up
 the sentinel state in the cache.  GCPRO the Lisp string we are
 iterating.
 (bidi_push_it, bidi_pop_it): New functions.
 (bidi_initialize): Initialize the bidi cache start stack pointer.
 (bidi_cache_ensure_space): New function, refactored from part of
 bidi_cache_iterator_state.  Don't assume the required size is just
 one BIDI_CACHE_CHUNK away.
 (bidi_cache_start_stack, bidi_push_it): Use IT_STACK_SIZE.
 (bidi_count_bytes, bidi_char_at_pos): New functions.
 (bidi_cache_search): Don't assume bidi_cache_last_idx is
 always valid if bidi_cache_idx is valid.
 (bidi_cache_find_level_change): xassert that bidi_cache_last_idx
 is valid if it's going to be used.
 (bidi_shelve_cache, bidi_unshelve_cache): New functions.
 (bidi_cache_fetch_state, bidi_cache_search)
 (bidi_cache_find_level_change, bidi_cache_ensure_space)
 (bidi_cache_iterator_state, bidi_cache_find)
 (bidi_find_other_level_edge, bidi_cache_start_stack): All
 variables related to cache indices are now EMACS_INT.
 src/dispextern.h (struct bidi_string_data): New structure.
 (struct bidi_it): New member `string'.  Make flag members be 1-bit
 fields, and put them last in the struct.
 (compute_display_string_pos, compute_display_string_end): Update
 prototypes.
 (bidi_push_it, bidi_pop_it): Add prototypes.
 (struct iterator_stack_entry): New members bidi_p,
 paragraph_embedding, and from_disp_prop_p.
 (struct it): Member bidi_p is now a bit field 1 bit wide.
 (bidi_shelve_cache, bidi_unshelve_cache): Declare
 prototypes.
 src/.gdbinit (xvectype, xvector, xcompiled, xchartable, xboolvector)
 (xpr, xfont, xbacktrace): Use "header.size" when accessing vectors
 and vector-like objects.
 src/dispnew.c (buffer_posn_from_coords): Save and restore the bidi
 cache around display iteration.
 src/window.c (Fwindow_end, window_scroll_pixel_based)
 (displayed_window_lines, Frecenter): Save and restore the bidi
 cache around display iteration.
 lisp/buff-menu.el (Buffer-menu-buffer+size): Accept an additional
 argument LRM; if non-nil, append an invisible LRM character to the
 buffer name.
 (list-buffers-noselect): Call Buffer-menu-buffer+size with the
 last argument non-nil, when formatting buffer names.
 (Buffer-menu-mode, list-buffers-noselect): Force left-to-right
 paragraph direction.
 doc/lispref/display.texi (Other Display Specs): Document that `left-fringe'
 and `right-fringe' display specifications are of the "replacing"
 kind.

1 files changed, 206 insertions, 0 deletions

diff --git a/src/ChangeLog b/src/ChangeLog
index 35c56e5f478..c2bd1981e76 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,3 +1,209 @@
+2011-07-14  Eli Zaretskii  <eliz@gnu.org>
+
+        Support bidi reordering of display and overlay strings.
+        * xdisp.c (compute_display_string_pos)
+        (compute_display_string_end): Accept additional argument STRING.
+        (init_iterator, reseat_1): Initialize bidi_it->string.s to NULL.
+        (reseat_to_string): Initialize bidi_it->string.s and
+        bidi_it->string.schars.
+        (Fcurrent_bidi_paragraph_direction): Initialize itb.string.s to
+        NULL (avoids a crash in bidi_paragraph_init).  Initialize
+        itb.string.lstring.
+        (init_iterator): Call bidi_init_it only of a valid
+        buffer position was specified.  Initialize paragraph_embedding to
+        L2R.
+        (reseat_to_string): Initialize the bidi iterator.
+        (display_string): If we need to ignore text properties of
+        LISP_STRING, set IT->stop_charpos to IT->end_charpos.  (The
+        original value of -1 will not work with bidi.)
+        (compute_display_string_pos): First arg is now struct
+        `text_pos *'; all callers changed.  Support display properties on
+        Lisp strings.
+        (compute_display_string_end): Support display properties on Lisp
+        strings.
+        (init_iterator, reseat_1, reseat_to_string): Initialize the
+        string.bufpos member to 0 (zero, for compatibility with IT_CHARPOS
+        when iterating on a string not from display properties).
+        (compute_display_string_pos, compute_display_string_end): Fix
+        calculation of the object to scan.  Fixes an error when using
+        arrow keys.
+        (next_element_from_buffer): Don't abort when IT_CHARPOS is before
+        base_level_stop; instead, set base_level_stop to BEGV.  Fixes
+        crashes in vertical-motion.
+        (next_element_from_buffer): Improve commentary for when
+        the iterator is before prev_stop.
+        (init_iterator): Initialize bidi_p from the default value of
+        bidi-display-reordering, not from buffer-local value.  Use the
+        buffer-local value only if initializing for buffer iteration.
+        (handle_invisible_prop): Support invisible properties on strings
+        that are being bidi-reordered.
+        (set_iterator_to_next): Support bidi reordering of C strings and
+        Lisp strings.
+        (next_element_from_string): Support bidi reordering of Lisp
+        strings.
+        (handle_stop_backwards): Support Lisp strings as well.
+        (display_string): Support display of R2L glyph rows.  Use
+        IT_STRING_CHARPOS when displaying from a Lisp string.
+        (init_iterator): Don't initialize it->bidi_p for strings
+        here.
+        (reseat_to_string): Initialize it->bidi_p for strings here.
+        (next_element_from_string, next_element_from_c_string)
+        (next_element_from_buffer): Add xassert's for correspondence
+        between IT's object being iterated and it->bidi_it.string
+        structure.
+        (face_before_or_after_it_pos): Support bidi iteration.
+        (next_element_from_c_string): Handle the case of the first string
+        character that is not the first one in the visual order.
+        (get_visually_first_element): New function, refactored from common
+        parts of next_element_from_buffer, next_element_from_string, and
+        next_element_from_c_string.
+        (tool_bar_lines_needed, redisplay_tool_bar)
+        (display_menu_bar): Force left-to-right direction.  Add a FIXME
+        comment for making that be controlled by a user option.
+        (push_it, pop_it): Save and restore the state of the
+        bidi iterator.  Save and restore the bidi_p flag.
+        (pop_it): Iterate out of display property for string iteration as
+        well.
+        (iterate_out_of_display_property): Support iteration over strings.
+        (handle_single_display_spec): Set up it->bidi_it for iteration
+        over a display string, and call bidi_init_it.
+        (handle_single_display_spec, next_overlay_string)
+        (get_overlay_strings_1, push_display_prop): Set up the bidi
+        iterator for displaying display or overlay strings.
+        (forward_to_next_line_start): Don't use the shortcut if
+        bidi-iterating.
+        (back_to_previous_visible_line_start): If handle_display_prop
+        pushed the iterator stack, restore the internal state of the bidi
+        iterator by calling bidi_pop_it same number of times.
+        (reseat_at_next_visible_line_start): If ON_NEWLINE_P is non-zero,
+        and we are bidi-iterating, don't decrement the iterator position;
+        instead, set the first_elt flag in the bidi iterator, to produce
+        the same effect.
+        (reseat_1): Remove redundant setting of string_from_display_prop_p.
+        (push_display_prop): xassert that we are iterating a buffer.
+        (push_it, pop_it): Save and restore paragraph_embedding member.
+        (handle_single_display_spec, next_overlay_string)
+        (get_overlay_strings_1, reseat_1, reseat_to_string)
+        (push_display_prop): Set up the `unibyte' member of bidi_it.string
+        correctly.  Don't assume unibyte strings are not bidi-reordered.
+        (compute_display_string_pos)
+        (compute_display_string_end): Fix handling the case of C string.
+        (push_it, pop_it): Save and restore from_disp_prop_p.
+        (handle_single_display_spec, push_display_prop): Set the
+        from_disp_prop_p flag.
+        (get_overlay_strings_1): Reset the from_disp_prop_p flag.
+        (pop_it): Call iterate_out_of_display_property only if we are
+        popping after iteration over a string that came from a display
+        property.  Fix a typo in popping stretch info.  Add an assertion
+        for verifying that the iterator position is in sync with the bidi
+        iterator.
+        (handle_single_display_spec, get_overlay_strings_1)
+        (push_display_prop): Fix initialization of paragraph direction for
+        string when that of the parent object is not yet determined.
+        (reseat_1): Call bidi_init_it to resync the bidi
+        iterator with IT's position.  (Bug#7616)
+        (find_row_edges): If ROW->start.pos gives position
+        smaller than min_pos, use it as ROW->minpos.  (Bug#7616)
+        (handle_stop, back_to_previous_visible_line_start, reseat_1):
+        Reset the from_disp_prop_p flag.
+        (SAVE_IT, RESTORE_IT): New macros.
+        (pos_visible_p, face_before_or_after_it_pos)
+        (back_to_previous_visible_line_start)
+        (move_it_in_display_line_to, move_it_in_display_line)
+        (move_it_to, move_it_vertically_backward, move_it_by_lines)
+        (try_scrolling, redisplay_window, display_line): Use them when
+        saving a temporary copy of the iterator and restoring it back.
+        (back_to_previous_visible_line_start, reseat_1)
+        (init_iterator): Empty the bidi cache "stack".
+        (move_it_in_display_line_to): If iterator ended up at
+        EOL, but we never saw any buffer positions smaller than
+        to_charpos, return MOVE_POS_MATCH_OR_ZV.  Fixes vertical cursor
+        motion in bidi-reordered lines.
+        (move_it_in_display_line_to): Record prev_method and prev_pos
+        immediately before the call to set_iterator_to_next.  Fixes cursor
+        motion in bidi-reordered lines with stretch glyphs and strings
+        displayed in margins.  (Bug#8133) (Bug#8867)
+        Return MOVE_POS_MATCH_OR_ZV only if iterator position is past
+        TO_CHARPOS.
+        (pos_visible_p): Support positions in bidi-reordered lines.  Save
+        and restore bidi cache.
+
+        * bidi.c (bidi_level_of_next_char): clen should be EMACS_NT, not int.
+        (bidi_paragraph_info): Delete unused struct.
+        (bidi_cache_idx, bidi_cache_last_idx): Declare EMACS_INT.
+        (bidi_cache_start): New variable.
+        (bidi_cache_reset): Reset bidi_cache_idx to bidi_cache_start, not
+        to zero.
+        (bidi_cache_fetch_state, bidi_cache_search)
+        (bidi_cache_find_level_change, bidi_cache_iterator_state)
+        (bidi_cache_find, bidi_peek_at_next_level)
+        (bidi_level_of_next_char, bidi_find_other_level_edge)
+        (bidi_move_to_visually_next): Compare cache index with
+        bidi_cache_start rather than with zero.
+        (bidi_fetch_char): Accept new argument STRING; all callers
+        changed.  Support iteration over a string.  Support strings with
+        display properties.  Support unibyte strings.  Fix the type of
+        `len' according to what STRING_CHAR_AND_LENGTH expects.
+        (bidi_paragraph_init, bidi_resolve_explicit_1)
+        (bidi_resolve_explicit, bidi_resolve_weak)
+        (bidi_level_of_next_char, bidi_move_to_visually_next): Support
+        iteration over a string.
+        (bidi_set_sor_type, bidi_resolve_explicit_1)
+        (bidi_resolve_explicit, bidi_type_of_next_char): ignore_bn_limit
+        can now be zero (for strings); special values 0 and -1 were
+        changed to -1 and -2, respectively.
+        (bidi_char_at_pos): New function.
+        (bidi_paragraph_init, bidi_resolve_explicit, bidi_resolve_weak):
+        Call it instead of FETCH_MULTIBYTE_CHAR.
+        (bidi_move_to_visually_next): Abort if charpos or bytepos were not
+        initialized to valid values.
+        (bidi_init_it): Don't initialize charpos and bytepos with invalid
+        values.
+        (bidi_level_of_next_char): Allow the sentinel "position" to pass
+        the test for valid cached positions.  Fix the logic for looking up
+        the sentinel state in the cache.  GCPRO the Lisp string we are
+        iterating.
+        (bidi_push_it, bidi_pop_it): New functions.
+        (bidi_initialize): Initialize the bidi cache start stack pointer.
+        (bidi_cache_ensure_space): New function, refactored from part of
+        bidi_cache_iterator_state.  Don't assume the required size is just
+        one BIDI_CACHE_CHUNK away.
+        (bidi_cache_start_stack, bidi_push_it): Use IT_STACK_SIZE.
+        (bidi_count_bytes, bidi_char_at_pos): New functions.
+        (bidi_cache_search): Don't assume bidi_cache_last_idx is
+        always valid if bidi_cache_idx is valid.
+        (bidi_cache_find_level_change): xassert that bidi_cache_last_idx
+        is valid if it's going to be used.
+        (bidi_shelve_cache, bidi_unshelve_cache): New functions.
+        (bidi_cache_fetch_state, bidi_cache_search)
+        (bidi_cache_find_level_change, bidi_cache_ensure_space)
+        (bidi_cache_iterator_state, bidi_cache_find)
+        (bidi_find_other_level_edge, bidi_cache_start_stack): All
+        variables related to cache indices are now EMACS_INT.
+
+        * dispextern.h (struct bidi_string_data): New structure.
+        (struct bidi_it): New member `string'.  Make flag members be 1-bit
+        fields, and put them last in the struct.
+        (compute_display_string_pos, compute_display_string_end): Update
+        prototypes.
+        (bidi_push_it, bidi_pop_it): Add prototypes.
+        (struct iterator_stack_entry): New members bidi_p,
+        paragraph_embedding, and from_disp_prop_p.
+        (struct it): Member bidi_p is now a bit field 1 bit wide.
+        (bidi_shelve_cache, bidi_unshelve_cache): Declare
+        prototypes.
+
+        * .gdbinit (xvectype, xvector, xcompiled, xchartable, xboolvector)
+        (xpr, xfont, xbacktrace): Use "header.size" when accessing vectors
+        and vector-like objects.
+
+        * dispnew.c (buffer_posn_from_coords): Save and restore the bidi
+        cache around display iteration.
+
+        * window.c (Fwindow_end, window_scroll_pixel_based)
+        (displayed_window_lines, Frecenter): Save and restore the bidi
+        cache around display iteration.
+
 2011-07-14  Lars Magne Ingebrigtsen  <larsi@gnus.org>
 
         * editfns.c (Fdelete_region): Clarify the use of the named