LCOV - code coverage report
Current view: top level - gcc - combine.cc (source / functions) Coverage Total Hit
Test: gcc.info Lines: 90.8 % 6585 5982
Test Date: 2026-11-14 16:39:50 Functions: 97.1 % 105 102
Legend: Lines:     hit not hit

            Line data    Source code
       1              : /* Optimize by combining instructions for GNU compiler.
       2              :    Copyright (C) 1987-2026 Free Software Foundation, Inc.
       3              : 
       4              : This file is part of GCC.
       5              : 
       6              : GCC is free software; you can redistribute it and/or modify it under
       7              : the terms of the GNU General Public License as published by the Free
       8              : Software Foundation; either version 3, or (at your option) any later
       9              : version.
      10              : 
      11              : GCC is distributed in the hope that it will be useful, but WITHOUT ANY
      12              : WARRANTY; without even the implied warranty of MERCHANTABILITY or
      13              : FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
      14              : for more details.
      15              : 
      16              : You should have received a copy of the GNU General Public License
      17              : along with GCC; see the file COPYING3.  If not see
      18              : <http://www.gnu.org/licenses/>.  */
      19              : 
      20              : /* This module is essentially the "combiner" phase of the U. of Arizona
      21              :    Portable Optimizer, but redone to work on our list-structured
      22              :    representation for RTL instead of their string representation.
      23              : 
      24              :    The LOG_LINKS of each insn identify the most recent assignment
      25              :    to each REG used in the insn.  It is a list of previous insns,
      26              :    each of which contains a SET for a REG that is used in this insn
      27              :    and not used or set in between.  LOG_LINKs never cross basic blocks.
      28              :    They were set up by the preceding pass (lifetime analysis).
      29              : 
      30              :    We try to combine each pair of insns joined by a logical link.
      31              :    We also try to combine triplets of insns A, B and C when C has
      32              :    a link back to B and B has a link back to A.  Likewise for a
      33              :    small number of quadruplets of insns A, B, C and D for which
      34              :    there's high likelihood of success.
      35              : 
      36              :    We check (with modified_between_p) to avoid combining in such a way
      37              :    as to move a computation to a place where its value would be different.
      38              : 
      39              :    Combination is done by mathematically substituting the previous
      40              :    insn(s) values for the regs they set into the expressions in
      41              :    the later insns that refer to these regs.  If the result is a valid insn
      42              :    for our target machine, according to the machine description,
      43              :    we install it, delete the earlier insns, and update the data flow
      44              :    information (LOG_LINKS and REG_NOTES) for what we did.
      45              : 
      46              :    There are a few exceptions where the dataflow information isn't
      47              :    completely updated (however this is only a local issue since it is
      48              :    regenerated before the next pass that uses it):
      49              : 
      50              :    - reg_live_length is not updated
      51              :    - reg_n_refs is not adjusted in the rare case when a register is
      52              :      no longer required in a computation
      53              :    - there are extremely rare cases (see distribute_notes) when a
      54              :      REG_DEAD note is lost
      55              :    - a LOG_LINKS entry that refers to an insn with multiple SETs may be
      56              :      removed because there is no way to know which register it was
      57              :      linking
      58              : 
      59              :    To simplify substitution, we combine only when the earlier insn(s)
      60              :    consist of only a single assignment.  To simplify updating afterward,
      61              :    we never combine when a subroutine call appears in the middle.  */
      62              : 
      63              : #include "config.h"
      64              : #include "system.h"
      65              : #include "coretypes.h"
      66              : #include "backend.h"
      67              : #include "target.h"
      68              : #include "rtl.h"
      69              : #include "tree.h"
      70              : #include "cfghooks.h"
      71              : #include "predict.h"
      72              : #include "df.h"
      73              : #include "memmodel.h"
      74              : #include "tm_p.h"
      75              : #include "optabs.h"
      76              : #include "regs.h"
      77              : #include "emit-rtl.h"
      78              : #include "recog.h"
      79              : #include "cgraph.h"
      80              : #include "stor-layout.h"
      81              : #include "cfgrtl.h"
      82              : #include "cfgcleanup.h"
      83              : /* Include expr.h after insn-config.h so we get HAVE_conditional_move.  */
      84              : #include "explow.h"
      85              : #include "insn-attr.h"
      86              : #include "rtlhooks-def.h"
      87              : #include "expr.h"
      88              : #include "tree-pass.h"
      89              : #include "valtrack.h"
      90              : #include "rtl-iter.h"
      91              : #include "print-rtl.h"
      92              : #include "function-abi.h"
      93              : #include "rtlanal.h"
      94              : 
      95              : /* Number of attempts to combine instructions in this function.  */
      96              : 
      97              : static int combine_attempts;
      98              : 
      99              : /* Number of attempts that got as far as substitution in this function.  */
     100              : 
     101              : static int combine_merges;
     102              : 
     103              : /* Number of instructions combined with added SETs in this function.  */
     104              : 
     105              : static int combine_extras;
     106              : 
     107              : /* Number of instructions combined in this function.  */
     108              : 
     109              : static int combine_successes;
     110              : 
     111              : /* combine_instructions may try to replace the right hand side of the
     112              :    second instruction with the value of an associated REG_EQUAL note
     113              :    before throwing it at try_combine.  That is problematic when there
     114              :    is a REG_DEAD note for a register used in the old right hand side
     115              :    and can cause distribute_notes to do wrong things.  This is the
     116              :    second instruction if it has been so modified, null otherwise.  */
     117              : 
     118              : static rtx_insn *i2mod;
     119              : 
     120              : /* When I2MOD is nonnull, this is a copy of the old right hand side.  */
     121              : 
     122              : static rtx i2mod_old_rhs;
     123              : 
     124              : /* When I2MOD is nonnull, this is a copy of the new right hand side.  */
     125              : 
     126              : static rtx i2mod_new_rhs;
     127              : 
     128              : struct reg_stat_type {
     129              :   /* Record last point of death of (hard or pseudo) register n.  */
     130              :   rtx_insn                      *last_death;
     131              : 
     132              :   /* Record last point of modification of (hard or pseudo) register n.  */
     133              :   rtx_insn                      *last_set;
     134              : 
     135              :   /* The next group of fields allows the recording of the last value assigned
     136              :      to (hard or pseudo) register n.  We use this information to see if an
     137              :      operation being processed is redundant given a prior operation performed
     138              :      on the register.  For example, an `and' with a constant is redundant if
     139              :      all the zero bits are already known to be turned off.
     140              : 
     141              :      We use an approach similar to that used by cse, but change it in the
     142              :      following ways:
     143              : 
     144              :      (1) We do not want to reinitialize at each label.
     145              :      (2) It is useful, but not critical, to know the actual value assigned
     146              :          to a register.  Often just its form is helpful.
     147              : 
     148              :      Therefore, we maintain the following fields:
     149              : 
     150              :      last_set_value             the last value assigned
     151              :      last_set_label             records the value of label_tick when the
     152              :                                 register was assigned
     153              :      last_set_table_tick        records the value of label_tick when a
     154              :                                 value using the register is assigned
     155              :      last_set_invalid           set to true when it is not valid
     156              :                                 to use the value of this register in some
     157              :                                 register's value
     158              : 
     159              :      To understand the usage of these tables, it is important to understand
     160              :      the distinction between the value in last_set_value being valid and
     161              :      the register being validly contained in some other expression in the
     162              :      table.
     163              : 
     164              :      (The next two parameters are out of date).
     165              : 
     166              :      reg_stat[i].last_set_value is valid if it is nonzero, and either
     167              :      reg_n_sets[i] is 1 or reg_stat[i].last_set_label == label_tick.
     168              : 
     169              :      Register I may validly appear in any expression returned for the value
     170              :      of another register if reg_n_sets[i] is 1.  It may also appear in the
     171              :      value for register J if reg_stat[j].last_set_invalid is zero, or
     172              :      reg_stat[i].last_set_label < reg_stat[j].last_set_label.
     173              : 
     174              :      If an expression is found in the table containing a register which may
     175              :      not validly appear in an expression, the register is replaced by
     176              :      something that won't match, (clobber (const_int 0)).  */
     177              : 
     178              :   /* Record last value assigned to (hard or pseudo) register n.  */
     179              : 
     180              :   rtx                           last_set_value;
     181              : 
     182              :   /* Record the value of label_tick when an expression involving register n
     183              :      is placed in last_set_value.  */
     184              : 
     185              :   int                           last_set_table_tick;
     186              : 
     187              :   /* Record the value of label_tick when the value for register n is placed in
     188              :      last_set_value.  */
     189              : 
     190              :   int                           last_set_label;
     191              : 
     192              :   /* These fields are maintained in parallel with last_set_value and are
     193              :      used to store the mode in which the register was last set, the bits
     194              :      that were known to be zero when it was last set, and the number of
     195              :      sign bits copies it was known to have when it was last set.  */
     196              : 
     197              :   unsigned HOST_WIDE_INT        last_set_nonzero_bits;
     198              :   unsigned short                last_set_sign_bit_copies;
     199              :   machine_mode                  last_set_mode : MACHINE_MODE_BITSIZE;
     200              : 
     201              :   /* Set to true if references to register n in expressions should not be
     202              :      used.  last_set_invalid is set nonzero when this register is being
     203              :      assigned to and last_set_table_tick == label_tick.  */
     204              : 
     205              :   bool                          last_set_invalid;
     206              : 
     207              :   /* Some registers that are set more than once and used in more than one
     208              :      basic block are nevertheless always set in similar ways.  For example,
     209              :      a QImode register may be loaded from memory in two places on a machine
     210              :      where byte loads zero extend.
     211              : 
     212              :      We record in the following fields if a register has some leading bits
     213              :      that are always equal to the sign bit, and what we know about the
     214              :      nonzero bits of a register, specifically which bits are known to be
     215              :      zero.
     216              : 
     217              :      If an entry is zero, it means that we don't know anything special.  */
     218              : 
     219              :   unsigned short                sign_bit_copies;
     220              : 
     221              :   unsigned HOST_WIDE_INT        nonzero_bits;
     222              : 
     223              :   /* Record the value of the label_tick when the last truncation
     224              :      happened.  The field truncated_to_mode is only valid if
     225              :      truncation_label == label_tick.  */
     226              : 
     227              :   int                           truncation_label;
     228              : 
     229              :   /* Record the last truncation seen for this register.  If truncation
     230              :      is not a nop to this mode we might be able to save an explicit
     231              :      truncation if we know that value already contains a truncated
     232              :      value.  */
     233              : 
     234              :   machine_mode                  truncated_to_mode : MACHINE_MODE_BITSIZE;
     235              : };
     236              : 
     237              : 
     238              : static vec<reg_stat_type> reg_stat;
     239              : 
     240              : /* One plus the highest pseudo for which we track REG_N_SETS.
     241              :    regstat_init_n_sets_and_refs allocates the array for REG_N_SETS just once,
     242              :    but during combine_split_insns new pseudos can be created.  As we don't have
     243              :    updated DF information in that case, it is hard to initialize the array
     244              :    after growing.  The combiner only cares about REG_N_SETS (regno) == 1,
     245              :    so instead of growing the arrays, just assume all newly created pseudos
     246              :    during combine might be set multiple times.  */
     247              : 
     248              : static unsigned int reg_n_sets_max;
     249              : 
     250              : /* Record the luid of the last insn that invalidated memory
     251              :    (anything that writes memory, and subroutine calls, but not pushes).  */
     252              : 
     253              : static int mem_last_set;
     254              : 
     255              : /* Record the luid of the last CALL_INSN
     256              :    so we can tell whether a potential combination crosses any calls.  */
     257              : 
     258              : static int last_call_luid;
     259              : 
     260              : /* When `subst' is called, this is the insn that is being modified
     261              :    (by combining in a previous insn).  The PATTERN of this insn
     262              :    is still the old pattern partially modified and it should not be
     263              :    looked at, but this may be used to examine the successors of the insn
     264              :    to judge whether a simplification is valid.  */
     265              : 
     266              : static rtx_insn *subst_insn;
     267              : 
     268              : /* This is the lowest LUID that `subst' is currently dealing with.
     269              :    get_last_value will not return a value if the register was set at or
     270              :    after this LUID.  If not for this mechanism, we could get confused if
     271              :    I2 or I1 in try_combine were an insn that used the old value of a register
     272              :    to obtain a new value.  In that case, we might erroneously get the
     273              :    new value of the register when we wanted the old one.  */
     274              : 
     275              : static int subst_low_luid;
     276              : 
     277              : /* This contains any hard registers that are used in newpat; reg_dead_at_p
     278              :    must consider all these registers to be always live.  */
     279              : 
     280              : static HARD_REG_SET newpat_used_regs;
     281              : 
     282              : /* This is an insn to which a LOG_LINKS entry has been added.  If this
     283              :    insn is the earlier than I2 or I3, combine should rescan starting at
     284              :    that location.  */
     285              : 
     286              : static rtx_insn *added_links_insn;
     287              : 
     288              : /* And similarly, for notes.  */
     289              : 
     290              : static rtx_insn *added_notes_insn;
     291              : 
     292              : /* Basic block in which we are performing combines.  */
     293              : static basic_block this_basic_block;
     294              : static bool optimize_this_for_speed_p;
     295              : 
     296              : 
     297              : /* Length of the currently allocated uid_insn_cost array.  */
     298              : 
     299              : static int max_uid_known;
     300              : 
     301              : /* The following array records the insn_cost for every insn
     302              :    in the instruction stream.  */
     303              : 
     304              : static int *uid_insn_cost;
     305              : 
     306              : /* The following array records the LOG_LINKS for every insn in the
     307              :    instruction stream as struct insn_link pointers.  */
     308              : 
     309              : struct insn_link {
     310              :   rtx_insn *insn;
     311              :   unsigned int regno;
     312              :   int insn_count;
     313              :   struct insn_link *next;
     314              : };
     315              : 
     316              : static struct insn_link **uid_log_links;
     317              : 
     318              : static inline int
     319    778096534 : insn_uid_check (const rtx_insn *insn)
     320              : {
     321    778096534 :   int uid = INSN_UID (insn);
     322    778096534 :   gcc_checking_assert (uid <= max_uid_known);
     323    778096534 :   return uid;
     324              : }
     325              : 
     326              : #define INSN_COST(INSN)         (uid_insn_cost[insn_uid_check (INSN)])
     327              : #define LOG_LINKS(INSN)         (uid_log_links[insn_uid_check (INSN)])
     328              : 
     329              : #define FOR_EACH_LOG_LINK(L, INSN)                              \
     330              :   for ((L) = LOG_LINKS (INSN); (L); (L) = (L)->next)
     331              : 
     332              : /* Links for LOG_LINKS are allocated from this obstack.  */
     333              : 
     334              : static struct obstack insn_link_obstack;
     335              : 
     336              : /* Allocate a link.  */
     337              : 
     338              : static inline struct insn_link *
     339     39040073 : alloc_insn_link (rtx_insn *insn, unsigned int regno, struct insn_link *next)
     340              : {
     341     39040073 :   struct insn_link *l
     342     39040073 :     = (struct insn_link *) obstack_alloc (&insn_link_obstack,
     343              :                                           sizeof (struct insn_link));
     344     39040073 :   l->insn = insn;
     345     39040073 :   l->regno = regno;
     346     39040073 :   l->insn_count = 0;
     347     39040073 :   l->next = next;
     348     39040073 :   return l;
     349              : }
     350              : 
     351              : /* Incremented for each basic block.  */
     352              : 
     353              : static int label_tick;
     354              : 
     355              : /* Reset to label_tick for each extended basic block in scanning order.  */
     356              : 
     357              : static int label_tick_ebb_start;
     358              : 
     359              : /* Mode used to compute significance in reg_stat[].nonzero_bits.  It is the
     360              :    largest integer mode that can fit in HOST_BITS_PER_WIDE_INT.  */
     361              : 
     362              : static scalar_int_mode nonzero_bits_mode;
     363              : 
     364              : /* Nonzero when reg_stat[].nonzero_bits and reg_stat[].sign_bit_copies can
     365              :    be safely used.  It is zero while computing them and after combine has
     366              :    completed.  This former test prevents propagating values based on
     367              :    previously set values, which can be incorrect if a variable is modified
     368              :    in a loop.  */
     369              : 
     370              : static int nonzero_sign_valid;
     371              : 
     372              : 
     373              : /* Record one modification to rtl structure
     374              :    to be undone by storing old_contents into *where.  */
     375              : 
     376              : enum undo_kind { UNDO_RTX, UNDO_INT, UNDO_MODE, UNDO_LINKS };
     377              : 
     378              : struct undo
     379              : {
     380              :   struct undo *next;
     381              :   enum undo_kind kind;
     382              :   union { rtx r; int i; machine_mode m; struct insn_link *l; } old_contents;
     383              :   union { rtx *r; int *i; int regno; struct insn_link **l; } where;
     384              : };
     385              : 
     386              : /* Record a bunch of changes to be undone, up to MAX_UNDO of them.
     387              :    num_undo says how many are currently recorded.
     388              : 
     389              :    other_insn is nonzero if we have modified some other insn in the process
     390              :    of working on subst_insn.  It must be verified too.  */
     391              : 
     392              : struct undobuf
     393              : {
     394              :   struct undo *undos;
     395              :   struct undo *frees;
     396              :   rtx_insn *other_insn;
     397              : };
     398              : 
     399              : static struct undobuf undobuf;
     400              : 
     401              : /* Number of times the pseudo being substituted for
     402              :    was found and replaced.  */
     403              : 
     404              : static int n_occurrences;
     405              : 
     406              : static rtx reg_nonzero_bits_for_combine (const_rtx, scalar_int_mode,
     407              :                                          scalar_int_mode,
     408              :                                          unsigned HOST_WIDE_INT *);
     409              : static rtx reg_num_sign_bit_copies_for_combine (const_rtx, scalar_int_mode,
     410              :                                                 scalar_int_mode,
     411              :                                                 unsigned int *);
     412              : static void do_SUBST (rtx *, rtx);
     413              : static void do_SUBST_INT (int *, int);
     414              : static void init_reg_last (void);
     415              : static void setup_incoming_promotions (rtx_insn *);
     416              : static void set_nonzero_bits_and_sign_copies (rtx, const_rtx, void *);
     417              : static bool cant_combine_insn_p (rtx_insn *);
     418              : static bool can_combine_p (rtx_insn *, rtx_insn *, rtx_insn *, rtx_insn *,
     419              :                            rtx_insn *, rtx_insn *, rtx *, rtx *);
     420              : static bool combinable_i3pat (rtx_insn *, rtx *, rtx, rtx, rtx,
     421              :                               bool, bool, rtx *);
     422              : static bool contains_muldiv (rtx);
     423              : static rtx_insn *try_combine (rtx_insn *, rtx_insn *, rtx_insn *, rtx_insn *,
     424              :                               bool *, rtx_insn *);
     425              : static void undo_all (void);
     426              : static void undo_commit (void);
     427              : static rtx *find_split_point (rtx *, rtx_insn *, bool);
     428              : static rtx subst (rtx, rtx, rtx, bool, bool, bool);
     429              : static rtx combine_simplify_rtx (rtx, machine_mode, bool, bool);
     430              : static rtx simplify_if_then_else (rtx);
     431              : static rtx simplify_set (rtx);
     432              : static rtx simplify_logical (rtx);
     433              : static rtx expand_compound_operation (rtx);
     434              : static const_rtx expand_field_assignment (const_rtx);
     435              : static rtx make_extraction (machine_mode, rtx, HOST_WIDE_INT, rtx,
     436              :                             unsigned HOST_WIDE_INT, bool, bool, bool);
     437              : static int get_pos_from_mask (unsigned HOST_WIDE_INT,
     438              :                               unsigned HOST_WIDE_INT *);
     439              : static rtx canon_reg_for_combine (rtx, rtx);
     440              : static rtx force_int_to_mode (rtx, scalar_int_mode, scalar_int_mode,
     441              :                               scalar_int_mode, unsigned HOST_WIDE_INT, bool);
     442              : static rtx force_to_mode (rtx, machine_mode,
     443              :                           unsigned HOST_WIDE_INT, bool);
     444              : static rtx if_then_else_cond (rtx, rtx *, rtx *);
     445              : static rtx known_cond (rtx, enum rtx_code, rtx, rtx);
     446              : static bool rtx_equal_for_field_assignment_p (rtx, rtx, bool = false);
     447              : static rtx make_field_assignment (rtx);
     448              : static rtx apply_distributive_law (rtx);
     449              : static rtx distribute_and_simplify_rtx (rtx, int);
     450              : static rtx simplify_and_const_int_1 (scalar_int_mode, rtx,
     451              :                                      unsigned HOST_WIDE_INT);
     452              : static rtx simplify_and_const_int (rtx, scalar_int_mode, rtx,
     453              :                                    unsigned HOST_WIDE_INT);
     454              : static bool merge_outer_ops (enum rtx_code *, HOST_WIDE_INT *, enum rtx_code,
     455              :                              HOST_WIDE_INT, machine_mode, bool *);
     456              : static rtx simplify_shift_const_1 (enum rtx_code, machine_mode, rtx, int);
     457              : static rtx simplify_shift_const (rtx, enum rtx_code, machine_mode, rtx,
     458              :                                  int);
     459              : static int recog_for_combine (rtx *, rtx_insn *, rtx *, unsigned = 0, unsigned = 0);
     460              : static rtx gen_lowpart_for_combine (machine_mode, rtx);
     461              : static rtx gen_lowpart_for_combine_no_emit (machine_mode, rtx);
     462              : static enum rtx_code simplify_compare_const (enum rtx_code, machine_mode,
     463              :                                              rtx *, rtx *);
     464              : static enum rtx_code simplify_comparison (enum rtx_code, rtx *, rtx *);
     465              : static void update_table_tick (rtx);
     466              : static void record_value_for_reg (rtx, rtx_insn *, rtx);
     467              : static void check_promoted_subreg (rtx_insn *, rtx);
     468              : static void record_dead_and_set_regs_1 (rtx, const_rtx, void *);
     469              : static void record_dead_and_set_regs (rtx_insn *);
     470              : static bool get_last_value_validate (rtx *, rtx_insn *, int, bool);
     471              : static rtx get_last_value (const_rtx);
     472              : static void reg_dead_at_p_1 (rtx, const_rtx, void *);
     473              : static bool reg_dead_at_p (rtx, rtx_insn *);
     474              : static void move_deaths (rtx, rtx, int, rtx_insn *, rtx *);
     475              : static bool reg_bitfield_target_p (rtx, rtx);
     476              : static void distribute_notes (rtx, rtx_insn *, rtx_insn *, rtx_insn *,
     477              :                               rtx, rtx, rtx);
     478              : static void distribute_links (struct insn_link *, rtx_insn * = nullptr,
     479              :                               int limit = INT_MAX);
     480              : static void mark_used_regs_combine (rtx);
     481              : static void record_promoted_value (rtx_insn *, rtx);
     482              : static bool unmentioned_reg_p (rtx, rtx);
     483              : static void record_truncated_values (rtx *, void *);
     484              : static bool reg_truncated_to_mode (machine_mode, const_rtx);
     485              : static rtx gen_lowpart_or_truncate (machine_mode, rtx);
     486              : 
     487              : 
     488              : /* It is not safe to use ordinary gen_lowpart in combine.
     489              :    See comments in gen_lowpart_for_combine.  */
     490              : #undef RTL_HOOKS_GEN_LOWPART
     491              : #define RTL_HOOKS_GEN_LOWPART              gen_lowpart_for_combine
     492              : 
     493              : /* Our implementation of gen_lowpart never emits a new pseudo.  */
     494              : #undef RTL_HOOKS_GEN_LOWPART_NO_EMIT
     495              : #define RTL_HOOKS_GEN_LOWPART_NO_EMIT      gen_lowpart_for_combine_no_emit
     496              : 
     497              : #undef RTL_HOOKS_REG_NONZERO_REG_BITS
     498              : #define RTL_HOOKS_REG_NONZERO_REG_BITS     reg_nonzero_bits_for_combine
     499              : 
     500              : #undef RTL_HOOKS_REG_NUM_SIGN_BIT_COPIES
     501              : #define RTL_HOOKS_REG_NUM_SIGN_BIT_COPIES  reg_num_sign_bit_copies_for_combine
     502              : 
     503              : #undef RTL_HOOKS_REG_TRUNCATED_TO_MODE
     504              : #define RTL_HOOKS_REG_TRUNCATED_TO_MODE    reg_truncated_to_mode
     505              : 
     506              : static const struct rtl_hooks combine_rtl_hooks = RTL_HOOKS_INITIALIZER;
     507              : 
     508              : 
     509              : /* Convenience wrapper for the canonicalize_comparison target hook.
     510              :    Target hooks cannot use enum rtx_code.  */
     511              : static inline void
     512     25256951 : target_canonicalize_comparison (enum rtx_code *code, rtx *op0, rtx *op1,
     513              :                                 bool op0_preserve_value)
     514              : {
     515     25256951 :   int code_int = (int)*code;
     516     25256951 :   targetm.canonicalize_comparison (&code_int, op0, op1, op0_preserve_value);
     517     25256951 :   *code = (enum rtx_code)code_int;
     518              : }
     519              : 
     520              : /* Try to split PATTERN found in INSN.  This returns NULL_RTX if
     521              :    PATTERN cannot be split.  Otherwise, it returns an insn sequence.
     522              :    Updates OLD_NREGS with the max number of regs before the split
     523              :    and NEW_NREGS after the split.
     524              :    This is a wrapper around split_insns which ensures that the
     525              :    reg_stat vector is made larger if the splitter creates a new
     526              :    register.  */
     527              : 
     528              : static rtx_insn *
     529     12213578 : combine_split_insns (rtx pattern, rtx_insn *insn,
     530              :                      unsigned int *old_nregs,
     531              :                      unsigned int *new_regs)
     532              : {
     533     12213578 :   rtx_insn *ret;
     534     12213578 :   unsigned int nregs;
     535     12213578 :   *old_nregs = max_reg_num ();
     536     12213578 :   ret = split_insns (pattern, insn);
     537     12213578 :   *new_regs = nregs = max_reg_num ();
     538     24427156 :   if (nregs > reg_stat.length ())
     539         2255 :     reg_stat.safe_grow_cleared (nregs, true);
     540     12213578 :   return ret;
     541              : }
     542              : 
     543              : /* This is used by find_single_use to locate an rtx in LOC that
     544              :    contains exactly one use of DEST, which is typically a REG.
     545              :    It returns a pointer to the innermost rtx expression
     546              :    containing DEST.  Appearances of DEST that are being used to
     547              :    totally replace it are not counted.  */
     548              : 
     549              : static rtx *
     550     33394358 : find_single_use_1 (rtx dest, rtx *loc)
     551              : {
     552     40432486 :   rtx x = *loc;
     553     40432486 :   enum rtx_code code = GET_CODE (x);
     554     40432486 :   rtx *result = NULL;
     555     40432486 :   rtx *this_result;
     556     40432486 :   int i;
     557     40432486 :   const char *fmt;
     558              : 
     559     40432486 :   switch (code)
     560              :     {
     561              :     case CONST:
     562              :     case LABEL_REF:
     563              :     case SYMBOL_REF:
     564              :     CASE_CONST_ANY:
     565              :     case CLOBBER:
     566              :       return 0;
     567              : 
     568      6991767 :     case SET:
     569              :       /* If the destination is anything other than PC, a REG or a SUBREG
     570              :          of a REG that occupies all of the REG, the insn uses DEST if
     571              :          it is mentioned in the destination or the source.  Otherwise, we
     572              :          need just check the source.  */
     573      6991767 :       if (GET_CODE (SET_DEST (x)) != PC
     574      6991767 :           && !REG_P (SET_DEST (x))
     575      6994288 :           && ! (GET_CODE (SET_DEST (x)) == SUBREG
     576         2521 :                 && REG_P (SUBREG_REG (SET_DEST (x)))
     577         2521 :                 && !read_modify_subreg_p (SET_DEST (x))))
     578              :         break;
     579              : 
     580      6990388 :       return find_single_use_1 (dest, &SET_SRC (x));
     581              : 
     582        47740 :     case MEM:
     583        47740 :     case SUBREG:
     584        47740 :       return find_single_use_1 (dest, &XEXP (x, 0));
     585              : 
     586              :     default:
     587              :       break;
     588              :     }
     589              : 
     590              :   /* If it wasn't one of the common cases above, check each expression and
     591              :      vector of this code.  Look for a unique usage of DEST.  */
     592              : 
     593     20116794 :   fmt = GET_RTX_FORMAT (code);
     594     53790501 :   for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
     595              :     {
     596     33681221 :       if (fmt[i] == 'e')
     597              :         {
     598     33291516 :           if (dest == XEXP (x, i)
     599     33291516 :               || (REG_P (dest) && REG_P (XEXP (x, i))
     600       882478 :                   && REGNO (dest) == REGNO (XEXP (x, i))))
     601              :             this_result = loc;
     602              :           else
     603     26296915 :             this_result = find_single_use_1 (dest, &XEXP (x, i));
     604              : 
     605     33291516 :           if (result == NULL)
     606              :             result = this_result;
     607        41558 :           else if (this_result)
     608              :             /* Duplicate usage.  */
     609              :             return NULL;
     610              :         }
     611       389705 :       else if (fmt[i] == 'E')
     612              :         {
     613        53141 :           int j;
     614              : 
     615       162102 :           for (j = XVECLEN (x, i) - 1; j >= 0; j--)
     616              :             {
     617       113058 :               if (XVECEXP (x, i, j) == dest
     618       113058 :                   || (REG_P (dest)
     619       113058 :                       && REG_P (XVECEXP (x, i, j))
     620         5057 :                       && REGNO (XVECEXP (x, i, j)) == REGNO (dest)))
     621              :                 this_result = loc;
     622              :               else
     623       113058 :                 this_result = find_single_use_1 (dest, &XVECEXP (x, i, j));
     624              : 
     625       113058 :               if (result == NULL)
     626              :                 result = this_result;
     627        17920 :               else if (this_result)
     628              :                 return NULL;
     629              :             }
     630              :         }
     631              :     }
     632              : 
     633              :   return result;
     634              : }
     635              : 
     636              : 
     637              : /* See if DEST, produced in INSN, is used only a single time in the
     638              :    sequel.  If so, return a pointer to the innermost rtx expression in which
     639              :    it is used.
     640              : 
     641              :    If PLOC is nonzero, *PLOC is set to the insn containing the single use.
     642              : 
     643              :    Otherwise, we find the single use by finding an insn that has a
     644              :    LOG_LINKS pointing at INSN and has a REG_DEAD note for DEST.  If DEST is
     645              :    only referenced once in that insn, we know that it must be the first
     646              :    and last insn referencing DEST.  */
     647              : 
     648              : static rtx *
     649      7608129 : find_single_use (rtx dest, rtx_insn *insn, rtx_insn **ploc)
     650              : {
     651      7608129 :   basic_block bb;
     652      7608129 :   rtx_insn *next;
     653      7608129 :   rtx *result;
     654      7608129 :   struct insn_link *link;
     655              : 
     656      7608129 :   if (!REG_P (dest))
     657              :     return 0;
     658              : 
     659      7608129 :   bb = BLOCK_FOR_INSN (insn);
     660     10543562 :   for (next = NEXT_INSN (insn);
     661     10543562 :        next && BLOCK_FOR_INSN (next) == bb;
     662      2935433 :        next = NEXT_INSN (next))
     663      9919818 :     if (NONDEBUG_INSN_P (next) && dead_or_set_p (next, dest))
     664              :       {
     665      9343103 :         FOR_EACH_LOG_LINK (link, next)
     666      8294630 :           if (link->insn == insn && link->regno == REGNO (dest))
     667              :             break;
     668              : 
     669      8032858 :         if (link)
     670              :           {
     671      6984385 :             result = find_single_use_1 (dest, &PATTERN (next));
     672      6984385 :             if (ploc)
     673      6984384 :               *ploc = next;
     674              :             return result;
     675              :           }
     676              :       }
     677              : 
     678              :   return 0;
     679              : }
     680              : 
     681              : /* Substitute NEWVAL, an rtx expression, into INTO, a place in some
     682              :    insn.  The substitution can be undone by undo_all.  If INTO is already
     683              :    set to NEWVAL, do not record this change.  Because computing NEWVAL might
     684              :    also call SUBST, we have to compute it before we put anything into
     685              :    the undo table.  */
     686              : 
     687              : static void
     688    768437752 : do_SUBST (rtx *into, rtx newval)
     689              : {
     690    768437752 :   struct undo *buf;
     691    768437752 :   rtx oldval = *into;
     692              : 
     693    768437752 :   if (oldval == newval)
     694              :     return;
     695              : 
     696              :   /* We'd like to catch as many invalid transformations here as
     697              :      possible.  Unfortunately, there are way too many mode changes
     698              :      that are perfectly valid, so we'd waste too much effort for
     699              :      little gain doing the checks here.  Focus on catching invalid
     700              :      transformations involving integer constants.  */
     701     97340739 :   if (GET_MODE_CLASS (GET_MODE (oldval)) == MODE_INT
     702     59202675 :       && CONST_INT_P (newval))
     703              :     {
     704              :       /* Sanity check that we're replacing oldval with a CONST_INT
     705              :          that is a valid sign-extension for the original mode.  */
     706      1841172 :       gcc_assert (INTVAL (newval)
     707              :                   == trunc_int_for_mode (INTVAL (newval), GET_MODE (oldval)));
     708              : 
     709              :       /* Replacing the operand of a SUBREG or a ZERO_EXTEND with a
     710              :          CONST_INT is not valid, because after the replacement, the
     711              :          original mode would be gone.  Unfortunately, we can't tell
     712              :          when do_SUBST is called to replace the operand thereof, so we
     713              :          perform this test on oldval instead, checking whether an
     714              :          invalid replacement took place before we got here.  */
     715      1841172 :       gcc_assert (!(GET_CODE (oldval) == SUBREG
     716              :                     && CONST_INT_P (SUBREG_REG (oldval))));
     717      1841172 :       gcc_assert (!(GET_CODE (oldval) == ZERO_EXTEND
     718              :                     && CONST_INT_P (XEXP (oldval, 0))));
     719              :     }
     720              : 
     721     97340739 :   if (undobuf.frees)
     722     93154770 :     buf = undobuf.frees, undobuf.frees = buf->next;
     723              :   else
     724      4185969 :     buf = XNEW (struct undo);
     725              : 
     726     97340739 :   buf->kind = UNDO_RTX;
     727     97340739 :   buf->where.r = into;
     728     97340739 :   buf->old_contents.r = oldval;
     729     97340739 :   *into = newval;
     730              : 
     731     97340739 :   buf->next = undobuf.undos, undobuf.undos = buf;
     732              : }
     733              : 
     734              : #define SUBST(INTO, NEWVAL)     do_SUBST (&(INTO), (NEWVAL))
     735              : 
     736              : /* Similar to SUBST, but NEWVAL is an int expression.  Note that substitution
     737              :    for the value of a HOST_WIDE_INT value (including CONST_INT) is
     738              :    not safe.  */
     739              : 
     740              : static void
     741     16324411 : do_SUBST_INT (int *into, int newval)
     742              : {
     743     16324411 :   struct undo *buf;
     744     16324411 :   int oldval = *into;
     745              : 
     746     16324411 :   if (oldval == newval)
     747              :     return;
     748              : 
     749      7107602 :   if (undobuf.frees)
     750      6570695 :     buf = undobuf.frees, undobuf.frees = buf->next;
     751              :   else
     752       536907 :     buf = XNEW (struct undo);
     753              : 
     754      7107602 :   buf->kind = UNDO_INT;
     755      7107602 :   buf->where.i = into;
     756      7107602 :   buf->old_contents.i = oldval;
     757      7107602 :   *into = newval;
     758              : 
     759      7107602 :   buf->next = undobuf.undos, undobuf.undos = buf;
     760              : }
     761              : 
     762              : #define SUBST_INT(INTO, NEWVAL)  do_SUBST_INT (&(INTO), (NEWVAL))
     763              : 
     764              : /* Similar to SUBST, but just substitute the mode.  This is used when
     765              :    changing the mode of a pseudo-register, so that any other
     766              :    references to the entry in the regno_reg_rtx array will change as
     767              :    well.  */
     768              : 
     769              : static void
     770      1445972 : subst_mode (int regno, machine_mode newval)
     771              : {
     772      1445972 :   struct undo *buf;
     773      1445972 :   rtx reg = regno_reg_rtx[regno];
     774      1445972 :   machine_mode oldval = GET_MODE (reg);
     775              : 
     776      1445972 :   if (oldval == newval)
     777              :     return;
     778              : 
     779      1445972 :   if (undobuf.frees)
     780      1368892 :     buf = undobuf.frees, undobuf.frees = buf->next;
     781              :   else
     782        77080 :     buf = XNEW (struct undo);
     783              : 
     784      1445972 :   buf->kind = UNDO_MODE;
     785      1445972 :   buf->where.regno = regno;
     786      1445972 :   buf->old_contents.m = oldval;
     787      1445972 :   adjust_reg_mode (reg, newval);
     788              : 
     789      1445972 :   buf->next = undobuf.undos, undobuf.undos = buf;
     790              : }
     791              : 
     792              : /* Similar to SUBST, but NEWVAL is a LOG_LINKS expression.  */
     793              : 
     794              : static void
     795        71886 : do_SUBST_LINK (struct insn_link **into, struct insn_link *newval)
     796              : {
     797        71886 :   struct undo *buf;
     798        71886 :   struct insn_link * oldval = *into;
     799              : 
     800        71886 :   if (oldval == newval)
     801              :     return;
     802              : 
     803        71886 :   if (undobuf.frees)
     804        68710 :     buf = undobuf.frees, undobuf.frees = buf->next;
     805              :   else
     806         3176 :     buf = XNEW (struct undo);
     807              : 
     808        71886 :   buf->kind = UNDO_LINKS;
     809        71886 :   buf->where.l = into;
     810        71886 :   buf->old_contents.l = oldval;
     811        71886 :   *into = newval;
     812              : 
     813        71886 :   buf->next = undobuf.undos, undobuf.undos = buf;
     814              : }
     815              : 
     816              : #define SUBST_LINK(oldval, newval) do_SUBST_LINK (&oldval, newval)
     817              : 
     818              : /* Subroutine of try_combine.  Determine whether the replacement patterns
     819              :    NEWPAT, NEWI2PAT and NEWOTHERPAT are more expensive according to insn_cost
     820              :    than the original sequence I0, I1, I2, I3 and undobuf.other_insn.  Note
     821              :    that I0, I1 and/or NEWI2PAT may be NULL_RTX.  Similarly, NEWOTHERPAT and
     822              :    undobuf.other_insn may also both be NULL_RTX.  Return false if the cost
     823              :    of all the instructions can be estimated and the replacements are more
     824              :    expensive than the original sequence.  */
     825              : 
     826              : static bool
     827      4322895 : combine_validate_cost (rtx_insn *i0, rtx_insn *i1, rtx_insn *i2, rtx_insn *i3,
     828              :                        rtx newpat, rtx newi2pat, rtx newotherpat,
     829              :                        int insn_code, int i2_code, int other_code)
     830              : {
     831      4322895 :   int i0_cost, i1_cost, i2_cost, i3_cost;
     832      4322895 :   int new_i2_cost, new_i3_cost;
     833      4322895 :   int old_cost, new_cost;
     834              : 
     835              :   /* Lookup the original insn_costs.  */
     836      4322895 :   i2_cost = INSN_COST (i2);
     837      4322895 :   i3_cost = INSN_COST (i3);
     838              : 
     839      4322895 :   if (i1)
     840              :     {
     841       122367 :       i1_cost = INSN_COST (i1);
     842       122367 :       if (i0)
     843              :         {
     844         5083 :           i0_cost = INSN_COST (i0);
     845         4954 :           old_cost = (i0_cost > 0 && i1_cost > 0 && i2_cost > 0 && i3_cost > 0
     846        10028 :                       ? i0_cost + i1_cost + i2_cost + i3_cost : 0);
     847              :         }
     848              :       else
     849              :         {
     850       112687 :           old_cost = (i1_cost > 0 && i2_cost > 0 && i3_cost > 0
     851       229969 :                       ? i1_cost + i2_cost + i3_cost : 0);
     852              :           i0_cost = 0;
     853              :         }
     854              :     }
     855              :   else
     856              :     {
     857      4200528 :       old_cost = (i2_cost > 0 && i3_cost > 0) ? i2_cost + i3_cost : 0;
     858              :       i1_cost = i0_cost = 0;
     859              :     }
     860              : 
     861              :   /* If we have split a PARALLEL I2 to I1,I2, we have counted its cost twice;
     862              :      correct that.  */
     863      4322895 :   if (old_cost && i1 && INSN_UID (i1) == INSN_UID (i2))
     864         2260 :     old_cost -= i1_cost;
     865              : 
     866              : 
     867              :   /* Calculate the replacement insn_costs.  */
     868      4322895 :   rtx tmp = PATTERN (i3);
     869      4322895 :   PATTERN (i3) = newpat;
     870      4322895 :   int tmpi = INSN_CODE (i3);
     871      4322895 :   INSN_CODE (i3) = insn_code;
     872      4322895 :   new_i3_cost = insn_cost (i3, optimize_this_for_speed_p);
     873      4322895 :   PATTERN (i3) = tmp;
     874      4322895 :   INSN_CODE (i3) = tmpi;
     875      4322895 :   if (newi2pat)
     876              :     {
     877       224611 :       tmp = PATTERN (i2);
     878       224611 :       PATTERN (i2) = newi2pat;
     879       224611 :       tmpi = INSN_CODE (i2);
     880       224611 :       INSN_CODE (i2) = i2_code;
     881       224611 :       new_i2_cost = insn_cost (i2, optimize_this_for_speed_p);
     882       224611 :       PATTERN (i2) = tmp;
     883       224611 :       INSN_CODE (i2) = tmpi;
     884       224611 :       new_cost = (new_i2_cost > 0 && new_i3_cost > 0)
     885       224611 :                  ? new_i2_cost + new_i3_cost : 0;
     886              :     }
     887              :   else
     888              :     {
     889              :       new_cost = new_i3_cost;
     890              :       new_i2_cost = 0;
     891              :     }
     892              : 
     893      4322895 :   if (undobuf.other_insn)
     894              :     {
     895       225028 :       int old_other_cost, new_other_cost;
     896              : 
     897       225028 :       old_other_cost = INSN_COST (undobuf.other_insn);
     898       225028 :       tmp = PATTERN (undobuf.other_insn);
     899       225028 :       PATTERN (undobuf.other_insn) = newotherpat;
     900       225028 :       tmpi = INSN_CODE (undobuf.other_insn);
     901       225028 :       INSN_CODE (undobuf.other_insn) = other_code;
     902       225028 :       new_other_cost = insn_cost (undobuf.other_insn,
     903              :                                   optimize_this_for_speed_p);
     904       225028 :       PATTERN (undobuf.other_insn) = tmp;
     905       225028 :       INSN_CODE (undobuf.other_insn) = tmpi;
     906       225028 :       if (old_other_cost > 0 && new_other_cost > 0)
     907              :         {
     908       225028 :           old_cost += old_other_cost;
     909       225028 :           new_cost += new_other_cost;
     910              :         }
     911              :       else
     912              :         old_cost = 0;
     913              :     }
     914              : 
     915              :   /* Disallow this combination if both new_cost and old_cost are greater than
     916              :      zero, and new_cost is greater than old cost.  */
     917      4322895 :   bool reject = old_cost > 0 && new_cost > old_cost;
     918              : 
     919      4322895 :   if (dump_file)
     920              :     {
     921          492 :       fprintf (dump_file, "%s combination of insns ",
     922              :                reject ? "rejecting" : "allowing");
     923          248 :       if (i0)
     924            0 :         fprintf (dump_file, "%d, ", INSN_UID (i0));
     925          248 :       if (i1 && INSN_UID (i1) != INSN_UID (i2))
     926            1 :         fprintf (dump_file, "%d, ", INSN_UID (i1));
     927          248 :       fprintf (dump_file, "%d and %d\n", INSN_UID (i2), INSN_UID (i3));
     928              : 
     929          248 :       fprintf (dump_file, "original costs ");
     930          248 :       if (i0)
     931            0 :         fprintf (dump_file, "%d + ", i0_cost);
     932          248 :       if (i1 && INSN_UID (i1) != INSN_UID (i2))
     933            1 :         fprintf (dump_file, "%d + ", i1_cost);
     934          248 :       fprintf (dump_file, "%d + %d = %d\n", i2_cost, i3_cost, old_cost);
     935              : 
     936          248 :       if (newi2pat)
     937           23 :         fprintf (dump_file, "replacement costs %d + %d = %d\n",
     938              :                  new_i2_cost, new_i3_cost, new_cost);
     939              :       else
     940          225 :         fprintf (dump_file, "replacement cost %d\n", new_cost);
     941              :     }
     942              : 
     943      4322895 :   if (reject)
     944              :     return false;
     945              : 
     946              :   /* Update the uid_insn_cost array with the replacement costs.  */
     947      4105154 :   INSN_COST (i2) = new_i2_cost;
     948      4105154 :   INSN_COST (i3) = new_i3_cost;
     949      4105154 :   if (i1)
     950              :     {
     951       103494 :       INSN_COST (i1) = 0;
     952       103494 :       if (i0)
     953         4711 :         INSN_COST (i0) = 0;
     954              :     }
     955              : 
     956              :   return true;
     957              : }
     958              : 
     959              : 
     960              : /* Delete any insns that copy a register to itself.
     961              :    Return true if the CFG was changed.  */
     962              : 
     963              : static bool
     964      1020049 : delete_noop_moves (void)
     965              : {
     966      1020049 :   rtx_insn *insn, *next;
     967      1020049 :   basic_block bb;
     968              : 
     969      1020049 :   bool edges_deleted = false;
     970              : 
     971     11578663 :   FOR_EACH_BB_FN (bb, cfun)
     972              :     {
     973    142835344 :       for (insn = BB_HEAD (bb); insn != NEXT_INSN (BB_END (bb)); insn = next)
     974              :         {
     975    132276730 :           next = NEXT_INSN (insn);
     976    132276730 :           if (INSN_P (insn) && noop_move_p (insn))
     977              :             {
     978         6467 :               if (dump_file)
     979            0 :                 fprintf (dump_file, "deleting noop move %d\n", INSN_UID (insn));
     980              : 
     981         6467 :               edges_deleted |= delete_insn_and_edges (insn);
     982              :             }
     983              :         }
     984              :     }
     985              : 
     986      1020049 :   return edges_deleted;
     987              : }
     988              : 
     989              : 
     990              : /* Return false if we do not want to (or cannot) combine DEF.  */
     991              : static bool
     992     42925779 : can_combine_def_p (df_ref def)
     993              : {
     994              :   /* Do not consider if it is pre/post modification in MEM.  */
     995     42925779 :   if (DF_REF_FLAGS (def) & DF_REF_PRE_POST_MODIFY)
     996              :     return false;
     997              : 
     998     41247654 :   unsigned int regno = DF_REF_REGNO (def);
     999              : 
    1000              :   /* Do not combine frame pointer adjustments.  */
    1001     41247654 :   if ((regno == FRAME_POINTER_REGNUM
    1002            0 :        && (!reload_completed || frame_pointer_needed))
    1003         2095 :       || (!HARD_FRAME_POINTER_IS_FRAME_POINTER
    1004     41247654 :           && regno == HARD_FRAME_POINTER_REGNUM
    1005              :           && (!reload_completed || frame_pointer_needed))
    1006     41245559 :       || (FRAME_POINTER_REGNUM != ARG_POINTER_REGNUM
    1007            0 :           && regno == ARG_POINTER_REGNUM && fixed_regs[regno]))
    1008         2095 :     return false;
    1009              : 
    1010              :   return true;
    1011              : }
    1012              : 
    1013              : /* Return false if we do not want to (or cannot) combine USE.  */
    1014              : static bool
    1015     79524057 : can_combine_use_p (df_ref use)
    1016              : {
    1017              :   /* Do not consider the usage of the stack pointer by function call.  */
    1018            0 :   if (DF_REF_FLAGS (use) & DF_REF_CALL_STACK_USAGE)
    1019            0 :     return false;
    1020              : 
    1021              :   return true;
    1022              : }
    1023              : 
    1024              : /* Fill in log links field for all insns.  */
    1025              : 
    1026              : static void
    1027      1020049 : create_log_links (void)
    1028              : {
    1029      1020049 :   basic_block bb;
    1030      1020049 :   rtx_insn **next_use;
    1031      1020049 :   rtx_insn *insn;
    1032      1020049 :   df_ref def, use;
    1033              : 
    1034      1020049 :   next_use = XCNEWVEC (rtx_insn *, max_reg_num ());
    1035              : 
    1036              :   /* Pass through each block from the end, recording the uses of each
    1037              :      register and establishing log links when def is encountered.
    1038              :      Note that we do not clear next_use array in order to save time,
    1039              :      so we have to test whether the use is in the same basic block as def.
    1040              : 
    1041              :      There are a few cases below when we do not consider the definition or
    1042              :      usage -- these are taken from original flow.c did. Don't ask me why it is
    1043              :      done this way; I don't know and if it works, I don't want to know.  */
    1044              : 
    1045     11578663 :   FOR_EACH_BB_FN (bb, cfun)
    1046              :     {
    1047    142818894 :       FOR_BB_INSNS_REVERSE (bb, insn)
    1048              :         {
    1049    132260280 :           if (!NONDEBUG_INSN_P (insn))
    1050     68987251 :             continue;
    1051              : 
    1052              :           /* Log links are created only once.  */
    1053     63273029 :           gcc_assert (!LOG_LINKS (insn));
    1054              : 
    1055    519596908 :           FOR_EACH_INSN_DEF (def, insn)
    1056              :             {
    1057    456323879 :               unsigned int regno = DF_REF_REGNO (def);
    1058    456323879 :               rtx_insn *use_insn;
    1059              : 
    1060    456323879 :               if (!next_use[regno])
    1061    413398100 :                 continue;
    1062              : 
    1063     42925779 :               if (!can_combine_def_p (def))
    1064      1680220 :                 continue;
    1065              : 
    1066     41245559 :               use_insn = next_use[regno];
    1067     41245559 :               next_use[regno] = NULL;
    1068              : 
    1069     41245559 :               if (BLOCK_FOR_INSN (use_insn) != bb)
    1070      2293835 :                 continue;
    1071              : 
    1072              :               /* flow.c claimed:
    1073              : 
    1074              :                  We don't build a LOG_LINK for hard registers contained
    1075              :                  in ASM_OPERANDs.  If these registers get replaced,
    1076              :                  we might wind up changing the semantics of the insn,
    1077              :                  even if reload can make what appear to be valid
    1078              :                  assignments later.  */
    1079     38952600 :               if (regno < FIRST_PSEUDO_REGISTER
    1080     38951724 :                   && asm_noperands (PATTERN (use_insn)) >= 0)
    1081          876 :                 continue;
    1082              : 
    1083              :               /* Don't add duplicate links between instructions.  */
    1084     38950848 :               struct insn_link *links;
    1085     52292131 :               FOR_EACH_LOG_LINK (links, use_insn)
    1086     13341283 :                 if (insn == links->insn && regno == links->regno)
    1087              :                   break;
    1088              : 
    1089     38950848 :               if (!links)
    1090     38950848 :                 LOG_LINKS (use_insn)
    1091     77901696 :                   = alloc_insn_link (insn, regno, LOG_LINKS (use_insn));
    1092              :             }
    1093              : 
    1094    142797086 :           FOR_EACH_INSN_USE (use, insn)
    1095    154249224 :             if (can_combine_use_p (use))
    1096     74725167 :               next_use[DF_REF_REGNO (use)] = insn;
    1097              :         }
    1098              :     }
    1099              : 
    1100      1020049 :   free (next_use);
    1101      1020049 : }
    1102              : 
    1103              : /* Walk the LOG_LINKS of insn B to see if we find a reference to A.  Return
    1104              :    true if we found a LOG_LINK that proves that A feeds B.  This only works
    1105              :    if there are no instructions between A and B which could have a link
    1106              :    depending on A, since in that case we would not record a link for B.  */
    1107              : 
    1108              : static bool
    1109     13367602 : insn_a_feeds_b (rtx_insn *a, rtx_insn *b)
    1110              : {
    1111     13367602 :   struct insn_link *links;
    1112     16879722 :   FOR_EACH_LOG_LINK (links, b)
    1113     14209691 :     if (links->insn == a)
    1114              :       return true;
    1115              :   return false;
    1116              : }
    1117              : 
    1118              : /* Main entry point for combiner.  F is the first insn of the function.
    1119              :    NREGS is the first unused pseudo-reg number.
    1120              : 
    1121              :    Return nonzero if the CFG was changed (e.g. if the combiner has
    1122              :    turned an indirect jump instruction into a direct jump).  */
    1123              : static bool
    1124      1064963 : combine_instructions (rtx_insn *f, unsigned int nregs)
    1125              : {
    1126      1064963 :   rtx_insn *insn, *next;
    1127      1064963 :   struct insn_link *links, *nextlinks;
    1128      1064963 :   rtx_insn *first;
    1129      1064963 :   basic_block last_bb;
    1130              : 
    1131      1064963 :   bool new_direct_jump_p = false;
    1132              : 
    1133      3190232 :   for (first = f; first && !NONDEBUG_INSN_P (first); )
    1134      2125269 :     first = NEXT_INSN (first);
    1135      1064963 :   if (!first)
    1136              :     return false;
    1137              : 
    1138      1020049 :   combine_attempts = 0;
    1139      1020049 :   combine_merges = 0;
    1140      1020049 :   combine_extras = 0;
    1141      1020049 :   combine_successes = 0;
    1142              : 
    1143      1020049 :   rtl_hooks = combine_rtl_hooks;
    1144              : 
    1145      1020049 :   reg_stat.safe_grow_cleared (nregs, true);
    1146              : 
    1147      1020049 :   init_recog_no_volatile ();
    1148              : 
    1149              :   /* Allocate array for insn info.  */
    1150      1020049 :   max_uid_known = get_max_uid ();
    1151      1020049 :   uid_log_links = XCNEWVEC (struct insn_link *, max_uid_known + 1);
    1152      1020049 :   uid_insn_cost = XCNEWVEC (int, max_uid_known + 1);
    1153      1020049 :   gcc_obstack_init (&insn_link_obstack);
    1154              : 
    1155      1020049 :   nonzero_bits_mode = int_mode_for_size (HOST_BITS_PER_WIDE_INT, 0).require ();
    1156              : 
    1157              :   /* Don't use reg_stat[].nonzero_bits when computing it.  This can cause
    1158              :      problems when, for example, we have j <<= 1 in a loop.  */
    1159              : 
    1160      1020049 :   nonzero_sign_valid = 0;
    1161      1020049 :   label_tick = label_tick_ebb_start = 1;
    1162              : 
    1163              :   /* Scan all SETs and see if we can deduce anything about what
    1164              :      bits are known to be zero for some registers and how many copies
    1165              :      of the sign bit are known to exist for those registers.
    1166              : 
    1167              :      Also set any known values so that we can use it while searching
    1168              :      for what bits are known to be set.  */
    1169              : 
    1170      1020049 :   setup_incoming_promotions (first);
    1171              :   /* Allow the entry block and the first block to fall into the same EBB.
    1172              :      Conceptually the incoming promotions are assigned to the entry block.  */
    1173      1020049 :   last_bb = ENTRY_BLOCK_PTR_FOR_FN (cfun);
    1174              : 
    1175      1020049 :   create_log_links ();
    1176     11578663 :   FOR_EACH_BB_FN (this_basic_block, cfun)
    1177              :     {
    1178     10558614 :       optimize_this_for_speed_p = optimize_bb_for_speed_p (this_basic_block);
    1179     10558614 :       last_call_luid = 0;
    1180     10558614 :       mem_last_set = -1;
    1181              : 
    1182     10558614 :       label_tick++;
    1183     10558614 :       if (!single_pred_p (this_basic_block)
    1184     10558614 :           || single_pred (this_basic_block) != last_bb)
    1185      5061684 :         label_tick_ebb_start = label_tick;
    1186     10558614 :       last_bb = this_basic_block;
    1187              : 
    1188    142818894 :       FOR_BB_INSNS (this_basic_block, insn)
    1189    132260280 :         if (INSN_P (insn) && BLOCK_FOR_INSN (insn))
    1190              :           {
    1191    115519794 :             rtx links;
    1192              : 
    1193    115519794 :             subst_low_luid = DF_INSN_LUID (insn);
    1194    115519794 :             subst_insn = insn;
    1195              : 
    1196    115519794 :             note_stores (insn, set_nonzero_bits_and_sign_copies, insn);
    1197    115519794 :             record_dead_and_set_regs (insn);
    1198              : 
    1199    115519794 :             if (AUTO_INC_DEC)
    1200              :               for (links = REG_NOTES (insn); links; links = XEXP (links, 1))
    1201              :                 if (REG_NOTE_KIND (links) == REG_INC)
    1202              :                   set_nonzero_bits_and_sign_copies (XEXP (links, 0), NULL_RTX,
    1203              :                                                     insn);
    1204              : 
    1205              :             /* Record the current insn_cost of this instruction.  */
    1206    115519794 :             INSN_COST (insn) = insn_cost (insn, optimize_this_for_speed_p);
    1207    115519794 :             if (dump_file)
    1208              :               {
    1209         1695 :                 fprintf (dump_file, "insn_cost %d for ", INSN_COST (insn));
    1210         1695 :                 dump_insn_slim (dump_file, insn);
    1211              :               }
    1212              :           }
    1213              :     }
    1214              : 
    1215      1020049 :   nonzero_sign_valid = 1;
    1216              : 
    1217              :   /* Now scan all the insns in forward order.  */
    1218      1020049 :   label_tick = label_tick_ebb_start = 1;
    1219      1020049 :   init_reg_last ();
    1220      1020049 :   setup_incoming_promotions (first);
    1221      1020049 :   last_bb = ENTRY_BLOCK_PTR_FOR_FN (cfun);
    1222      1020049 :   int max_combine = param_max_combine_insns;
    1223              : 
    1224     11578663 :   FOR_EACH_BB_FN (this_basic_block, cfun)
    1225              :     {
    1226     10558614 :       rtx_insn *last_combined_insn = NULL;
    1227              : 
    1228              :       /* Ignore instruction combination in basic blocks that are going to
    1229              :          be removed as unreachable anyway.  See PR82386.  */
    1230     10558614 :       if (EDGE_COUNT (this_basic_block->preds) == 0)
    1231         1660 :         continue;
    1232              : 
    1233     10556954 :       optimize_this_for_speed_p = optimize_bb_for_speed_p (this_basic_block);
    1234     10556954 :       last_call_luid = 0;
    1235     10556954 :       mem_last_set = -1;
    1236              : 
    1237     10556954 :       label_tick++;
    1238     10556954 :       if (!single_pred_p (this_basic_block)
    1239     10556954 :           || single_pred (this_basic_block) != last_bb)
    1240      5061316 :         label_tick_ebb_start = label_tick;
    1241     10556954 :       last_bb = this_basic_block;
    1242              : 
    1243     10556954 :       rtl_profile_for_bb (this_basic_block);
    1244     10556954 :       for (insn = BB_HEAD (this_basic_block);
    1245    147453277 :            insn != NEXT_INSN (BB_END (this_basic_block));
    1246    132791169 :            insn = next ? next : NEXT_INSN (insn))
    1247              :         {
    1248    136896323 :           next = 0;
    1249    136896323 :           if (!NONDEBUG_INSN_P (insn))
    1250     69223800 :             continue;
    1251              : 
    1252              :           while (last_combined_insn
    1253     67674306 :                  && (!NONDEBUG_INSN_P (last_combined_insn)
    1254     57305281 :                      || last_combined_insn->deleted ()))
    1255         1783 :             last_combined_insn = PREV_INSN (last_combined_insn);
    1256     67672523 :           if (last_combined_insn == NULL_RTX
    1257     57304697 :               || BLOCK_FOR_INSN (last_combined_insn) != this_basic_block
    1258    124977015 :               || DF_INSN_LUID (last_combined_insn) <= DF_INSN_LUID (insn))
    1259              :             last_combined_insn = insn;
    1260              : 
    1261              :           /* See if we know about function return values before this
    1262              :              insn based upon SUBREG flags.  */
    1263     67672523 :           check_promoted_subreg (insn, PATTERN (insn));
    1264              : 
    1265              :           /* See if we can find hardregs and subreg of pseudos in
    1266              :              narrower modes.  This could help turning TRUNCATEs
    1267              :              into SUBREGs.  */
    1268     67672523 :           note_uses (&PATTERN (insn), record_truncated_values, NULL);
    1269              : 
    1270              :           /* Try this insn with each insn it links back to.  */
    1271              : 
    1272    105685401 :           FOR_EACH_LOG_LINK (links, insn)
    1273     41989914 :             if ((next = try_combine (insn, links->insn, NULL,
    1274              :                                      NULL, &new_direct_jump_p,
    1275              :                                      last_combined_insn)) != 0)
    1276              :               {
    1277      3977036 :                 statistics_counter_event (cfun, "two-insn combine", 1);
    1278      3977036 :                 goto retry;
    1279              :               }
    1280              : 
    1281              :           /* Try each sequence of three linked insns ending with this one.  */
    1282              : 
    1283     63695487 :           if (max_combine >= 3)
    1284    101103639 :             FOR_EACH_LOG_LINK (links, insn)
    1285              :               {
    1286     37591038 :                 rtx_insn *link = links->insn;
    1287              : 
    1288              :                 /* If the linked insn has been replaced by a note, then there
    1289              :                    is no point in pursuing this chain any further.  */
    1290     37591038 :                 if (NOTE_P (link))
    1291          227 :                   continue;
    1292              : 
    1293     55971337 :                 FOR_EACH_LOG_LINK (nextlinks, link)
    1294     18461375 :                   if ((next = try_combine (insn, link, nextlinks->insn,
    1295              :                                            NULL, &new_direct_jump_p,
    1296              :                                            last_combined_insn)) != 0)
    1297              :                     {
    1298        80849 :                       statistics_counter_event (cfun, "three-insn combine", 1);
    1299        80849 :                       goto retry;
    1300              :                     }
    1301              :               }
    1302              : 
    1303              :           /* Try combining an insn with two different insns whose results it
    1304              :              uses.  */
    1305     63512601 :           if (max_combine >= 3)
    1306    100985187 :             FOR_EACH_LOG_LINK (links, insn)
    1307     50509470 :               for (nextlinks = links->next; nextlinks;
    1308     13021422 :                    nextlinks = nextlinks->next)
    1309     13036884 :                 if ((next = try_combine (insn, links->insn,
    1310              :                                          nextlinks->insn, NULL,
    1311              :                                          &new_direct_jump_p,
    1312              :                                          last_combined_insn)) != 0)
    1313              : 
    1314              :                   {
    1315        15462 :                     statistics_counter_event (cfun, "three-insn combine", 1);
    1316        15462 :                     goto retry;
    1317              :                   }
    1318              : 
    1319              :           /* Try four-instruction combinations.  */
    1320     63497139 :           if (max_combine >= 4)
    1321    100961742 :             FOR_EACH_LOG_LINK (links, insn)
    1322              :               {
    1323     37469261 :                 struct insn_link *next1;
    1324     37469261 :                 rtx_insn *link = links->insn;
    1325              : 
    1326              :                 /* If the linked insn has been replaced by a note, then there
    1327              :                    is no point in pursuing this chain any further.  */
    1328     37469261 :                 if (NOTE_P (link))
    1329          226 :                   continue;
    1330              : 
    1331     55825947 :                 FOR_EACH_LOG_LINK (next1, link)
    1332              :                   {
    1333     18358349 :                     rtx_insn *link1 = next1->insn;
    1334     18358349 :                     if (NOTE_P (link1))
    1335           57 :                       continue;
    1336              :                     /* I0 -> I1 -> I2 -> I3.  */
    1337     30021627 :                     FOR_EACH_LOG_LINK (nextlinks, link1)
    1338     11664663 :                       if ((next = try_combine (insn, link, link1,
    1339              :                                                nextlinks->insn,
    1340              :                                                &new_direct_jump_p,
    1341              :                                                last_combined_insn)) != 0)
    1342              :                         {
    1343         1328 :                           statistics_counter_event (cfun, "four-insn combine", 1);
    1344         1328 :                           goto retry;
    1345              :                         }
    1346              :                     /* I0, I1 -> I2, I2 -> I3.  */
    1347     22360304 :                     for (nextlinks = next1->next; nextlinks;
    1348      4003340 :                          nextlinks = nextlinks->next)
    1349      4003449 :                       if ((next = try_combine (insn, link, link1,
    1350              :                                                nextlinks->insn,
    1351              :                                                &new_direct_jump_p,
    1352              :                                                last_combined_insn)) != 0)
    1353              :                         {
    1354          109 :                           statistics_counter_event (cfun, "four-insn combine", 1);
    1355          109 :                           goto retry;
    1356              :                         }
    1357              :                   }
    1358              : 
    1359     50485542 :                 for (next1 = links->next; next1; next1 = next1->next)
    1360              :                   {
    1361     13021086 :                     rtx_insn *link1 = next1->insn;
    1362     13021086 :                     if (NOTE_P (link1))
    1363            8 :                       continue;
    1364              :                     /* I0 -> I2; I1, I2 -> I3.  */
    1365     16447099 :                     FOR_EACH_LOG_LINK (nextlinks, link)
    1366      3428973 :                       if ((next = try_combine (insn, link, link1,
    1367              :                                                nextlinks->insn,
    1368              :                                                &new_direct_jump_p,
    1369              :                                                last_combined_insn)) != 0)
    1370              :                         {
    1371         2952 :                           statistics_counter_event (cfun, "four-insn combine", 1);
    1372         2952 :                           goto retry;
    1373              :                         }
    1374              :                     /* I0 -> I1; I1, I2 -> I3.  */
    1375     16756142 :                     FOR_EACH_LOG_LINK (nextlinks, link1)
    1376      3738206 :                       if ((next = try_combine (insn, link, link1,
    1377              :                                                nextlinks->insn,
    1378              :                                                &new_direct_jump_p,
    1379              :                                                last_combined_insn)) != 0)
    1380              :                         {
    1381          190 :                           statistics_counter_event (cfun, "four-insn combine", 1);
    1382          190 :                           goto retry;
    1383              :                         }
    1384              :                   }
    1385              :               }
    1386              : 
    1387              :           /* Try this insn with each REG_EQUAL note it links back to.  */
    1388    101083930 :           FOR_EACH_LOG_LINK (links, insn)
    1389              :             {
    1390     37516561 :               rtx set, note;
    1391     37516561 :               rtx_insn *temp = links->insn;
    1392     37516561 :               if ((set = single_set (temp)) != 0
    1393     37120865 :                   && (note = find_reg_equal_equiv_note (temp)) != 0
    1394      2765221 :                   && (note = XEXP (note, 0), GET_CODE (note)) != EXPR_LIST
    1395      2765221 :                   && ! side_effects_p (SET_SRC (set))
    1396              :                   /* Avoid using a register that may already been marked
    1397              :                      dead by an earlier instruction.  */
    1398      2765221 :                   && ! unmentioned_reg_p (note, SET_SRC (set))
    1399     38888956 :                   && (GET_MODE (note) == VOIDmode
    1400        27325 :                       ? SCALAR_INT_MODE_P (GET_MODE (SET_DEST (set)))
    1401      1345070 :                       : (GET_MODE (SET_DEST (set)) == GET_MODE (note)
    1402      1345037 :                          && (GET_CODE (SET_DEST (set)) != ZERO_EXTRACT
    1403            0 :                              || (GET_MODE (XEXP (SET_DEST (set), 0))
    1404              :                                  == GET_MODE (note))))))
    1405              :                 {
    1406              :                   /* Temporarily replace the set's source with the
    1407              :                      contents of the REG_EQUAL note.  The insn will
    1408              :                      be deleted or recognized by try_combine.  */
    1409      1372345 :                   rtx orig_src = SET_SRC (set);
    1410      1372345 :                   rtx orig_dest = SET_DEST (set);
    1411      1372345 :                   if (GET_CODE (SET_DEST (set)) == ZERO_EXTRACT)
    1412            0 :                     SET_DEST (set) = XEXP (SET_DEST (set), 0);
    1413      1372345 :                   SET_SRC (set) = note;
    1414      1372345 :                   i2mod = temp;
    1415      1372345 :                   i2mod_old_rhs = copy_rtx (orig_src);
    1416      1372345 :                   i2mod_new_rhs = copy_rtx (note);
    1417      1372345 :                   next = try_combine (insn, i2mod, NULL, NULL,
    1418              :                                       &new_direct_jump_p,
    1419              :                                       last_combined_insn);
    1420      1372345 :                   i2mod = NULL;
    1421      1372345 :                   if (next)
    1422              :                     {
    1423        27228 :                       statistics_counter_event (cfun, "insn-with-note combine", 1);
    1424        27228 :                       goto retry;
    1425              :                     }
    1426      1345117 :                   INSN_CODE (temp) = -1;
    1427      1345117 :                   SET_SRC (set) = orig_src;
    1428      1345117 :                   SET_DEST (set) = orig_dest;
    1429              :                 }
    1430              :             }
    1431              : 
    1432     63567369 :           if (!NOTE_P (insn))
    1433     63567369 :             record_dead_and_set_regs (insn);
    1434              : 
    1435    136896323 : retry:
    1436    136896323 :           ;
    1437              :         }
    1438              :     }
    1439              : 
    1440      1020049 :   default_rtl_profile ();
    1441      1020049 :   clear_bb_flags ();
    1442              : 
    1443      1020049 :   if (purge_all_dead_edges ())
    1444         1278 :     new_direct_jump_p = true;
    1445      1020049 :   if (delete_noop_moves ())
    1446            0 :     new_direct_jump_p = true;
    1447              : 
    1448              :   /* Clean up.  */
    1449      1020049 :   obstack_free (&insn_link_obstack, NULL);
    1450      1020049 :   free (uid_log_links);
    1451      1020049 :   free (uid_insn_cost);
    1452      1020049 :   reg_stat.release ();
    1453              : 
    1454      1020049 :   {
    1455      1020049 :     struct undo *undo, *next;
    1456      5823181 :     for (undo = undobuf.frees; undo; undo = next)
    1457              :       {
    1458      4803132 :         next = undo->next;
    1459      4803132 :         free (undo);
    1460              :       }
    1461      1020049 :     undobuf.frees = 0;
    1462              :   }
    1463              : 
    1464      1020049 :   statistics_counter_event (cfun, "attempts", combine_attempts);
    1465      1020049 :   statistics_counter_event (cfun, "merges", combine_merges);
    1466      1020049 :   statistics_counter_event (cfun, "extras", combine_extras);
    1467      1020049 :   statistics_counter_event (cfun, "successes", combine_successes);
    1468              : 
    1469      1020049 :   nonzero_sign_valid = 0;
    1470      1020049 :   rtl_hooks = general_rtl_hooks;
    1471              : 
    1472              :   /* Make recognizer allow volatile MEMs again.  */
    1473      1020049 :   init_recog ();
    1474              : 
    1475      1020049 :   return new_direct_jump_p;
    1476              : }
    1477              : 
    1478              : /* Wipe the last_xxx fields of reg_stat in preparation for another pass.  */
    1479              : 
    1480              : static void
    1481      1020049 : init_reg_last (void)
    1482              : {
    1483      1020049 :   unsigned int i;
    1484      1020049 :   reg_stat_type *p;
    1485              : 
    1486    147699375 :   FOR_EACH_VEC_ELT (reg_stat, i, p)
    1487    146679326 :     memset (p, 0, offsetof (reg_stat_type, sign_bit_copies));
    1488      1020049 : }
    1489              : 
    1490              : /* Set up any promoted values for incoming argument registers.  */
    1491              : 
    1492              : static void
    1493      2040098 : setup_incoming_promotions (rtx_insn *first)
    1494              : {
    1495      2040098 :   tree arg;
    1496      2040098 :   bool strictly_local = false;
    1497              : 
    1498      5518640 :   for (arg = DECL_ARGUMENTS (current_function_decl); arg;
    1499      3478542 :        arg = DECL_CHAIN (arg))
    1500              :     {
    1501      3478542 :       rtx x, reg = DECL_INCOMING_RTL (arg);
    1502      3478542 :       int uns1, uns3;
    1503      3478542 :       machine_mode mode1, mode2, mode3, mode4;
    1504              : 
    1505              :       /* Only continue if the incoming argument is in a register.  */
    1506      3478542 :       if (!REG_P (reg))
    1507      3478442 :         continue;
    1508              : 
    1509              :       /* Determine, if possible, whether all call sites of the current
    1510              :          function lie within the current compilation unit.  (This does
    1511              :          take into account the exporting of a function via taking its
    1512              :          address, and so forth.)  */
    1513      2735302 :       strictly_local
    1514      2735302 :         = cgraph_node::local_info_node (current_function_decl)->local;
    1515              : 
    1516              :       /* The mode and signedness of the argument before any promotions happen
    1517              :          (equal to the mode of the pseudo holding it at that stage).  */
    1518      2735302 :       mode1 = TYPE_MODE (TREE_TYPE (arg));
    1519      2735302 :       uns1 = TYPE_UNSIGNED (TREE_TYPE (arg));
    1520              : 
    1521              :       /* The mode and signedness of the argument after any source language and
    1522              :          TARGET_PROMOTE_PROTOTYPES-driven promotions.  */
    1523      2735302 :       mode2 = TYPE_MODE (DECL_ARG_TYPE (arg));
    1524      2735302 :       uns3 = TYPE_UNSIGNED (DECL_ARG_TYPE (arg));
    1525              : 
    1526              :       /* The mode and signedness of the argument as it is actually passed,
    1527              :          see assign_parm_setup_reg in function.cc.  */
    1528      2735302 :       mode3 = promote_function_mode (TREE_TYPE (arg), mode1, &uns3,
    1529      2735302 :                                      TREE_TYPE (cfun->decl), 0);
    1530              : 
    1531              :       /* The mode of the register in which the argument is being passed.  */
    1532      2735302 :       mode4 = GET_MODE (reg);
    1533              : 
    1534              :       /* Eliminate sign extensions in the callee when:
    1535              :          (a) A mode promotion has occurred;  */
    1536      2735302 :       if (mode1 == mode3)
    1537      2735202 :         continue;
    1538              :       /* (b) The mode of the register is the same as the mode of
    1539              :              the argument as it is passed; */
    1540          100 :       if (mode3 != mode4)
    1541            0 :         continue;
    1542              :       /* (c) There's no language level extension;  */
    1543          100 :       if (mode1 == mode2)
    1544              :         ;
    1545              :       /* (c.1) All callers are from the current compilation unit.  If that's
    1546              :          the case we don't have to rely on an ABI, we only have to know
    1547              :          what we're generating right now, and we know that we will do the
    1548              :          mode1 to mode2 promotion with the given sign.  */
    1549            0 :       else if (!strictly_local)
    1550            0 :         continue;
    1551              :       /* (c.2) The combination of the two promotions is useful.  This is
    1552              :          true when the signs match, or if the first promotion is unsigned.
    1553              :          In the later case, (sign_extend (zero_extend x)) is the same as
    1554              :          (zero_extend (zero_extend x)), so make sure to force UNS3 true.  */
    1555            0 :       else if (uns1)
    1556            0 :         uns3 = true;
    1557            0 :       else if (uns3)
    1558            0 :         continue;
    1559              : 
    1560              :       /* Record that the value was promoted from mode1 to mode3,
    1561              :          so that any sign extension at the head of the current
    1562              :          function may be eliminated.  */
    1563          100 :       x = gen_rtx_CLOBBER (mode1, const0_rtx);
    1564          100 :       x = gen_rtx_fmt_e ((uns3 ? ZERO_EXTEND : SIGN_EXTEND), mode3, x);
    1565          100 :       record_value_for_reg (reg, first, x);
    1566              :     }
    1567      2040098 : }
    1568              : 
    1569              : /* If MODE has a precision lower than PREC and SRC is a non-negative constant
    1570              :    that would appear negative in MODE, sign-extend SRC for use in nonzero_bits
    1571              :    because some machines (maybe most) will actually do the sign-extension and
    1572              :    this is the conservative approach.
    1573              : 
    1574              :    ??? For 2.5, try to tighten up the MD files in this regard instead of this
    1575              :    kludge.  */
    1576              : 
    1577              : static rtx
    1578            0 : sign_extend_short_imm (rtx src, machine_mode mode, unsigned int prec)
    1579              : {
    1580            0 :   scalar_int_mode int_mode;
    1581            0 :   if (CONST_INT_P (src)
    1582            0 :       && is_a <scalar_int_mode> (mode, &int_mode)
    1583            0 :       && GET_MODE_PRECISION (int_mode) < prec
    1584            0 :       && INTVAL (src) > 0
    1585            0 :       && val_signbit_known_set_p (int_mode, INTVAL (src)))
    1586            0 :     src = GEN_INT (INTVAL (src) | ~GET_MODE_MASK (int_mode));
    1587              : 
    1588            0 :   return src;
    1589              : }
    1590              : 
    1591              : /* Update RSP for pseudo-register X from INSN's REG_EQUAL note (if one exists)
    1592              :    and SET.  */
    1593              : 
    1594              : static void
    1595     23953722 : update_rsp_from_reg_equal (reg_stat_type *rsp, rtx_insn *insn, const_rtx set,
    1596              :                            rtx x)
    1597              : {
    1598     23953722 :   rtx reg_equal_note = insn ? find_reg_equal_equiv_note (insn) : NULL_RTX;
    1599     23953722 :   unsigned HOST_WIDE_INT bits = 0;
    1600     23953722 :   rtx reg_equal = NULL, src = SET_SRC (set);
    1601     23953722 :   unsigned int num = 0;
    1602              : 
    1603     23953722 :   if (reg_equal_note)
    1604      1003034 :     reg_equal = XEXP (reg_equal_note, 0);
    1605              : 
    1606     23953722 :   if (SHORT_IMMEDIATES_SIGN_EXTEND)
    1607              :     {
    1608              :       src = sign_extend_short_imm (src, GET_MODE (x), BITS_PER_WORD);
    1609              :       if (reg_equal)
    1610              :         reg_equal = sign_extend_short_imm (reg_equal, GET_MODE (x), BITS_PER_WORD);
    1611              :     }
    1612              : 
    1613              :   /* Don't call nonzero_bits if it cannot change anything.  */
    1614     23953722 :   if (rsp->nonzero_bits != HOST_WIDE_INT_M1U)
    1615              :     {
    1616     20724162 :       machine_mode mode = GET_MODE (x);
    1617     20724162 :       if (GET_MODE_CLASS (mode) == MODE_INT
    1618     20724162 :           && HWI_COMPUTABLE_MODE_P (mode))
    1619     20724030 :         mode = nonzero_bits_mode;
    1620     20724162 :       bits = nonzero_bits (src, mode);
    1621     20724162 :       if (reg_equal && bits)
    1622       950327 :         bits &= nonzero_bits (reg_equal, mode);
    1623     20724162 :       rsp->nonzero_bits |= bits;
    1624              :     }
    1625              : 
    1626              :   /* Don't call num_sign_bit_copies if it cannot change anything.  */
    1627     23953722 :   if (rsp->sign_bit_copies != 1)
    1628              :     {
    1629     20572317 :       num = num_sign_bit_copies (SET_SRC (set), GET_MODE (x));
    1630     20572317 :       if (reg_equal && maybe_ne (num, GET_MODE_PRECISION (GET_MODE (x))))
    1631              :         {
    1632       947846 :           unsigned int numeq = num_sign_bit_copies (reg_equal, GET_MODE (x));
    1633       947846 :           if (num == 0 || numeq > num)
    1634     20572317 :             num = numeq;
    1635              :         }
    1636     20572317 :       if (rsp->sign_bit_copies == 0 || num < rsp->sign_bit_copies)
    1637     19847920 :         rsp->sign_bit_copies = num;
    1638              :     }
    1639     23953722 : }
    1640              : 
    1641              : /* Called via note_stores.  If X is a pseudo that is narrower than
    1642              :    HOST_BITS_PER_WIDE_INT and is being set, record what bits are known zero.
    1643              : 
    1644              :    If we are setting only a portion of X and we can't figure out what
    1645              :    portion, assume all bits will be used since we don't know what will
    1646              :    be happening.
    1647              : 
    1648              :    Similarly, set how many bits of X are known to be copies of the sign bit
    1649              :    at all locations in the function.  This is the smallest number implied
    1650              :    by any set of X.  */
    1651              : 
    1652              : static void
    1653     73768812 : set_nonzero_bits_and_sign_copies (rtx x, const_rtx set, void *data)
    1654              : {
    1655     73768812 :   rtx_insn *insn = (rtx_insn *) data;
    1656     73768812 :   scalar_int_mode mode;
    1657              : 
    1658     73768812 :   if (REG_P (x)
    1659     59411347 :       && REGNO (x) >= FIRST_PSEUDO_REGISTER
    1660              :       /* If this register is undefined at the start of the file, we can't
    1661              :          say what its contents were.  */
    1662     59429488 :       && ! REGNO_REG_SET_P
    1663              :            (DF_LR_IN (ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb), REGNO (x))
    1664     29612910 :       && is_a <scalar_int_mode> (GET_MODE (x), &mode)
    1665     98670072 :       && HWI_COMPUTABLE_MODE_P (mode))
    1666              :     {
    1667     24161719 :       reg_stat_type *rsp = &reg_stat[REGNO (x)];
    1668              : 
    1669     24161719 :       if (set == 0 || GET_CODE (set) == CLOBBER)
    1670              :         {
    1671        22565 :           rsp->nonzero_bits = GET_MODE_MASK (mode);
    1672        22565 :           rsp->sign_bit_copies = 1;
    1673        22565 :           return;
    1674              :         }
    1675              : 
    1676              :       /* If this register is being initialized using itself, and the
    1677              :          register is uninitialized in this basic block, and there are
    1678              :          no LOG_LINKS which set the register, then part of the
    1679              :          register is uninitialized.  In that case we can't assume
    1680              :          anything about the number of nonzero bits.
    1681              : 
    1682              :          ??? We could do better if we checked this in
    1683              :          reg_{nonzero_bits,num_sign_bit_copies}_for_combine.  Then we
    1684              :          could avoid making assumptions about the insn which initially
    1685              :          sets the register, while still using the information in other
    1686              :          insns.  We would have to be careful to check every insn
    1687              :          involved in the combination.  */
    1688              : 
    1689     24139154 :       if (insn
    1690     22692657 :           && reg_referenced_p (x, PATTERN (insn))
    1691     26791434 :           && !REGNO_REG_SET_P (DF_LR_IN (BLOCK_FOR_INSN (insn)),
    1692              :                                REGNO (x)))
    1693              :         {
    1694       267997 :           struct insn_link *link;
    1695              : 
    1696       398387 :           FOR_EACH_LOG_LINK (link, insn)
    1697       311209 :             if (dead_or_set_p (link->insn, x))
    1698              :               break;
    1699       267997 :           if (!link)
    1700              :             {
    1701        87178 :               rsp->nonzero_bits = GET_MODE_MASK (mode);
    1702        87178 :               rsp->sign_bit_copies = 1;
    1703        87178 :               return;
    1704              :             }
    1705              :         }
    1706              : 
    1707              :       /* If this is a complex assignment, see if we can convert it into a
    1708              :          simple assignment.  */
    1709     24051976 :       set = expand_field_assignment (set);
    1710              : 
    1711              :       /* If this is a simple assignment, or we have a paradoxical SUBREG,
    1712              :          set what we know about X.  */
    1713              : 
    1714     24051976 :       if (SET_DEST (set) == x
    1715     24051976 :           || (paradoxical_subreg_p (SET_DEST (set))
    1716         4731 :               && SUBREG_REG (SET_DEST (set)) == x))
    1717     23953722 :         update_rsp_from_reg_equal (rsp, insn, set, x);
    1718              :       else
    1719              :         {
    1720        98254 :           rsp->nonzero_bits = GET_MODE_MASK (mode);
    1721        98254 :           rsp->sign_bit_copies = 1;
    1722              :         }
    1723              :     }
    1724              : }
    1725              : 
    1726              : /* See if INSN can be combined into I3.  PRED, PRED2, SUCC and SUCC2 are
    1727              :    optionally insns that were previously combined into I3 or that will be
    1728              :    combined into the merger of INSN and I3.  The order is PRED, PRED2,
    1729              :    INSN, SUCC, SUCC2, I3.
    1730              : 
    1731              :    Return false if the combination is not allowed for any reason.
    1732              : 
    1733              :    If the combination is allowed, *PDEST will be set to the single
    1734              :    destination of INSN and *PSRC to the single source, and this function
    1735              :    will return true.  */
    1736              : 
    1737              : static bool
    1738     62564657 : can_combine_p (rtx_insn *insn, rtx_insn *i3, rtx_insn *pred ATTRIBUTE_UNUSED,
    1739              :                rtx_insn *pred2 ATTRIBUTE_UNUSED, rtx_insn *succ, rtx_insn *succ2,
    1740              :                rtx *pdest, rtx *psrc)
    1741              : {
    1742     62564657 :   int i;
    1743     62564657 :   const_rtx set = 0;
    1744     62564657 :   rtx src, dest;
    1745     62564657 :   rtx_insn *p;
    1746     62564657 :   rtx link;
    1747     62564657 :   bool all_adjacent = true;
    1748     62564657 :   bool (*is_volatile_p) (const_rtx);
    1749              : 
    1750     62564657 :   if (succ)
    1751              :     {
    1752     14763803 :       if (succ2)
    1753              :         {
    1754      2201353 :           if (next_active_insn (succ2) != i3)
    1755       193323 :             all_adjacent = false;
    1756      2201353 :           if (next_active_insn (succ) != succ2)
    1757      2054492 :             all_adjacent = false;
    1758              :         }
    1759     12562450 :       else if (next_active_insn (succ) != i3)
    1760      2054492 :         all_adjacent = false;
    1761     14763803 :       if (next_active_insn (insn) != succ)
    1762     17501720 :         all_adjacent = false;
    1763              :     }
    1764     47800854 :   else if (next_active_insn (insn) != i3)
    1765     17501720 :     all_adjacent = false;
    1766              : 
    1767              :   /* Can combine only if previous insn is a SET of a REG or a SUBREG,
    1768              :      or a PARALLEL consisting of such a SET and CLOBBERs.
    1769              : 
    1770              :      If INSN has CLOBBER parallel parts, ignore them for our processing.
    1771              :      By definition, these happen during the execution of the insn.  When it
    1772              :      is merged with another insn, all bets are off.  If they are, in fact,
    1773              :      needed and aren't also supplied in I3, they may be added by
    1774              :      recog_for_combine.  Otherwise, it won't match.
    1775              : 
    1776              :      We can also ignore a SET whose SET_DEST is mentioned in a REG_UNUSED
    1777              :      note.
    1778              : 
    1779              :      Get the source and destination of INSN.  If more than one, can't
    1780              :      combine.  */
    1781              : 
    1782     62564657 :   if (GET_CODE (PATTERN (insn)) == SET)
    1783              :     set = PATTERN (insn);
    1784     16555964 :   else if (GET_CODE (PATTERN (insn)) == PARALLEL
    1785     16555964 :            && GET_CODE (XVECEXP (PATTERN (insn), 0, 0)) == SET)
    1786              :     {
    1787     49613165 :       for (i = 0; i < XVECLEN (PATTERN (insn), 0); i++)
    1788              :         {
    1789     33582776 :           rtx elt = XVECEXP (PATTERN (insn), 0, i);
    1790              : 
    1791     33582776 :           switch (GET_CODE (elt))
    1792              :             {
    1793              :             /* This is important to combine floating point insns
    1794              :                for the SH4 port.  */
    1795       137556 :             case USE:
    1796              :               /* Combining an isolated USE doesn't make sense.
    1797              :                  We depend here on combinable_i3pat to reject them.  */
    1798              :               /* The code below this loop only verifies that the inputs of
    1799              :                  the SET in INSN do not change.  We call reg_set_between_p
    1800              :                  to verify that the REG in the USE does not change between
    1801              :                  I3 and INSN.
    1802              :                  If the USE in INSN was for a pseudo register, the matching
    1803              :                  insn pattern will likely match any register; combining this
    1804              :                  with any other USE would only be safe if we knew that the
    1805              :                  used registers have identical values, or if there was
    1806              :                  something to tell them apart, e.g. different modes.  For
    1807              :                  now, we forgo such complicated tests and simply disallow
    1808              :                  combining of USES of pseudo registers with any other USE.  */
    1809       137556 :               if (REG_P (XEXP (elt, 0))
    1810       137556 :                   && GET_CODE (PATTERN (i3)) == PARALLEL)
    1811              :                 {
    1812          235 :                   rtx i3pat = PATTERN (i3);
    1813          235 :                   int i = XVECLEN (i3pat, 0) - 1;
    1814          235 :                   unsigned int regno = REGNO (XEXP (elt, 0));
    1815              : 
    1816          481 :                   do
    1817              :                     {
    1818          481 :                       rtx i3elt = XVECEXP (i3pat, 0, i);
    1819              : 
    1820          481 :                       if (GET_CODE (i3elt) == USE
    1821          225 :                           && REG_P (XEXP (i3elt, 0))
    1822          733 :                           && (REGNO (XEXP (i3elt, 0)) == regno
    1823          198 :                               ? reg_set_between_p (XEXP (elt, 0),
    1824           27 :                                                    PREV_INSN (insn), i3)
    1825              :                               : regno >= FIRST_PSEUDO_REGISTER))
    1826              :                         return false;
    1827              :                     }
    1828          283 :                   while (--i >= 0);
    1829              :                 }
    1830              :               break;
    1831              : 
    1832              :               /* We can ignore CLOBBERs.  */
    1833              :             case CLOBBER:
    1834              :               break;
    1835              : 
    1836     17153781 :             case SET:
    1837              :               /* Ignore SETs whose result isn't used but not those that
    1838              :                  have side-effects.  */
    1839     17153781 :               if (find_reg_note (insn, REG_UNUSED, SET_DEST (elt))
    1840       196296 :                   && insn_nothrow_p (insn)
    1841     17337038 :                   && !side_effects_p (elt))
    1842              :                 break;
    1843              : 
    1844              :               /* If we have already found a SET, this is a second one and
    1845              :                  so we cannot combine with this insn.  */
    1846     17055519 :               if (set)
    1847              :                 return false;
    1848              : 
    1849              :               set = elt;
    1850              :               break;
    1851              : 
    1852              :             default:
    1853              :               /* Anything else means we can't combine.  */
    1854              :               return false;
    1855              :             }
    1856              :         }
    1857              : 
    1858     16030389 :       if (set == 0
    1859              :           /* If SET_SRC is an ASM_OPERANDS we can't throw away these CLOBBERs,
    1860              :              so don't do anything with it.  */
    1861     16030389 :           || GET_CODE (SET_SRC (set)) == ASM_OPERANDS)
    1862              :         return false;
    1863              :     }
    1864              :   else
    1865              :     return false;
    1866              : 
    1867              :   if (set == 0)
    1868              :     return false;
    1869              : 
    1870              :   /* The simplification in expand_field_assignment may call back to
    1871              :      get_last_value, so set safe guard here.  */
    1872     62018624 :   subst_low_luid = DF_INSN_LUID (insn);
    1873              : 
    1874     62018624 :   set = expand_field_assignment (set);
    1875     62018624 :   src = SET_SRC (set), dest = SET_DEST (set);
    1876              : 
    1877              :   /* Do not eliminate user-specified register if it is in an
    1878              :      asm input because we may break the register asm usage defined
    1879              :      in GCC manual if allow to do so.
    1880              :      Be aware that this may cover more cases than we expect but this
    1881              :      should be harmless.  */
    1882     61456071 :   if (REG_P (dest) && REG_USERVAR_P (dest) && HARD_REGISTER_P (dest)
    1883     62018627 :       && extract_asm_operands (PATTERN (i3)))
    1884              :     return false;
    1885              : 
    1886              :   /* Don't eliminate a store in the stack pointer.  */
    1887     62018624 :   if (dest == stack_pointer_rtx
    1888              :       /* Don't combine with an insn that sets a register to itself if it has
    1889              :          a REG_EQUAL note.  This may be part of a LIBCALL sequence.  */
    1890     60117375 :       || (rtx_equal_p (src, dest) && find_reg_note (insn, REG_EQUAL, NULL_RTX))
    1891              :       /* Can't merge an ASM_OPERANDS.  */
    1892     60117375 :       || GET_CODE (src) == ASM_OPERANDS
    1893              :       /* Can't merge a function call.  */
    1894     60113751 :       || GET_CODE (src) == CALL
    1895              :       /* Don't eliminate a function call argument.  */
    1896     60113751 :       || (CALL_P (i3)
    1897      9019010 :           && (find_reg_fusage (i3, USE, dest)
    1898       169074 :               || (REG_P (dest)
    1899       169074 :                   && REGNO (dest) < FIRST_PSEUDO_REGISTER
    1900          283 :                   && global_regs[REGNO (dest)])))
    1901              :       /* Don't substitute into an incremented register.  */
    1902              :       || FIND_REG_INC_NOTE (i3, dest)
    1903              :       || (succ && FIND_REG_INC_NOTE (succ, dest))
    1904     60113751 :       || (succ2 && FIND_REG_INC_NOTE (succ2, dest))
    1905              :       /* Don't substitute into a non-local goto, this confuses CFG.  */
    1906     51263812 :       || (JUMP_P (i3) && find_reg_note (i3, REG_NON_LOCAL_GOTO, NULL_RTX))
    1907              :       /* Make sure that DEST is not used after INSN but before SUCC, or
    1908              :          after SUCC and before SUCC2, or after SUCC2 but before I3.  */
    1909     51263080 :       || (!all_adjacent
    1910     12622668 :           && ((succ2
    1911       961295 :                && (reg_used_between_p (dest, succ2, i3)
    1912       938417 :                    || reg_used_between_p (dest, succ, succ2)))
    1913     12553934 :               || (!succ2 && succ && reg_used_between_p (dest, succ, i3))
    1914     12273465 :               || (!succ2 && !succ && reg_used_between_p (dest, insn, i3))
    1915     12273465 :               || (succ
    1916              :                   /* SUCC and SUCC2 can be split halves from a PARALLEL; in
    1917              :                      that case SUCC is not in the insn stream, so use SUCC2
    1918              :                      instead for this test.  */
    1919     10110320 :                   && reg_used_between_p (dest, insn,
    1920              :                                          succ2
    1921       892561 :                                          && INSN_UID (succ) == INSN_UID (succ2)
    1922              :                                          ? succ2 : succ))))
    1923              :       /* Make sure that the value that is to be substituted for the register
    1924              :          does not use any registers whose values alter in between.  However,
    1925              :          If the insns are adjacent, a use can't cross a set even though we
    1926              :          think it might (this can happen for a sequence of insns each setting
    1927              :          the same destination; last_set of that register might point to
    1928              :          a NOTE).  If INSN has a REG_EQUIV note, the register is always
    1929              :          equivalent to the memory so the substitution is valid even if there
    1930              :          are intervening stores.  Also, don't move a volatile asm or
    1931              :          UNSPEC_VOLATILE across any other insns.  */
    1932              :       || (! all_adjacent
    1933     12273465 :           && (((!MEM_P (src)
    1934      3499086 :                 || ! find_reg_note (insn, REG_EQUIV, src))
    1935     12158052 :                && modified_between_p (src, insn, i3))
    1936     11116147 :               || (GET_CODE (src) == ASM_OPERANDS && MEM_VOLATILE_P (src))
    1937     11116147 :               || GET_CODE (src) == UNSPEC_VOLATILE))
    1938              :       /* Don't combine across a CALL_INSN, because that would possibly
    1939              :          change whether the life span of some REGs crosses calls or not,
    1940              :          and it is a pain to update that information.
    1941              :          Exception: if source is a constant, moving it later can't hurt.
    1942              :          Accept that as a special case.  */
    1943    111764719 :       || (DF_INSN_LUID (insn) < last_call_luid && ! CONSTANT_P (src)))
    1944              :     return false;
    1945              : 
    1946              :   /* DEST must be a REG.  */
    1947     49447003 :   if (REG_P (dest))
    1948              :     {
    1949              :       /* If register alignment is being enforced for multi-word items in all
    1950              :          cases except for parameters, it is possible to have a register copy
    1951              :          insn referencing a hard register that is not allowed to contain the
    1952              :          mode being copied and which would not be valid as an operand of most
    1953              :          insns.  Eliminate this problem by not combining with such an insn.
    1954              : 
    1955              :          Also, on some machines we don't want to extend the life of a hard
    1956              :          register.  */
    1957              : 
    1958     48889346 :       if (REG_P (src)
    1959     48889346 :           && ((REGNO (dest) < FIRST_PSEUDO_REGISTER
    1960        29834 :                && !targetm.hard_regno_mode_ok (REGNO (dest), GET_MODE (dest)))
    1961              :               /* Don't extend the life of a hard register unless it is
    1962              :                  user variable (if we have few registers) or it can't
    1963              :                  fit into the desired register (meaning something special
    1964              :                  is going on).
    1965              :                  Also avoid substituting a return register into I3, because
    1966              :                  reload can't handle a conflict with constraints of other
    1967              :                  inputs.  */
    1968      2591685 :               || (REGNO (src) < FIRST_PSEUDO_REGISTER
    1969        37784 :                   && !targetm.hard_regno_mode_ok (REGNO (src),
    1970        37784 :                                                   GET_MODE (src)))))
    1971              :         return false;
    1972              :     }
    1973              :   else
    1974              :     return false;
    1975              : 
    1976              : 
    1977     48889346 :   if (GET_CODE (PATTERN (i3)) == PARALLEL)
    1978     37112170 :     for (i = XVECLEN (PATTERN (i3), 0) - 1; i >= 0; i--)
    1979     25021926 :       if (GET_CODE (XVECEXP (PATTERN (i3), 0, i)) == CLOBBER)
    1980              :         {
    1981     11756804 :           rtx reg = XEXP (XVECEXP (PATTERN (i3), 0, i), 0);
    1982              : 
    1983              :           /* If the clobber represents an earlyclobber operand, we must not
    1984              :              substitute an expression containing the clobbered register.
    1985              :              As we do not analyze the constraint strings here, we have to
    1986              :              make the conservative assumption.  However, if the register is
    1987              :              a fixed hard reg, the clobber cannot represent any operand;
    1988              :              we leave it up to the machine description to either accept or
    1989              :              reject use-and-clobber patterns.  */
    1990     11756804 :           if (!REG_P (reg)
    1991     11398818 :               || REGNO (reg) >= FIRST_PSEUDO_REGISTER
    1992     23106690 :               || !fixed_regs[REGNO (reg)])
    1993       445026 :             if (reg_overlap_mentioned_p (reg, src))
    1994              :               return false;
    1995              :         }
    1996              : 
    1997              :   /* If INSN contains anything volatile, or is an `asm' (whether volatile
    1998              :      or not), reject, unless nothing volatile comes between it and I3 */
    1999              : 
    2000     48888274 :   if (GET_CODE (src) == ASM_OPERANDS || volatile_refs_p (src))
    2001              :     {
    2002              :       /* Make sure neither succ nor succ2 contains a volatile reference.  */
    2003       724919 :       if (succ2 != 0 && volatile_refs_p (PATTERN (succ2)))
    2004              :         return false;
    2005       724825 :       if (succ != 0 && volatile_refs_p (PATTERN (succ)))
    2006              :         return false;
    2007              :       /* We'll check insns between INSN and I3 below.  */
    2008              :     }
    2009              : 
    2010              :   /* If INSN is an asm, and DEST is a hard register, reject, since it has
    2011              :      to be an explicit register variable, and was chosen for a reason.  */
    2012              : 
    2013     48841430 :   if (GET_CODE (src) == ASM_OPERANDS
    2014     48841430 :       && REG_P (dest) && REGNO (dest) < FIRST_PSEUDO_REGISTER)
    2015              :     return false;
    2016              : 
    2017              :   /* If INSN contains volatile references (specifically volatile MEMs),
    2018              :      we cannot combine across any other volatile references.
    2019              :      Even if INSN doesn't contain volatile references, any intervening
    2020              :      volatile insn might affect machine state.  */
    2021              : 
    2022     97003917 :   is_volatile_p = volatile_refs_p (PATTERN (insn))
    2023     48841430 :     ? volatile_refs_p
    2024              :     : volatile_insn_p;
    2025              : 
    2026    226783556 :   for (p = NEXT_INSN (insn); p != i3; p = NEXT_INSN (p))
    2027    129307721 :     if (NONDEBUG_INSN_P (p)
    2028     64414371 :         && p != succ
    2029     64414371 :         && p != succ2
    2030    178782822 :         && is_volatile_p (PATTERN (p)))
    2031              :       return false;
    2032              : 
    2033              :   /* If INSN contains an autoincrement or autodecrement, make sure that
    2034              :      register is not used between there and I3, and not already used in
    2035              :      I3 either.  Neither must it be used in PRED or SUCC, if they exist.
    2036              :      Also insist that I3 not be a jump if using LRA; if it were one
    2037              :      and the incremented register were spilled, we would lose.
    2038              :      Reload handles this correctly.  */
    2039              : 
    2040     48634405 :   if (AUTO_INC_DEC)
    2041              :     for (link = REG_NOTES (insn); link; link = XEXP (link, 1))
    2042              :       if (REG_NOTE_KIND (link) == REG_INC
    2043              :           && ((JUMP_P (i3) && targetm.lra_p ())
    2044              :               || reg_used_between_p (XEXP (link, 0), insn, i3)
    2045              :               || (pred != NULL_RTX
    2046              :                   && reg_overlap_mentioned_p (XEXP (link, 0), PATTERN (pred)))
    2047              :               || (pred2 != NULL_RTX
    2048              :                   && reg_overlap_mentioned_p (XEXP (link, 0), PATTERN (pred2)))
    2049              :               || (succ != NULL_RTX
    2050              :                   && reg_overlap_mentioned_p (XEXP (link, 0), PATTERN (succ)))
    2051              :               || (succ2 != NULL_RTX
    2052              :                   && reg_overlap_mentioned_p (XEXP (link, 0), PATTERN (succ2)))
    2053              :               || reg_overlap_mentioned_p (XEXP (link, 0), PATTERN (i3))))
    2054              :         return false;
    2055              : 
    2056              :   /* If we get here, we have passed all the tests and the combination is
    2057              :      to be allowed.  */
    2058              : 
    2059     48634405 :   *pdest = dest;
    2060     48634405 :   *psrc = src;
    2061              : 
    2062     48634405 :   return true;
    2063              : }
    2064              : 
    2065              : /* LOC is the location within I3 that contains its pattern or the component
    2066              :    of a PARALLEL of the pattern.  We validate that it is valid for combining.
    2067              : 
    2068              :    One problem is if I3 modifies its output, as opposed to replacing it
    2069              :    entirely, we can't allow the output to contain I2DEST, I1DEST or I0DEST as
    2070              :    doing so would produce an insn that is not equivalent to the original insns.
    2071              : 
    2072              :    Consider:
    2073              : 
    2074              :          (set (reg:DI 101) (reg:DI 100))
    2075              :          (set (subreg:SI (reg:DI 101) 0) <foo>)
    2076              : 
    2077              :    This is NOT equivalent to:
    2078              : 
    2079              :          (parallel [(set (subreg:SI (reg:DI 100) 0) <foo>)
    2080              :                     (set (reg:DI 101) (reg:DI 100))])
    2081              : 
    2082              :    Not only does this modify 100 (in which case it might still be valid
    2083              :    if 100 were dead in I2), it sets 101 to the ORIGINAL value of 100.
    2084              : 
    2085              :    We can also run into a problem if I2 sets a register that I1
    2086              :    uses and I1 gets directly substituted into I3 (not via I2).  In that
    2087              :    case, we would be getting the wrong value of I2DEST into I3, so we
    2088              :    must reject the combination.  This case occurs when I2 and I1 both
    2089              :    feed into I3, rather than when I1 feeds into I2, which feeds into I3.
    2090              :    If I1_NOT_IN_SRC is nonzero, it means that finding I1 in the source
    2091              :    of a SET must prevent combination from occurring.  The same situation
    2092              :    can occur for I0, in which case I0_NOT_IN_SRC is set.
    2093              : 
    2094              :    Before doing the above check, we first try to expand a field assignment
    2095              :    into a set of logical operations.
    2096              : 
    2097              :    If PI3_DEST_KILLED is nonzero, it is a pointer to a location in which
    2098              :    we place a register that is both set and used within I3.  If more than one
    2099              :    such register is detected, we fail.
    2100              : 
    2101              :    Return true if the combination is valid, false otherwise.  */
    2102              : 
    2103              : static bool
    2104     70125367 : combinable_i3pat (rtx_insn *i3, rtx *loc, rtx i2dest, rtx i1dest, rtx i0dest,
    2105              :                   bool i1_not_in_src, bool i0_not_in_src, rtx *pi3dest_killed)
    2106              : {
    2107     70125367 :   rtx x = *loc;
    2108              : 
    2109     70125367 :   if (GET_CODE (x) == SET)
    2110              :     {
    2111     47135354 :       rtx set = x ;
    2112     47135354 :       rtx dest = SET_DEST (set);
    2113     47135354 :       rtx src = SET_SRC (set);
    2114     47135354 :       rtx inner_dest = dest;
    2115     47135354 :       rtx subdest;
    2116              : 
    2117     47135354 :       while (GET_CODE (inner_dest) == STRICT_LOW_PART
    2118     47651660 :              || GET_CODE (inner_dest) == SUBREG
    2119     47651660 :              || GET_CODE (inner_dest) == ZERO_EXTRACT)
    2120       516306 :         inner_dest = XEXP (inner_dest, 0);
    2121              : 
    2122              :       /* Check for the case where I3 modifies its output, as discussed
    2123              :          above.  We don't want to prevent pseudos from being combined
    2124              :          into the address of a MEM, so only prevent the combination if
    2125              :          i1 or i2 set the same MEM.  */
    2126       495513 :       if ((inner_dest != dest &&
    2127              :            (!MEM_P (inner_dest)
    2128          787 :             || rtx_equal_p (i2dest, inner_dest)
    2129          787 :             || (i1dest && rtx_equal_p (i1dest, inner_dest))
    2130          787 :             || (i0dest && rtx_equal_p (i0dest, inner_dest)))
    2131       494726 :            && (reg_overlap_mentioned_p (i2dest, inner_dest)
    2132       364659 :                || (i1dest && reg_overlap_mentioned_p (i1dest, inner_dest))
    2133       363362 :                || (i0dest && reg_overlap_mentioned_p (i0dest, inner_dest))))
    2134              : 
    2135              :           /* This is the same test done in can_combine_p except we can't test
    2136              :              all_adjacent; we don't have to, since this instruction will stay
    2137              :              in place, thus we are not considering increasing the lifetime of
    2138              :              INNER_DEST.
    2139              : 
    2140              :              Also, if this insn sets a function argument, combining it with
    2141              :              something that might need a spill could clobber a previous
    2142              :              function argument; the all_adjacent test in can_combine_p also
    2143              :              checks this; here, we do a more specific test for this case.  */
    2144              : 
    2145     47003896 :           || (REG_P (inner_dest)
    2146     30382167 :               && REGNO (inner_dest) < FIRST_PSEUDO_REGISTER
    2147      7461875 :               && !targetm.hard_regno_mode_ok (REGNO (inner_dest),
    2148      7461875 :                                               GET_MODE (inner_dest)))
    2149     47003642 :           || (i1_not_in_src && reg_overlap_mentioned_p (i1dest, src))
    2150     94131852 :           || (i0_not_in_src && reg_overlap_mentioned_p (i0dest, src)))
    2151              :         return false;
    2152              : 
    2153              :       /* If DEST is used in I3, it is being killed in this insn, so
    2154              :          record that for later.  We have to consider paradoxical
    2155              :          subregs here, since they kill the whole register, but we
    2156              :          ignore partial subregs, STRICT_LOW_PART, etc.
    2157              :          Never add REG_DEAD notes for the FRAME_POINTER_REGNUM or the
    2158              :          STACK_POINTER_REGNUM, since these are always considered to be
    2159              :          live.  Similarly for ARG_POINTER_REGNUM if it is fixed.  */
    2160     46965875 :       subdest = dest;
    2161     46965875 :       if (GET_CODE (subdest) == SUBREG && !partial_subreg_p (subdest))
    2162       253551 :         subdest = SUBREG_REG (subdest);
    2163     46965875 :       if (pi3dest_killed
    2164     34084266 :           && REG_P (subdest)
    2165     21732577 :           && reg_referenced_p (subdest, PATTERN (i3))
    2166      1260444 :           && REGNO (subdest) != FRAME_POINTER_REGNUM
    2167      1260444 :           && (HARD_FRAME_POINTER_IS_FRAME_POINTER
    2168      1260444 :               || REGNO (subdest) != HARD_FRAME_POINTER_REGNUM)
    2169      1260444 :           && (FRAME_POINTER_REGNUM == ARG_POINTER_REGNUM
    2170      1260444 :               || (REGNO (subdest) != ARG_POINTER_REGNUM
    2171            0 :                   || ! fixed_regs [REGNO (subdest)]))
    2172     48226319 :           && REGNO (subdest) != STACK_POINTER_REGNUM)
    2173              :         {
    2174      1222785 :           if (*pi3dest_killed)
    2175              :             return false;
    2176              : 
    2177      1198630 :           *pi3dest_killed = subdest;
    2178              :         }
    2179              :     }
    2180              : 
    2181     22990013 :   else if (GET_CODE (x) == PARALLEL)
    2182              :     {
    2183              :       int i;
    2184              : 
    2185     34989368 :       for (i = 0; i < XVECLEN (x, 0); i++)
    2186     23591587 :         if (! combinable_i3pat (i3, &XVECEXP (x, 0, i), i2dest, i1dest, i0dest,
    2187              :                                 i1_not_in_src, i0_not_in_src, pi3dest_killed))
    2188              :           return false;
    2189              :     }
    2190              : 
    2191              :   return true;
    2192              : }
    2193              : 
    2194              : /* Return true if X is an arithmetic expression that contains a multiplication
    2195              :    and division.  We don't count multiplications by powers of two here.  */
    2196              : 
    2197              : static bool
    2198     17538418 : contains_muldiv (rtx x)
    2199              : {
    2200     18215936 :   switch (GET_CODE (x))
    2201              :     {
    2202              :     case MOD:  case DIV:  case UMOD:  case UDIV:
    2203              :       return true;
    2204              : 
    2205       540198 :     case MULT:
    2206       540198 :       return ! (CONST_INT_P (XEXP (x, 1))
    2207       126166 :                 && pow2p_hwi (UINTVAL (XEXP (x, 1))));
    2208     17515817 :     default:
    2209     17515817 :       if (BINARY_P (x))
    2210      6012745 :         return contains_muldiv (XEXP (x, 0))
    2211      6012745 :             || contains_muldiv (XEXP (x, 1));
    2212              : 
    2213     11503072 :       if (UNARY_P (x))
    2214       677518 :         return contains_muldiv (XEXP (x, 0));
    2215              : 
    2216              :       return false;
    2217              :     }
    2218              : }
    2219              : 
    2220              : /* Determine whether INSN can be used in a combination.  Return true if
    2221              :    not.  This is used in try_combine to detect early some cases where we
    2222              :    can't perform combinations.  */
    2223              : 
    2224              : static bool
    2225    169857920 : cant_combine_insn_p (rtx_insn *insn)
    2226              : {
    2227    169857920 :   rtx set;
    2228    169857920 :   rtx src, dest;
    2229              : 
    2230              :   /* If this isn't really an insn, we can't do anything.
    2231              :      This can occur when flow deletes an insn that it has merged into an
    2232              :      auto-increment address.  */
    2233    169857920 :   if (!NONDEBUG_INSN_P (insn))
    2234              :     return true;
    2235              : 
    2236              :   /* Never combine loads and stores involving hard regs that are likely
    2237              :      to be spilled.  The register allocator can usually handle such
    2238              :      reg-reg moves by tying.  If we allow the combiner to make
    2239              :      substitutions of likely-spilled regs, reload might die.
    2240              :      As an exception, we allow combinations involving fixed regs; these are
    2241              :      not available to the register allocator so there's no risk involved.  */
    2242              : 
    2243    169857563 :   set = single_set (insn);
    2244    169857563 :   if (! set)
    2245              :     return false;
    2246    156299838 :   src = SET_SRC (set);
    2247    156299838 :   dest = SET_DEST (set);
    2248    156299838 :   if (GET_CODE (src) == SUBREG)
    2249      1152053 :     src = SUBREG_REG (src);
    2250    156299838 :   if (GET_CODE (dest) == SUBREG)
    2251      1736134 :     dest = SUBREG_REG (dest);
    2252     42288314 :   if (REG_P (src) && REG_P (dest)
    2253    191755349 :       && ((HARD_REGISTER_P (src)
    2254      6850573 :            && ! TEST_HARD_REG_BIT (fixed_reg_set, REGNO (src))
    2255              : #ifdef LEAF_REGISTERS
    2256              :            && ! LEAF_REGISTERS [REGNO (src)])
    2257              : #else
    2258              :            )
    2259              : #endif
    2260     28945314 :           || (HARD_REGISTER_P (dest)
    2261     20626102 :               && ! TEST_HARD_REG_BIT (fixed_reg_set, REGNO (dest))
    2262     20322383 :               && targetm.class_likely_spilled_p (REGNO_REG_CLASS (REGNO (dest))))))
    2263     25238508 :     return true;
    2264              : 
    2265              :   return false;
    2266              : }
    2267              : 
    2268              : struct likely_spilled_retval_info
    2269              : {
    2270              :   unsigned regno, nregs;
    2271              :   unsigned mask;
    2272              : };
    2273              : 
    2274              : /* Called via note_stores by likely_spilled_retval_p.  Remove from info->mask
    2275              :    hard registers that are known to be written to / clobbered in full.  */
    2276              : static void
    2277       180149 : likely_spilled_retval_1 (rtx x, const_rtx set, void *data)
    2278              : {
    2279       180149 :   struct likely_spilled_retval_info *const info =
    2280              :     (struct likely_spilled_retval_info *) data;
    2281       180149 :   unsigned regno, nregs;
    2282       180149 :   unsigned new_mask;
    2283              : 
    2284       180149 :   if (!REG_P (XEXP (set, 0)))
    2285              :     return;
    2286       180149 :   regno = REGNO (x);
    2287       180149 :   if (regno >= info->regno + info->nregs)
    2288              :     return;
    2289       180149 :   nregs = REG_NREGS (x);
    2290       180149 :   if (regno + nregs <= info->regno)
    2291              :     return;
    2292       180149 :   new_mask = (2U << (nregs - 1)) - 1;
    2293       180149 :   if (regno < info->regno)
    2294            0 :     new_mask >>= info->regno - regno;
    2295              :   else
    2296       180149 :     new_mask <<= regno - info->regno;
    2297       180149 :   info->mask &= ~new_mask;
    2298              : }
    2299              : 
    2300              : /* Return true iff part of the return value is live during INSN, and
    2301              :    it is likely spilled.  This can happen when more than one insn is needed
    2302              :    to copy the return value, e.g. when we consider to combine into the
    2303              :    second copy insn for a complex value.  */
    2304              : 
    2305              : static bool
    2306     48156827 : likely_spilled_retval_p (rtx_insn *insn)
    2307              : {
    2308     48156827 :   rtx_insn *use = BB_END (this_basic_block);
    2309     48156827 :   rtx reg;
    2310     48156827 :   rtx_insn *p;
    2311     48156827 :   unsigned regno, nregs;
    2312              :   /* We assume here that no machine mode needs more than
    2313              :      32 hard registers when the value overlaps with a register
    2314              :      for which TARGET_FUNCTION_VALUE_REGNO_P is true.  */
    2315     48156827 :   unsigned mask;
    2316     48156827 :   struct likely_spilled_retval_info info;
    2317              : 
    2318              :   /* With -g, BB_END may be a DEBUG_INSN.  Ignore trailing debug
    2319              :      instructions without changing how notes are handled.  */
    2320     48156827 :   if (DEBUG_INSN_P (use))
    2321      1113720 :     use = prev_nondebug_insn (use);
    2322     48156827 :   if (!NONJUMP_INSN_P (use)
    2323     12593350 :       || GET_CODE (PATTERN (use)) != USE
    2324     51555985 :       || insn == use)
    2325              :     return false;
    2326      3276790 :   reg = XEXP (PATTERN (use), 0);
    2327      3276790 :   if (!REG_P (reg) || !targetm.calls.function_value_regno_p (REGNO (reg)))
    2328              :     return false;
    2329      3276790 :   regno = REGNO (reg);
    2330      3276790 :   nregs = REG_NREGS (reg);
    2331      3276790 :   if (nregs == 1)
    2332              :     return false;
    2333       177540 :   mask = (2U << (nregs - 1)) - 1;
    2334              : 
    2335              :   /* Disregard parts of the return value that are set later.  */
    2336       177540 :   info.regno = regno;
    2337       177540 :   info.nregs = nregs;
    2338       177540 :   info.mask = mask;
    2339       599155 :   for (p = PREV_INSN (use); info.mask && p != insn; p = PREV_INSN (p))
    2340       244075 :     if (INSN_P (p))
    2341       244075 :       note_stores (p, likely_spilled_retval_1, &info);
    2342       355036 :   mask = info.mask;
    2343              : 
    2344              :   /* Check if any of the (probably) live return value registers is
    2345              :      likely spilled.  */
    2346              :   nregs --;
    2347       355036 :   do
    2348              :     {
    2349       355036 :       if ((mask & 1 << nregs)
    2350       355036 :           && targetm.class_likely_spilled_p (REGNO_REG_CLASS (regno + nregs)))
    2351              :         return true;
    2352       354982 :     } while (nregs--);
    2353              :   return false;
    2354              : }
    2355              : 
    2356              : /* Adjust INSN after we made a change to its destination.
    2357              : 
    2358              :    Changing the destination can invalidate notes that say something about
    2359              :    the results of the insn and a LOG_LINK pointing to the insn.  */
    2360              : 
    2361              : static void
    2362        17339 : adjust_for_new_dest (rtx_insn *insn)
    2363              : {
    2364              :   /* For notes, be conservative and simply remove them.  */
    2365        17339 :   remove_reg_equal_equiv_notes (insn, true);
    2366              : 
    2367              :   /* The new insn will have a destination that was previously the destination
    2368              :      of an insn just above it.  Call distribute_links to make a LOG_LINK from
    2369              :      the next use of that destination.  */
    2370              : 
    2371        17339 :   rtx set = single_set (insn);
    2372        17339 :   gcc_assert (set);
    2373              : 
    2374        17339 :   rtx reg = SET_DEST (set);
    2375              : 
    2376        17339 :   while (GET_CODE (reg) == ZERO_EXTRACT
    2377        17339 :          || GET_CODE (reg) == STRICT_LOW_PART
    2378        34678 :          || GET_CODE (reg) == SUBREG)
    2379            0 :     reg = XEXP (reg, 0);
    2380        17339 :   gcc_assert (REG_P (reg));
    2381              : 
    2382        17339 :   distribute_links (alloc_insn_link (insn, REGNO (reg), NULL));
    2383              : 
    2384        17339 :   df_insn_rescan (insn);
    2385        17339 : }
    2386              : 
    2387              : /* Return TRUE if combine can reuse reg X in mode MODE.
    2388              :    ADDED_SETS is trueif the original set is still required.  */
    2389              : static bool
    2390      2731084 : can_change_dest_mode (rtx x, bool added_sets, machine_mode mode)
    2391              : {
    2392      2731084 :   unsigned int regno;
    2393              : 
    2394      2731084 :   if (!REG_P (x))
    2395              :     return false;
    2396              : 
    2397              :   /* Don't change between modes with different underlying register sizes,
    2398              :      since this could lead to invalid subregs.  */
    2399      2731084 :   if (maybe_ne (REGMODE_NATURAL_SIZE (mode),
    2400      2731084 :                 REGMODE_NATURAL_SIZE (GET_MODE (x))))
    2401              :     return false;
    2402              : 
    2403      2731084 :   regno = REGNO (x);
    2404              :   /* Allow hard registers if the new mode is legal, and occupies no more
    2405              :      registers than the old mode.  */
    2406      2731084 :   if (regno < FIRST_PSEUDO_REGISTER)
    2407      1221691 :     return (targetm.hard_regno_mode_ok (regno, mode)
    2408      1221691 :             && REG_NREGS (x) >= hard_regno_nregs (regno, mode));
    2409              : 
    2410              :   /* Or a pseudo that is only used once.  */
    2411      1509393 :   return (regno < reg_n_sets_max
    2412      1509369 :           && REG_N_SETS (regno) == 1
    2413      1457963 :           && !added_sets
    2414      2967356 :           && !REG_USERVAR_P (x));
    2415              : }
    2416              : 
    2417              : 
    2418              : /* Check whether X, the destination of a set, refers to part of
    2419              :    the register specified by REG.  */
    2420              : 
    2421              : static bool
    2422        17594 : reg_subword_p (rtx x, rtx reg)
    2423              : {
    2424              :   /* Check that reg is an integer mode register.  */
    2425        17594 :   if (!REG_P (reg) || GET_MODE_CLASS (GET_MODE (reg)) != MODE_INT)
    2426              :     return false;
    2427              : 
    2428        17036 :   if (GET_CODE (x) == STRICT_LOW_PART
    2429        16605 :       || GET_CODE (x) == ZERO_EXTRACT)
    2430          455 :     x = XEXP (x, 0);
    2431              : 
    2432        17036 :   return GET_CODE (x) == SUBREG
    2433        16842 :          && !paradoxical_subreg_p (x)
    2434        16842 :          && SUBREG_REG (x) == reg
    2435        33878 :          && GET_MODE_CLASS (GET_MODE (x)) == MODE_INT;
    2436              : }
    2437              : 
    2438              : /* Return whether PAT is a PARALLEL of exactly N register SETs followed
    2439              :    by an arbitrary number of CLOBBERs.  */
    2440              : static bool
    2441    102942091 : is_parallel_of_n_reg_sets (rtx pat, int n)
    2442              : {
    2443    102942091 :   if (GET_CODE (pat) != PARALLEL)
    2444              :     return false;
    2445              : 
    2446     27642453 :   int len = XVECLEN (pat, 0);
    2447     27642453 :   if (len < n)
    2448              :     return false;
    2449              : 
    2450              :   int i;
    2451     55036804 :   for (i = 0; i < n; i++)
    2452     52186144 :     if (GET_CODE (XVECEXP (pat, 0, i)) != SET
    2453     30802927 :         || !REG_P (SET_DEST (XVECEXP (pat, 0, i))))
    2454              :       return false;
    2455      3242383 :   for ( ; i < len; i++)
    2456      1021246 :     switch (GET_CODE (XVECEXP (pat, 0, i)))
    2457              :       {
    2458       391724 :       case CLOBBER:
    2459       391724 :         if (XEXP (XVECEXP (pat, 0, i), 0) == const0_rtx)
    2460              :           return false;
    2461       391723 :         break;
    2462              :       default:
    2463              :         return false;
    2464              :       }
    2465              :   return true;
    2466              : }
    2467              : 
    2468              : /* Return whether INSN, a PARALLEL of N register SETs (and maybe some
    2469              :    CLOBBERs), can be split into individual SETs in that order, without
    2470              :    changing semantics.  */
    2471              : static bool
    2472       367570 : can_split_parallel_of_n_reg_sets (rtx_insn *insn, int n)
    2473              : {
    2474       367570 :   if (!insn_nothrow_p (insn))
    2475              :     return false;
    2476              : 
    2477       366049 :   rtx pat = PATTERN (insn);
    2478              : 
    2479       366049 :   int i, j;
    2480       991347 :   for (i = 0; i < n; i++)
    2481              :     {
    2482       678698 :       if (side_effects_p (SET_SRC (XVECEXP (pat, 0, i))))
    2483              :         return false;
    2484              : 
    2485       675608 :       rtx reg = SET_DEST (XVECEXP (pat, 0, i));
    2486              : 
    2487       988257 :       for (j = i + 1; j < n; j++)
    2488       362959 :         if (reg_referenced_p (reg, XVECEXP (pat, 0, j)))
    2489              :           return false;
    2490              :     }
    2491              : 
    2492              :   return true;
    2493              : }
    2494              : 
    2495              : /* Return whether X is just a single_set, with the source
    2496              :    a general_operand.  */
    2497              : static bool
    2498     67740860 : is_just_move (rtx_insn *x)
    2499              : {
    2500     67740860 :   rtx set = single_set (x);
    2501     67740860 :   if (!set)
    2502              :     return false;
    2503              : 
    2504     67326654 :   return general_operand (SET_SRC (set), VOIDmode);
    2505              : }
    2506              : 
    2507              : /* Callback function to count autoincs.  */
    2508              : 
    2509              : static int
    2510      1023173 : count_auto_inc (rtx, rtx, rtx, rtx, rtx, void *arg)
    2511              : {
    2512      1023173 :   (*((int *) arg))++;
    2513              : 
    2514      1023173 :   return 0;
    2515              : }
    2516              : 
    2517              : /* Try to combine the insns I0, I1 and I2 into I3.
    2518              :    Here I0, I1 and I2 appear earlier than I3.
    2519              :    I0 and I1 can be zero; then we combine just I2 into I3, or I1 and I2 into
    2520              :    I3.
    2521              : 
    2522              :    If we are combining more than two insns and the resulting insn is not
    2523              :    recognized, try splitting it into two insns.  If that happens, I2 and I3
    2524              :    are retained and I1/I0 are pseudo-deleted by turning them into a NOTE.
    2525              :    Otherwise, I0, I1 and I2 are pseudo-deleted.
    2526              : 
    2527              :    Return 0 if the combination does not work.  Then nothing is changed.
    2528              :    If we did the combination, return the insn at which combine should
    2529              :    resume scanning.
    2530              : 
    2531              :    Set NEW_DIRECT_JUMP_P to true if try_combine creates a
    2532              :    new direct jump instruction.
    2533              : 
    2534              :    LAST_COMBINED_INSN is either I3, or some insn after I3 that has
    2535              :    been I3 passed to an earlier try_combine within the same basic
    2536              :    block.  */
    2537              : 
    2538              : static rtx_insn *
    2539     97695809 : try_combine (rtx_insn *i3, rtx_insn *i2, rtx_insn *i1, rtx_insn *i0,
    2540              :              bool *new_direct_jump_p, rtx_insn *last_combined_insn)
    2541              : {
    2542              :   /* New patterns for I3 and I2, respectively.  */
    2543     97695809 :   rtx newpat, newi2pat = 0;
    2544     97695809 :   rtvec newpat_vec_with_clobbers = 0;
    2545     97695809 :   bool substed_i2 = false, substed_i1 = false, substed_i0 = false;
    2546              :   /* Indicates need to preserve SET in I0, I1 or I2 in I3 if it is not
    2547              :      dead.  */
    2548     97695809 :   bool added_sets_0, added_sets_1, added_sets_2;
    2549              :   /* Total number of SETs to put into I3.  */
    2550     97695809 :   int total_sets;
    2551              :   /* Nonzero if I2's or I1's body now appears in I3.  */
    2552     97695809 :   int i2_is_used = 0, i1_is_used = 0;
    2553              :   /* INSN_CODEs for new I3, new I2, and user of condition code.  */
    2554     97695809 :   int insn_code_number, i2_code_number = 0, other_code_number = 0;
    2555              :   /* Contains I3 if the destination of I3 is used in its source, which means
    2556              :      that the old life of I3 is being killed.  If that usage is placed into
    2557              :      I2 and not in I3, a REG_DEAD note must be made.  */
    2558     97695809 :   rtx i3dest_killed = 0;
    2559              :   /* SET_DEST and SET_SRC of I2, I1 and I0.  */
    2560     97695809 :   rtx i2dest = 0, i2src = 0, i1dest = 0, i1src = 0, i0dest = 0, i0src = 0;
    2561              :   /* Copy of SET_SRC of I1 and I0, if needed.  */
    2562     97695809 :   rtx i1src_copy = 0, i0src_copy = 0, i0src_copy2 = 0;
    2563              :   /* Set if I2DEST was reused as a scratch register.  */
    2564     97695809 :   bool i2scratch = false;
    2565              :   /* The PATTERNs of I0, I1, and I2, or a copy of them in certain cases.  */
    2566     97695809 :   rtx i0pat = 0, i1pat = 0, i2pat = 0;
    2567              :   /* Indicates if I2DEST or I1DEST is in I2SRC or I1_SRC.  */
    2568     97695809 :   bool i2dest_in_i2src = false, i1dest_in_i1src = false;
    2569     97695809 :   bool i2dest_in_i1src = false, i0dest_in_i0src = false;
    2570     97695809 :   bool i1dest_in_i0src = false, i2dest_in_i0src = false;;
    2571     97695809 :   bool i2dest_killed = false, i1dest_killed = false, i0dest_killed = false;
    2572     97695809 :   bool i1_feeds_i2_n = false, i0_feeds_i2_n = false, i0_feeds_i1_n = false;
    2573              :   /* Notes that must be added to REG_NOTES in I3 and I2.  */
    2574     97695809 :   rtx new_i3_notes, new_i2_notes;
    2575              :   /* Notes that we substituted I3 into I2 instead of the normal case.  */
    2576     97695809 :   bool i3_subst_into_i2 = false;
    2577              :   /* Notes that I1, I2 or I3 is a MULT operation.  */
    2578     97695809 :   bool have_mult = false;
    2579     97695809 :   bool swap_i2i3 = false;
    2580     97695809 :   bool split_i2i3 = false;
    2581     97695809 :   bool changed_i3_dest = false;
    2582     97695809 :   bool i2_was_move = false, i3_was_move = false;
    2583     97695809 :   int n_auto_inc = 0;
    2584              : 
    2585     97695809 :   int maxreg;
    2586     97695809 :   rtx_insn *temp_insn;
    2587     97695809 :   rtx temp_expr;
    2588     97695809 :   struct insn_link *link;
    2589     97695809 :   rtx other_pat = 0;
    2590     97695809 :   rtx new_other_notes;
    2591     97695809 :   int i;
    2592     97695809 :   scalar_int_mode dest_mode, temp_mode;
    2593     97695809 :   bool has_non_call_exception = false;
    2594              : 
    2595              :   /* Immediately return if any of I0,I1,I2 are the same insn (I3 can
    2596              :      never be).  */
    2597     97695809 :   if (i1 == i2 || i0 == i2 || (i0 && i0 == i1))
    2598              :     return 0;
    2599              : 
    2600              :   /* Only try four-insn combinations when there's high likelihood of
    2601              :      success.  Look for simple insns, such as loads of constants or
    2602              :      binary operations involving a constant.  */
    2603     22435161 :   if (i0)
    2604              :     {
    2605     22435161 :       int i;
    2606     22435161 :       int ngood = 0;
    2607     22435161 :       int nshift = 0;
    2608     22435161 :       rtx set0, set3;
    2609              : 
    2610     22435161 :       if (!flag_expensive_optimizations)
    2611              :         return 0;
    2612              : 
    2613     88441385 :       for (i = 0; i < 4; i++)
    2614              :         {
    2615     72356237 :           rtx_insn *insn = i == 0 ? i0 : i == 1 ? i1 : i == 2 ? i2 : i3;
    2616     72356237 :           rtx set = single_set (insn);
    2617     72356237 :           rtx src;
    2618     72356237 :           if (!set)
    2619      2289952 :             continue;
    2620     70066285 :           src = SET_SRC (set);
    2621     70066285 :           if (CONSTANT_P (src))
    2622              :             {
    2623      4860327 :               ngood += 2;
    2624      4860327 :               break;
    2625              :             }
    2626     65205958 :           else if (BINARY_P (src) && CONSTANT_P (XEXP (src, 1)))
    2627      8400397 :             ngood++;
    2628     56805561 :           else if (GET_CODE (src) == IF_THEN_ELSE)
    2629      2152168 :             ngood++;
    2630     54653393 :           else if (GET_CODE (src) == ASHIFT || GET_CODE (src) == ASHIFTRT
    2631     54555093 :                    || GET_CODE (src) == LSHIFTRT)
    2632       130495 :             nshift++;
    2633              :         }
    2634              : 
    2635              :       /* If I0 loads a memory and I3 sets the same memory, then I1 and I2
    2636              :          are likely manipulating its value.  Ideally we'll be able to combine
    2637              :          all four insns into a bitfield insertion of some kind.
    2638              : 
    2639              :          Note the source in I0 might be inside a sign/zero extension and the
    2640              :          memory modes in I0 and I3 might be different.  So extract the address
    2641              :          from the destination of I3 and search for it in the source of I0.
    2642              : 
    2643              :          In the event that there's a match but the source/dest do not actually
    2644              :          refer to the same memory, the worst that happens is we try some
    2645              :          combinations that we wouldn't have otherwise.  */
    2646     20945475 :       if ((set0 = single_set (i0))
    2647              :           /* Ensure the source of SET0 is a MEM, possibly buried inside
    2648              :              an extension.  */
    2649     20817435 :           && (GET_CODE (SET_SRC (set0)) == MEM
    2650     17532070 :               || ((GET_CODE (SET_SRC (set0)) == ZERO_EXTEND
    2651     17532070 :                    || GET_CODE (SET_SRC (set0)) == SIGN_EXTEND)
    2652       552550 :                   && GET_CODE (XEXP (SET_SRC (set0), 0)) == MEM))
    2653      3400573 :           && (set3 = single_set (i3))
    2654              :           /* Ensure the destination of SET3 is a MEM.  */
    2655      2952306 :           && GET_CODE (SET_DEST (set3)) == MEM
    2656              :           /* Would it be better to extract the base address for the MEM
    2657              :              in SET3 and look for that?  I don't have cases where it matters
    2658              :              but I could envision such cases.  */
    2659     21247891 :           && rtx_referenced_p (XEXP (SET_DEST (set3), 0), SET_SRC (set0)))
    2660        22705 :         ngood += 2;
    2661              : 
    2662     20945475 :       if (ngood < 2 && nshift < 2)
    2663              :         return 0;
    2664              :     }
    2665              : 
    2666              :   /* Exit early if one of the insns involved can't be used for
    2667              :      combinations.  */
    2668     82425153 :   if (CALL_P (i2)
    2669     77326869 :       || (i1 && CALL_P (i1))
    2670     73865549 :       || (i0 && CALL_P (i0))
    2671     73395692 :       || cant_combine_insn_p (i3)
    2672     69959736 :       || cant_combine_insn_p (i2)
    2673     53583857 :       || (i1 && cant_combine_insn_p (i1))
    2674     48394739 :       || (i0 && cant_combine_insn_p (i0))
    2675    130581980 :       || likely_spilled_retval_p (i3))
    2676              :     return 0;
    2677              : 
    2678     48156773 :   combine_attempts++;
    2679     48156773 :   undobuf.other_insn = 0;
    2680              : 
    2681              :   /* Reset the hard register usage information.  */
    2682     48156773 :   CLEAR_HARD_REG_SET (newpat_used_regs);
    2683              : 
    2684     48156773 :   if (dump_file && (dump_flags & TDF_DETAILS))
    2685              :     {
    2686          174 :       if (i0)
    2687           20 :         fprintf (dump_file, "\nTrying %d, %d, %d -> %d:\n",
    2688           20 :                  INSN_UID (i0), INSN_UID (i1), INSN_UID (i2), INSN_UID (i3));
    2689          154 :       else if (i1)
    2690           26 :         fprintf (dump_file, "\nTrying %d, %d -> %d:\n",
    2691           26 :                  INSN_UID (i1), INSN_UID (i2), INSN_UID (i3));
    2692              :       else
    2693          128 :         fprintf (dump_file, "\nTrying %d -> %d:\n",
    2694          128 :                  INSN_UID (i2), INSN_UID (i3));
    2695              : 
    2696          174 :       if (i0)
    2697           20 :         dump_insn_slim (dump_file, i0);
    2698          174 :       if (i1)
    2699           46 :         dump_insn_slim (dump_file, i1);
    2700          174 :       dump_insn_slim (dump_file, i2);
    2701          174 :       dump_insn_slim (dump_file, i3);
    2702              :     }
    2703              : 
    2704              :   /* If multiple insns feed into one of I2 or I3, they can be in any
    2705              :      order.  To simplify the code below, reorder them in sequence.  */
    2706     48156773 :   if (i0 && DF_INSN_LUID (i0) > DF_INSN_LUID (i2))
    2707              :     std::swap (i0, i2);
    2708     48156773 :   if (i0 && DF_INSN_LUID (i0) > DF_INSN_LUID (i1))
    2709              :     std::swap (i0, i1);
    2710     48156773 :   if (i1 && DF_INSN_LUID (i1) > DF_INSN_LUID (i2))
    2711              :     std::swap (i1, i2);
    2712              : 
    2713     48156773 :   added_links_insn = 0;
    2714     48156773 :   added_notes_insn = 0;
    2715              : 
    2716              :   /* First check for one important special case that the code below will
    2717              :      not handle.  Namely, the case where I1 is zero, I2 is a PARALLEL
    2718              :      and I3 is a SET whose SET_SRC is a SET_DEST in I2.  In that case,
    2719              :      we may be able to replace that destination with the destination of I3.
    2720              :      This occurs in the common code where we compute both a quotient and
    2721              :      remainder into a structure, in which case we want to do the computation
    2722              :      directly into the structure to avoid register-register copies.
    2723              : 
    2724              :      Note that this case handles both multiple sets in I2 and also cases
    2725              :      where I2 has a number of CLOBBERs inside the PARALLEL.
    2726              : 
    2727              :      We make very conservative checks below and only try to handle the
    2728              :      most common cases of this.  For example, we only handle the case
    2729              :      where I2 and I3 are adjacent to avoid making difficult register
    2730              :      usage tests.  */
    2731              : 
    2732     30051422 :   if (i1 == 0 && NONJUMP_INSN_P (i3) && GET_CODE (PATTERN (i3)) == SET
    2733     15610748 :       && REG_P (SET_SRC (PATTERN (i3)))
    2734      5265475 :       && REGNO (SET_SRC (PATTERN (i3))) >= FIRST_PSEUDO_REGISTER
    2735      5022130 :       && find_reg_note (i3, REG_DEAD, SET_SRC (PATTERN (i3)))
    2736      4113639 :       && GET_CODE (PATTERN (i2)) == PARALLEL
    2737      1107814 :       && ! side_effects_p (SET_DEST (PATTERN (i3)))
    2738              :       /* If the dest of I3 is a ZERO_EXTRACT or STRICT_LOW_PART, the code
    2739              :          below would need to check what is inside (and reg_overlap_mentioned_p
    2740              :          doesn't support those codes anyway).  Don't allow those destinations;
    2741              :          the resulting insn isn't likely to be recognized anyway.  */
    2742       614598 :       && GET_CODE (SET_DEST (PATTERN (i3))) != ZERO_EXTRACT
    2743       614578 :       && GET_CODE (SET_DEST (PATTERN (i3))) != STRICT_LOW_PART
    2744       613366 :       && ! reg_overlap_mentioned_p (SET_SRC (PATTERN (i3)),
    2745       613366 :                                     SET_DEST (PATTERN (i3)))
    2746     48770033 :       && next_active_insn (i2) == i3)
    2747              :     {
    2748       402188 :       rtx p2 = PATTERN (i2);
    2749              : 
    2750              :       /* Make sure that the destination of I3,
    2751              :          which we are going to substitute into one output of I2,
    2752              :          is not used within another output of I2.  We must avoid making this:
    2753              :          (parallel [(set (mem (reg 69)) ...)
    2754              :                     (set (reg 69) ...)])
    2755              :          which is not well-defined as to order of actions.
    2756              :          (Besides, reload can't handle output reloads for this.)
    2757              : 
    2758              :          The problem can also happen if the dest of I3 is a memory ref,
    2759              :          if another dest in I2 is an indirect memory ref.
    2760              : 
    2761              :          Neither can this PARALLEL be an asm.  We do not allow combining
    2762              :          that usually (see can_combine_p), so do not here either.  */
    2763       402188 :       bool ok = true;
    2764      1219035 :       for (i = 0; ok && i < XVECLEN (p2, 0); i++)
    2765              :         {
    2766       816847 :           if ((GET_CODE (XVECEXP (p2, 0, i)) == SET
    2767       401793 :                || GET_CODE (XVECEXP (p2, 0, i)) == CLOBBER)
    2768      1632267 :               && reg_overlap_mentioned_p (SET_DEST (PATTERN (i3)),
    2769       815420 :                                           SET_DEST (XVECEXP (p2, 0, i))))
    2770              :             ok = false;
    2771       816062 :           else if (GET_CODE (XVECEXP (p2, 0, i)) == SET
    2772       414271 :                    && GET_CODE (SET_SRC (XVECEXP (p2, 0, i))) == ASM_OPERANDS)
    2773       816847 :             ok = false;
    2774              :         }
    2775              : 
    2776       402188 :       if (ok)
    2777       525935 :         for (i = 0; i < XVECLEN (p2, 0); i++)
    2778       464755 :           if (GET_CODE (XVECEXP (p2, 0, i)) == SET
    2779       464755 :               && SET_DEST (XVECEXP (p2, 0, i)) == SET_SRC (PATTERN (i3)))
    2780              :             {
    2781       339080 :               combine_merges++;
    2782              : 
    2783       339080 :               subst_insn = i3;
    2784       339080 :               subst_low_luid = DF_INSN_LUID (i2);
    2785              : 
    2786       339080 :               added_sets_2 = added_sets_1 = added_sets_0 = false;
    2787       339080 :               i2src = SET_SRC (XVECEXP (p2, 0, i));
    2788       339080 :               i2dest = SET_DEST (XVECEXP (p2, 0, i));
    2789       339080 :               i2dest_killed = dead_or_set_p (i2, i2dest);
    2790              : 
    2791              :               /* Replace the dest in I2 with our dest and make the resulting
    2792              :                  insn the new pattern for I3.  Then skip to where we validate
    2793              :                  the pattern.  Everything was set up above.  */
    2794       339080 :               SUBST (SET_DEST (XVECEXP (p2, 0, i)), SET_DEST (PATTERN (i3)));
    2795       339080 :               newpat = p2;
    2796       339080 :               i3_subst_into_i2 = true;
    2797       339080 :               goto validate_replacement;
    2798              :             }
    2799              :     }
    2800              : 
    2801              :   /* If I2 is setting a pseudo to a constant and I3 is setting some
    2802              :      sub-part of it to another constant, merge them by making a new
    2803              :      constant.  */
    2804     47817693 :   if (i1 == 0
    2805     29712342 :       && (temp_expr = single_set (i2)) != 0
    2806     29432711 :       && is_a <scalar_int_mode> (GET_MODE (SET_DEST (temp_expr)), &temp_mode)
    2807     19441608 :       && CONST_SCALAR_INT_P (SET_SRC (temp_expr))
    2808      2893862 :       && GET_CODE (PATTERN (i3)) == SET
    2809      1434466 :       && CONST_SCALAR_INT_P (SET_SRC (PATTERN (i3)))
    2810     47835287 :       && reg_subword_p (SET_DEST (PATTERN (i3)), SET_DEST (temp_expr)))
    2811              :     {
    2812        16842 :       rtx dest = SET_DEST (PATTERN (i3));
    2813        16842 :       rtx temp_dest = SET_DEST (temp_expr);
    2814        16842 :       int offset = -1;
    2815        16842 :       int width = 0;
    2816              : 
    2817        16842 :       if (GET_CODE (dest) == ZERO_EXTRACT)
    2818              :         {
    2819            1 :           if (CONST_INT_P (XEXP (dest, 1))
    2820            1 :               && CONST_INT_P (XEXP (dest, 2))
    2821            2 :               && is_a <scalar_int_mode> (GET_MODE (XEXP (dest, 0)),
    2822              :                                          &dest_mode))
    2823              :             {
    2824            1 :               width = INTVAL (XEXP (dest, 1));
    2825            1 :               offset = INTVAL (XEXP (dest, 2));
    2826            1 :               dest = XEXP (dest, 0);
    2827            1 :               if (BITS_BIG_ENDIAN)
    2828              :                 offset = GET_MODE_PRECISION (dest_mode) - width - offset;
    2829              :             }
    2830              :         }
    2831              :       else
    2832              :         {
    2833        16841 :           if (GET_CODE (dest) == STRICT_LOW_PART)
    2834          431 :             dest = XEXP (dest, 0);
    2835        16841 :           if (is_a <scalar_int_mode> (GET_MODE (dest), &dest_mode))
    2836              :             {
    2837        16841 :               width = GET_MODE_PRECISION (dest_mode);
    2838        16841 :               offset = 0;
    2839              :             }
    2840              :         }
    2841              : 
    2842        16842 :       if (offset >= 0)
    2843              :         {
    2844              :           /* If this is the low part, we're done.  */
    2845        16842 :           if (subreg_lowpart_p (dest))
    2846              :             ;
    2847              :           /* Handle the case where inner is twice the size of outer.  */
    2848         4982 :           else if (GET_MODE_PRECISION (temp_mode)
    2849         4982 :                    == 2 * GET_MODE_PRECISION (dest_mode))
    2850         4979 :             offset += GET_MODE_PRECISION (dest_mode);
    2851              :           /* Otherwise give up for now.  */
    2852              :           else
    2853              :             offset = -1;
    2854              :         }
    2855              : 
    2856        16839 :       if (offset >= 0)
    2857              :         {
    2858        16839 :           rtx inner = SET_SRC (PATTERN (i3));
    2859        16839 :           rtx outer = SET_SRC (temp_expr);
    2860              : 
    2861        33678 :           wide_int o = wi::insert (rtx_mode_t (outer, temp_mode),
    2862        16839 :                                    rtx_mode_t (inner, dest_mode),
    2863        33678 :                                    offset, width);
    2864              : 
    2865        16839 :           combine_merges++;
    2866        16839 :           subst_insn = i3;
    2867        16839 :           subst_low_luid = DF_INSN_LUID (i2);
    2868        16839 :           added_sets_2 = added_sets_1 = added_sets_0 = false;
    2869        16839 :           i2dest = temp_dest;
    2870        16839 :           i2dest_killed = dead_or_set_p (i2, i2dest);
    2871              : 
    2872              :           /* Replace the source in I2 with the new constant and make the
    2873              :              resulting insn the new pattern for I3.  Then skip to where we
    2874              :              validate the pattern.  Everything was set up above.  */
    2875        16839 :           SUBST (SET_SRC (temp_expr),
    2876              :                  immed_wide_int_const (o, temp_mode));
    2877              : 
    2878        16839 :           newpat = PATTERN (i2);
    2879              : 
    2880              :           /* The dest of I3 has been replaced with the dest of I2.  */
    2881        16839 :           changed_i3_dest = true;
    2882        16839 :           goto validate_replacement;
    2883        16839 :         }
    2884              :     }
    2885              : 
    2886              :   /* If we have no I1 and I2 looks like:
    2887              :         (parallel [(set (reg:CC X) (compare:CC OP (const_int 0)))
    2888              :                    (set Y OP)])
    2889              :      make up a dummy I1 that is
    2890              :         (set Y OP)
    2891              :      and change I2 to be
    2892              :         (set (reg:CC X) (compare:CC Y (const_int 0)))
    2893              : 
    2894              :      (We can ignore any trailing CLOBBERs.)
    2895              : 
    2896              :      This undoes a previous combination and allows us to match a branch-and-
    2897              :      decrement insn.  */
    2898              : 
    2899     47800854 :   if (i1 == 0
    2900     29695503 :       && is_parallel_of_n_reg_sets (PATTERN (i2), 2)
    2901       236165 :       && (GET_MODE_CLASS (GET_MODE (SET_DEST (XVECEXP (PATTERN (i2), 0, 0))))
    2902              :           == MODE_CC)
    2903       142828 :       && GET_CODE (SET_SRC (XVECEXP (PATTERN (i2), 0, 0))) == COMPARE
    2904       116023 :       && XEXP (SET_SRC (XVECEXP (PATTERN (i2), 0, 0)), 1) == const0_rtx
    2905        77681 :       && rtx_equal_p (XEXP (SET_SRC (XVECEXP (PATTERN (i2), 0, 0)), 0),
    2906        77681 :                       SET_SRC (XVECEXP (PATTERN (i2), 0, 1)))
    2907        72104 :       && !reg_used_between_p (SET_DEST (XVECEXP (PATTERN (i2), 0, 0)), i2, i3)
    2908     47872958 :       && !reg_used_between_p (SET_DEST (XVECEXP (PATTERN (i2), 0, 1)), i2, i3))
    2909              :     {
    2910              :       /* We make I1 with the same INSN_UID as I2.  This gives it
    2911              :          the same DF_INSN_LUID for value tracking.  Our fake I1 will
    2912              :          never appear in the insn stream so giving it the same INSN_UID
    2913              :          as I2 will not cause a problem.  */
    2914              : 
    2915       143772 :       i1 = gen_rtx_INSN (VOIDmode, NULL, i2, BLOCK_FOR_INSN (i2),
    2916        71886 :                          XVECEXP (PATTERN (i2), 0, 1), INSN_LOCATION (i2),
    2917              :                          -1, NULL_RTX);
    2918        71886 :       INSN_UID (i1) = INSN_UID (i2);
    2919              : 
    2920        71886 :       SUBST (PATTERN (i2), XVECEXP (PATTERN (i2), 0, 0));
    2921        71886 :       SUBST (XEXP (SET_SRC (PATTERN (i2)), 0),
    2922              :              SET_DEST (PATTERN (i1)));
    2923        71886 :       unsigned int regno = REGNO (SET_DEST (PATTERN (i1)));
    2924        71886 :       SUBST_LINK (LOG_LINKS (i2),
    2925              :                   alloc_insn_link (i1, regno, LOG_LINKS (i2)));
    2926              :     }
    2927              : 
    2928              :   /* If I2 is a PARALLEL of two SETs of REGs (and perhaps some CLOBBERs),
    2929              :      make those two SETs separate I1 and I2 insns, and make an I0 that is
    2930              :      the original I1.  */
    2931     47800854 :   if (i0 == 0
    2932     45068681 :       && is_parallel_of_n_reg_sets (PATTERN (i2), 2)
    2933       367570 :       && can_split_parallel_of_n_reg_sets (i2, 2)
    2934       312649 :       && !reg_used_between_p (SET_DEST (XVECEXP (PATTERN (i2), 0, 0)), i2, i3)
    2935       282026 :       && !reg_used_between_p (SET_DEST (XVECEXP (PATTERN (i2), 0, 1)), i2, i3)
    2936       265558 :       && !reg_set_between_p  (SET_DEST (XVECEXP (PATTERN (i2), 0, 0)), i2, i3)
    2937     48066403 :       && !reg_set_between_p  (SET_DEST (XVECEXP (PATTERN (i2), 0, 1)), i2, i3))
    2938              :     {
    2939              :       /* If there is no I1, there is no I0 either.  */
    2940       265549 :       i0 = i1;
    2941              : 
    2942              :       /* We make I1 with the same INSN_UID as I2.  This gives it
    2943              :          the same DF_INSN_LUID for value tracking.  Our fake I1 will
    2944              :          never appear in the insn stream so giving it the same INSN_UID
    2945              :          as I2 will not cause a problem.  */
    2946              : 
    2947       531098 :       i1 = gen_rtx_INSN (VOIDmode, NULL, i2, BLOCK_FOR_INSN (i2),
    2948       265549 :                          XVECEXP (PATTERN (i2), 0, 0), INSN_LOCATION (i2),
    2949              :                          -1, NULL_RTX);
    2950       265549 :       INSN_UID (i1) = INSN_UID (i2);
    2951              : 
    2952       265549 :       SUBST (PATTERN (i2), XVECEXP (PATTERN (i2), 0, 1));
    2953              :     }
    2954              : 
    2955              :   /* Verify that I2 and maybe I1 and I0 can be combined into I3.  */
    2956     47800854 :   if (!can_combine_p (i2, i3, i0, i1, NULL, NULL, &i2dest, &i2src))
    2957              :     {
    2958     12272504 :       if (dump_file && (dump_flags & TDF_DETAILS))
    2959            8 :         fprintf (dump_file, "Can't combine i2 into i3\n");
    2960     12272504 :       undo_all ();
    2961     12272504 :       return 0;
    2962              :     }
    2963     35528350 :   if (i1 && !can_combine_p (i1, i3, i0, NULL, i2, NULL, &i1dest, &i1src))
    2964              :     {
    2965      1419169 :       if (dump_file && (dump_flags & TDF_DETAILS))
    2966            0 :         fprintf (dump_file, "Can't combine i1 into i3\n");
    2967      1419169 :       undo_all ();
    2968      1419169 :       return 0;
    2969              :     }
    2970     34109181 :   if (i0 && !can_combine_p (i0, i3, NULL, NULL, i1, i2, &i0dest, &i0src))
    2971              :     {
    2972       238579 :       if (dump_file && (dump_flags & TDF_DETAILS))
    2973            0 :         fprintf (dump_file, "Can't combine i0 into i3\n");
    2974       238579 :       undo_all ();
    2975       238579 :       return 0;
    2976              :     }
    2977              : 
    2978              :   /* With non-call exceptions we can end up trying to combine multiple
    2979              :      insns with possible EH side effects.  Make sure we can combine
    2980              :      that to a single insn which means there must be at most one insn
    2981              :      in the combination with an EH side effect.  */
    2982     33870602 :   if (cfun->can_throw_non_call_exceptions)
    2983              :     {
    2984      6116706 :       if (find_reg_note (i3, REG_EH_REGION, NULL_RTX)
    2985      6093118 :           || find_reg_note (i2, REG_EH_REGION, NULL_RTX)
    2986      6093036 :           || (i1 && find_reg_note (i1, REG_EH_REGION, NULL_RTX))
    2987     12209741 :           || (i0 && find_reg_note (i0, REG_EH_REGION, NULL_RTX)))
    2988              :         {
    2989        23671 :           has_non_call_exception = true;
    2990        23671 :           if (insn_could_throw_p (i3)
    2991        23671 :               + insn_could_throw_p (i2)
    2992        23671 :               + (i1 ? insn_could_throw_p (i1) : 0)
    2993        23671 :               + (i0 ? insn_could_throw_p (i0) : 0) > 1)
    2994              :             {
    2995          172 :               if (dump_file && (dump_flags & TDF_DETAILS))
    2996            0 :                 fprintf (dump_file, "Can't combine multiple insns with EH "
    2997              :                          "side-effects\n");
    2998          172 :               undo_all ();
    2999          172 :               return 0;
    3000              :             }
    3001              :         }
    3002              :     }
    3003              : 
    3004              :   /* Record whether i2 and i3 are trivial moves.  */
    3005     33870430 :   i2_was_move = is_just_move (i2);
    3006     33870430 :   i3_was_move = is_just_move (i3);
    3007              : 
    3008              :   /* Record whether I2DEST is used in I2SRC and similarly for the other
    3009              :      cases.  Knowing this will help in register status updating below.  */
    3010     33870430 :   i2dest_in_i2src = reg_overlap_mentioned_p (i2dest, i2src);
    3011     33870430 :   i1dest_in_i1src = i1 && reg_overlap_mentioned_p (i1dest, i1src);
    3012     10904653 :   i2dest_in_i1src = i1 && reg_overlap_mentioned_p (i2dest, i1src);
    3013     33870430 :   i0dest_in_i0src = i0 && reg_overlap_mentioned_p (i0dest, i0src);
    3014      1962774 :   i1dest_in_i0src = i0 && reg_overlap_mentioned_p (i1dest, i0src);
    3015      1962774 :   i2dest_in_i0src = i0 && reg_overlap_mentioned_p (i2dest, i0src);
    3016     33870430 :   i2dest_killed = dead_or_set_p (i2, i2dest);
    3017     33870430 :   i1dest_killed = i1 && dead_or_set_p (i1, i1dest);
    3018     33870430 :   i0dest_killed = i0 && dead_or_set_p (i0, i0dest);
    3019              : 
    3020              :   /* For the earlier insns, determine which of the subsequent ones they
    3021              :      feed.  */
    3022     33870430 :   i1_feeds_i2_n = i1 && insn_a_feeds_b (i1, i2);
    3023     33870430 :   i0_feeds_i1_n = i0 && insn_a_feeds_b (i0, i1);
    3024      3425373 :   i0_feeds_i2_n = (i0 && (!i0_feeds_i1_n ? insn_a_feeds_b (i0, i2)
    3025      1462599 :                           : (!reg_overlap_mentioned_p (i1dest, i0dest)
    3026      1429824 :                              && reg_overlap_mentioned_p (i0dest, i2src))));
    3027              : 
    3028              :   /* Ensure that I3's pattern can be the destination of combines.  */
    3029     44775083 :   if (! combinable_i3pat (i3, &PATTERN (i3), i2dest, i1dest, i0dest,
    3030     10904653 :                           i1 && i2dest_in_i1src && !i1_feeds_i2_n,
    3031      1962774 :                           i0 && ((i2dest_in_i0src && !i0_feeds_i2_n)
    3032      1933836 :                                  || (i1dest_in_i0src && !i0_feeds_i1_n)),
    3033              :                           &i3dest_killed))
    3034              :     {
    3035       193550 :       undo_all ();
    3036       193550 :       return 0;
    3037              :     }
    3038              : 
    3039              :   /* See if any of the insns is a MULT operation.  Unless one is, we will
    3040              :      reject a combination that is, since it must be slower.  Be conservative
    3041              :      here.  */
    3042     33676880 :   if (GET_CODE (i2src) == MULT
    3043     32803507 :       || (i1 != 0 && GET_CODE (i1src) == MULT)
    3044     32451133 :       || (i0 != 0 && GET_CODE (i0src) == MULT)
    3045     66079127 :       || (GET_CODE (PATTERN (i3)) == SET
    3046     25301997 :           && GET_CODE (SET_SRC (PATTERN (i3))) == MULT))
    3047              :     have_mult = true;
    3048              : 
    3049              :   /* If I3 has an inc, then give up if I1 or I2 uses the reg that is inc'd.
    3050              :      We used to do this EXCEPT in one case: I3 has a post-inc in an
    3051              :      output operand.  However, that exception can give rise to insns like
    3052              :         mov r3,(r3)+
    3053              :      which is a famous insn on the PDP-11 where the value of r3 used as the
    3054              :      source was model-dependent.  Avoid this sort of thing.  */
    3055              : 
    3056              : #if 0
    3057              :   if (!(GET_CODE (PATTERN (i3)) == SET
    3058              :         && REG_P (SET_SRC (PATTERN (i3)))
    3059              :         && MEM_P (SET_DEST (PATTERN (i3)))
    3060              :         && (GET_CODE (XEXP (SET_DEST (PATTERN (i3)), 0)) == POST_INC
    3061              :             || GET_CODE (XEXP (SET_DEST (PATTERN (i3)), 0)) == POST_DEC)))
    3062              :     /* It's not the exception.  */
    3063              : #endif
    3064     33676880 :     if (AUTO_INC_DEC)
    3065              :       {
    3066              :         rtx link;
    3067              :         for (link = REG_NOTES (i3); link; link = XEXP (link, 1))
    3068              :           if (REG_NOTE_KIND (link) == REG_INC
    3069              :               && (reg_overlap_mentioned_p (XEXP (link, 0), PATTERN (i2))
    3070              :                   || (i1 != 0
    3071              :                       && reg_overlap_mentioned_p (XEXP (link, 0), PATTERN (i1)))))
    3072              :             {
    3073              :               undo_all ();
    3074              :               return 0;
    3075              :             }
    3076              :       }
    3077              : 
    3078              :   /* See if the SETs in I1 or I2 need to be kept around in the merged
    3079              :      instruction: whenever the value set there is still needed past I3.
    3080              :      For the SET in I2, this is easy: we see if I2DEST dies or is set in I3.
    3081              : 
    3082              :      For the SET in I1, we have two cases: if I1 and I2 independently feed
    3083              :      into I3, the set in I1 needs to be kept around unless I1DEST dies
    3084              :      or is set in I3.  Otherwise (if I1 feeds I2 which feeds I3), the set
    3085              :      in I1 needs to be kept around unless I1DEST dies or is set in either
    3086              :      I2 or I3.  The same considerations apply to I0.  */
    3087              : 
    3088     33676880 :   added_sets_2 = !dead_or_set_p (i3, i2dest);
    3089              : 
    3090     33676880 :   if (i1)
    3091     10837136 :     added_sets_1 = !(dead_or_set_p (i3, i1dest)
    3092      8285309 :                      || (i1_feeds_i2_n && dead_or_set_p (i2, i1dest)));
    3093              :   else
    3094              :     added_sets_1 = false;
    3095              : 
    3096     33676880 :   if (i0)
    3097      2840467 :     added_sets_0 =  !(dead_or_set_p (i3, i0dest)
    3098      1751453 :                       || (i0_feeds_i1_n && dead_or_set_p (i1, i0dest))
    3099       354346 :                       || ((i0_feeds_i2_n || (i0_feeds_i1_n && i1_feeds_i2_n))
    3100       847093 :                           && dead_or_set_p (i2, i0dest)));
    3101              :   else
    3102              :     added_sets_0 = false;
    3103              : 
    3104              :   /* We are about to copy insns for the case where they need to be kept
    3105              :      around.  Check that they can be copied in the merged instruction.  */
    3106              : 
    3107     33676880 :   if (targetm.cannot_copy_insn_p
    3108     33676880 :       && ((added_sets_2 && targetm.cannot_copy_insn_p (i2))
    3109            0 :           || (i1 && added_sets_1 && targetm.cannot_copy_insn_p (i1))
    3110            0 :           || (i0 && added_sets_0 && targetm.cannot_copy_insn_p (i0))))
    3111              :     {
    3112            0 :       undo_all ();
    3113            0 :       return 0;
    3114              :     }
    3115              : 
    3116              :   /* We cannot safely duplicate volatile references in any case.  */
    3117              : 
    3118      7538028 :   if ((added_sets_2 && volatile_refs_p (PATTERN (i2)))
    3119     33640579 :       || (added_sets_1 && volatile_refs_p (PATTERN (i1)))
    3120     67289139 :       || (added_sets_0 && volatile_refs_p (PATTERN (i0))))
    3121              :     {
    3122        66874 :       undo_all ();
    3123        66874 :       return 0;
    3124              :     }
    3125              : 
    3126              :   /* Count how many auto_inc expressions there were in the original insns;
    3127              :      we need to have the same number in the resulting patterns.  */
    3128              : 
    3129     33610006 :   if (i0)
    3130      1921915 :     for_each_inc_dec (PATTERN (i0), count_auto_inc, &n_auto_inc);
    3131     33610006 :   if (i1)
    3132     10805099 :     for_each_inc_dec (PATTERN (i1), count_auto_inc, &n_auto_inc);
    3133     33610006 :   for_each_inc_dec (PATTERN (i2), count_auto_inc, &n_auto_inc);
    3134     33610006 :   for_each_inc_dec (PATTERN (i3), count_auto_inc, &n_auto_inc);
    3135              : 
    3136              :   /* If the set in I2 needs to be kept around, we must make a copy of
    3137              :      PATTERN (I2), so that when we substitute I1SRC for I1DEST in
    3138              :      PATTERN (I2), we are only substituting for the original I1DEST, not into
    3139              :      an already-substituted copy.  This also prevents making self-referential
    3140              :      rtx.  If I2 is a PARALLEL, we just need the piece that assigns I2SRC to
    3141              :      I2DEST.  */
    3142              : 
    3143     33610006 :   if (added_sets_2)
    3144              :     {
    3145      7498798 :       if (GET_CODE (PATTERN (i2)) == PARALLEL)
    3146      2337019 :         i2pat = gen_rtx_SET (i2dest, copy_rtx (i2src));
    3147              :       else
    3148      5161779 :         i2pat = copy_rtx (PATTERN (i2));
    3149              :     }
    3150              : 
    3151     33610006 :   if (added_sets_1)
    3152              :     {
    3153      4066393 :       if (GET_CODE (PATTERN (i1)) == PARALLEL)
    3154      1293575 :         i1pat = gen_rtx_SET (i1dest, copy_rtx (i1src));
    3155              :       else
    3156      2772818 :         i1pat = copy_rtx (PATTERN (i1));
    3157              :     }
    3158              : 
    3159     33610006 :   if (added_sets_0)
    3160              :     {
    3161       528394 :       if (GET_CODE (PATTERN (i0)) == PARALLEL)
    3162       203446 :         i0pat = gen_rtx_SET (i0dest, copy_rtx (i0src));
    3163              :       else
    3164       324948 :         i0pat = copy_rtx (PATTERN (i0));
    3165              :     }
    3166              : 
    3167     33610006 :   combine_merges++;
    3168              : 
    3169              :   /* Substitute in the latest insn for the regs set by the earlier ones.  */
    3170              : 
    3171     33610006 :   maxreg = max_reg_num ();
    3172              : 
    3173     33610006 :   subst_insn = i3;
    3174              : 
    3175              :   /* Many machines have insns that can both perform an
    3176              :      arithmetic operation and set the condition code.  These operations will
    3177              :      be represented as a PARALLEL with the first element of the vector
    3178              :      being a COMPARE of an arithmetic operation with the constant zero.
    3179              :      The second element of the vector will set some pseudo to the result
    3180              :      of the same arithmetic operation.  If we simplify the COMPARE, we won't
    3181              :      match such a pattern and so will generate an extra insn.   Here we test
    3182              :      for this case, where both the comparison and the operation result are
    3183              :      needed, and make the PARALLEL by just replacing I2DEST in I3SRC with
    3184              :      I2SRC.  Later we will make the PARALLEL that contains I2.  */
    3185              : 
    3186     22804907 :   if (i1 == 0 && added_sets_2 && GET_CODE (PATTERN (i3)) == SET
    3187      4298932 :       && GET_CODE (SET_SRC (PATTERN (i3))) == COMPARE
    3188      1823672 :       && CONST_INT_P (XEXP (SET_SRC (PATTERN (i3)), 1))
    3189     34504538 :       && rtx_equal_p (XEXP (SET_SRC (PATTERN (i3)), 0), i2dest))
    3190              :     {
    3191       823469 :       rtx newpat_dest;
    3192       823469 :       rtx *cc_use_loc = NULL;
    3193       823469 :       rtx_insn *cc_use_insn = NULL;
    3194       823469 :       rtx op0 = i2src, op1 = XEXP (SET_SRC (PATTERN (i3)), 1);
    3195       823469 :       machine_mode compare_mode, orig_compare_mode;
    3196       823469 :       enum rtx_code compare_code = UNKNOWN, orig_compare_code = UNKNOWN;
    3197       823469 :       scalar_int_mode mode;
    3198              : 
    3199       823469 :       newpat = PATTERN (i3);
    3200       823469 :       newpat_dest = SET_DEST (newpat);
    3201       823469 :       compare_mode = orig_compare_mode = GET_MODE (newpat_dest);
    3202              : 
    3203       823469 :       if (undobuf.other_insn == 0
    3204       823469 :           && (cc_use_loc = find_single_use (SET_DEST (newpat), i3,
    3205              :                                             &cc_use_insn)))
    3206              :         {
    3207       816967 :           compare_code = orig_compare_code = GET_CODE (*cc_use_loc);
    3208       816967 :           if (is_a <scalar_int_mode> (GET_MODE (i2dest), &mode))
    3209       816967 :             compare_code = simplify_compare_const (compare_code, mode,
    3210              :                                                    &op0, &op1);
    3211       816967 :           target_canonicalize_comparison (&compare_code, &op0, &op1, 1);
    3212              :         }
    3213              : 
    3214              :       /* Do the rest only if op1 is const0_rtx, which may be the
    3215              :          result of simplification.  */
    3216       823469 :       if (op1 == const0_rtx)
    3217              :         {
    3218              :           /* If a single use of the CC is found, prepare to modify it
    3219              :              when SELECT_CC_MODE returns a new CC-class mode, or when
    3220              :              the above simplify_compare_const() returned a new comparison
    3221              :              operator.  undobuf.other_insn is assigned the CC use insn
    3222              :              when modifying it.  */
    3223       516633 :           if (cc_use_loc)
    3224              :             {
    3225              : #ifdef SELECT_CC_MODE
    3226       513738 :               machine_mode new_mode
    3227       513738 :                 = SELECT_CC_MODE (compare_code, op0, op1);
    3228       513738 :               if (new_mode != orig_compare_mode
    3229       513738 :                   && can_change_dest_mode (SET_DEST (newpat),
    3230              :                                            added_sets_2, new_mode))
    3231              :                 {
    3232          523 :                   unsigned int regno = REGNO (newpat_dest);
    3233          523 :                   compare_mode = new_mode;
    3234          523 :                   if (regno < FIRST_PSEUDO_REGISTER)
    3235          523 :                     newpat_dest = gen_rtx_REG (compare_mode, regno);
    3236              :                   else
    3237              :                     {
    3238            0 :                       subst_mode (regno, compare_mode);
    3239            0 :                       newpat_dest = regno_reg_rtx[regno];
    3240              :                     }
    3241              :                 }
    3242              : #endif
    3243              :               /* Cases for modifying the CC-using comparison.  */
    3244       513738 :               if (compare_code != orig_compare_code
    3245          545 :                   && COMPARISON_P (*cc_use_loc))
    3246              :                 {
    3247              :                   /* Replace cc_use_loc with entire new RTX.  */
    3248          545 :                   SUBST (*cc_use_loc,
    3249              :                          gen_rtx_fmt_ee (compare_code, GET_MODE (*cc_use_loc),
    3250              :                                          newpat_dest, const0_rtx));
    3251          545 :                   undobuf.other_insn = cc_use_insn;
    3252              :                 }
    3253       513193 :               else if (compare_mode != orig_compare_mode)
    3254              :                 {
    3255            1 :                   subrtx_ptr_iterator::array_type array;
    3256              : 
    3257              :                   /* Just replace the CC reg with a new mode.  */
    3258            4 :                   FOR_EACH_SUBRTX_PTR (iter, array, cc_use_loc, NONCONST)
    3259              :                     {
    3260            3 :                       rtx *loc = *iter;
    3261            3 :                       if (REG_P (*loc)
    3262            3 :                           && REGNO (*loc) == REGNO (newpat_dest))
    3263              :                         {
    3264            1 :                           SUBST (*loc, newpat_dest);
    3265            1 :                           iter.skip_subrtxes ();
    3266              :                         }
    3267              :                     }
    3268            1 :                   undobuf.other_insn = cc_use_insn;
    3269            1 :                 }
    3270              :             }
    3271              : 
    3272              :           /* Now we modify the current newpat:
    3273              :              First, SET_DEST(newpat) is updated if the CC mode has been
    3274              :              altered. For targets without SELECT_CC_MODE, this should be
    3275              :              optimized away.  */
    3276       516633 :           if (compare_mode != orig_compare_mode)
    3277          523 :             SUBST (SET_DEST (newpat), newpat_dest);
    3278              :           /* This is always done to propagate i2src into newpat.  */
    3279       516633 :           SUBST (SET_SRC (newpat),
    3280              :                  gen_rtx_COMPARE (compare_mode, op0, op1));
    3281              :           /* Create new version of i2pat if needed; the below PARALLEL
    3282              :              creation needs this to work correctly.  */
    3283       516633 :           if (! rtx_equal_p (i2src, op0))
    3284           27 :             i2pat = gen_rtx_SET (i2dest, op0);
    3285              :           i2_is_used = 1;
    3286              :         }
    3287              :     }
    3288              : 
    3289       823469 :   if (i2_is_used == 0)
    3290              :     {
    3291              :       /* It is possible that the source of I2 or I1 may be performing
    3292              :          an unneeded operation, such as a ZERO_EXTEND of something
    3293              :          that is known to have the high part zero.  Handle that case
    3294              :          by letting subst look at the inner insns.
    3295              : 
    3296              :          Another way to do this would be to have a function that tries
    3297              :          to simplify a single insn instead of merging two or more
    3298              :          insns.  We don't do this because of the potential of infinite
    3299              :          loops and because of the potential extra memory required.
    3300              :          However, doing it the way we are is a bit of a kludge and
    3301              :          doesn't catch all cases.
    3302              : 
    3303              :          But only do this if -fexpensive-optimizations since it slows
    3304              :          things down and doesn't usually win.
    3305              : 
    3306              :          This is not done in the COMPARE case above because the
    3307              :          unmodified I2PAT is used in the PARALLEL and so a pattern
    3308              :          with a modified I2SRC would not match.  */
    3309              : 
    3310     33093373 :       if (flag_expensive_optimizations)
    3311              :         {
    3312              :           /* Pass pc_rtx so no substitutions are done, just
    3313              :              simplifications.  */
    3314     30817877 :           if (i1)
    3315              :             {
    3316     10131824 :               subst_low_luid = DF_INSN_LUID (i1);
    3317     10131824 :               i1src = subst (i1src, pc_rtx, pc_rtx, false, false, false);
    3318              :             }
    3319              : 
    3320     30817877 :           subst_low_luid = DF_INSN_LUID (i2);
    3321     30817877 :           i2src = subst (i2src, pc_rtx, pc_rtx, false, false, false);
    3322              :         }
    3323              : 
    3324     33093373 :       n_occurrences = 0;                /* `subst' counts here */
    3325     33093373 :       subst_low_luid = DF_INSN_LUID (i2);
    3326              : 
    3327              :       /* If I1 feeds into I2 and I1DEST is in I1SRC, we need to make a unique
    3328              :          copy of I2SRC each time we substitute it, in order to avoid creating
    3329              :          self-referential RTL when we will be substituting I1SRC for I1DEST
    3330              :          later.  Likewise if I0 feeds into I2, either directly or indirectly
    3331              :          through I1, and I0DEST is in I0SRC.  */
    3332     65560756 :       newpat = subst (PATTERN (i3), i2dest, i2src, false, false,
    3333     33093373 :                       (i1_feeds_i2_n && i1dest_in_i1src)
    3334     31861533 :                       || ((i0_feeds_i2_n || (i0_feeds_i1_n && i1_feeds_i2_n))
    3335              :                           && i0dest_in_i0src));
    3336     33093373 :       substed_i2 = true;
    3337              : 
    3338              :       /* Record whether I2's body now appears within I3's body.  */
    3339     33093373 :       i2_is_used = n_occurrences;
    3340              :     }
    3341              : 
    3342              :   /* If we already got a failure, don't try to do more.  Otherwise, try to
    3343              :      substitute I1 if we have it.  */
    3344              : 
    3345     33610006 :   if (i1 && GET_CODE (newpat) != CLOBBER)
    3346              :     {
    3347              :       /* Before we can do this substitution, we must redo the test done
    3348              :          above (see detailed comments there) that ensures I1DEST isn't
    3349              :          mentioned in any SETs in NEWPAT that are field assignments.  */
    3350     10760667 :       if (!combinable_i3pat (NULL, &newpat, i1dest, NULL_RTX, NULL_RTX,
    3351              :                              false, false, 0))
    3352              :         {
    3353           82 :           undo_all ();
    3354           82 :           return 0;
    3355              :         }
    3356              : 
    3357     10760585 :       n_occurrences = 0;
    3358     10760585 :       subst_low_luid = DF_INSN_LUID (i1);
    3359              : 
    3360              :       /* If the following substitution will modify I1SRC, make a copy of it
    3361              :          for the case where it is substituted for I1DEST in I2PAT later.  */
    3362     10760585 :       if (added_sets_2 && i1_feeds_i2_n)
    3363      1510460 :         i1src_copy = copy_rtx (i1src);
    3364              : 
    3365              :       /* If I0 feeds into I1 and I0DEST is in I0SRC, we need to make a unique
    3366              :          copy of I1SRC each time we substitute it, in order to avoid creating
    3367              :          self-referential RTL when we will be substituting I0SRC for I0DEST
    3368              :          later.  */
    3369     10760585 :       newpat = subst (newpat, i1dest, i1src, false, false,
    3370              :                       i0_feeds_i1_n && i0dest_in_i0src);
    3371     10760585 :       substed_i1 = true;
    3372              : 
    3373              :       /* Record whether I1's body now appears within I3's body.  */
    3374     10760585 :       i1_is_used = n_occurrences;
    3375              :     }
    3376              : 
    3377              :   /* Likewise for I0 if we have it.  */
    3378              : 
    3379     33609924 :   if (i0 && GET_CODE (newpat) != CLOBBER)
    3380              :     {
    3381      1902683 :       if (!combinable_i3pat (NULL, &newpat, i0dest, NULL_RTX, NULL_RTX,
    3382              :                              false, false, 0))
    3383              :         {
    3384            2 :           undo_all ();
    3385            2 :           return 0;
    3386              :         }
    3387              : 
    3388              :       /* If the following substitution will modify I0SRC, make a copy of it
    3389              :          for the case where it is substituted for I0DEST in I1PAT later.  */
    3390      1902681 :       if (added_sets_1 && i0_feeds_i1_n)
    3391       365690 :         i0src_copy = copy_rtx (i0src);
    3392              :       /* And a copy for I0DEST in I2PAT substitution.  */
    3393      1902681 :       if (added_sets_2 && ((i0_feeds_i1_n && i1_feeds_i2_n)
    3394       212043 :                            || (i0_feeds_i2_n)))
    3395       347224 :         i0src_copy2 = copy_rtx (i0src);
    3396              : 
    3397      1902681 :       n_occurrences = 0;
    3398      1902681 :       subst_low_luid = DF_INSN_LUID (i0);
    3399      1902681 :       newpat = subst (newpat, i0dest, i0src, false, false, false);
    3400      1902681 :       substed_i0 = true;
    3401              :     }
    3402              : 
    3403     33609922 :   if (n_auto_inc)
    3404              :     {
    3405       512098 :       int new_n_auto_inc = 0;
    3406       512098 :       for_each_inc_dec (newpat, count_auto_inc, &new_n_auto_inc);
    3407              : 
    3408       512098 :       if (n_auto_inc != new_n_auto_inc)
    3409              :         {
    3410         1025 :           if (dump_file && (dump_flags & TDF_DETAILS))
    3411            0 :             fprintf (dump_file, "Number of auto_inc expressions changed\n");
    3412         1025 :           undo_all ();
    3413         1025 :           return 0;
    3414              :         }
    3415              :     }
    3416              : 
    3417              :   /* Fail if an autoincrement side-effect has been duplicated.  Be careful
    3418              :      to count all the ways that I2SRC and I1SRC can be used.  */
    3419     33608897 :   if ((FIND_REG_INC_NOTE (i2, NULL_RTX) != 0
    3420              :        && i2_is_used + added_sets_2 > 1)
    3421              :       || (i1 != 0 && FIND_REG_INC_NOTE (i1, NULL_RTX) != 0
    3422              :           && (i1_is_used + added_sets_1 + (added_sets_2 && i1_feeds_i2_n) > 1))
    3423              :       || (i0 != 0 && FIND_REG_INC_NOTE (i0, NULL_RTX) != 0
    3424              :           && (n_occurrences + added_sets_0
    3425              :               + (added_sets_1 && i0_feeds_i1_n)
    3426              :               + (added_sets_2 && i0_feeds_i2_n) > 1))
    3427              :       /* Fail if we tried to make a new register.  */
    3428     33608897 :       || max_reg_num () != maxreg
    3429              :       /* Fail if we couldn't do something and have a CLOBBER.  */
    3430     33608897 :       || GET_CODE (newpat) == CLOBBER
    3431              :       /* Fail if this new pattern is a MULT and we didn't have one before
    3432              :          at the outer level.  */
    3433     66861066 :       || (GET_CODE (newpat) == SET && GET_CODE (SET_SRC (newpat)) == MULT
    3434       310855 :           && ! have_mult))
    3435              :     {
    3436       388351 :       undo_all ();
    3437       388351 :       return 0;
    3438              :     }
    3439              : 
    3440              :   /* If the actions of the earlier insns must be kept
    3441              :      in addition to substituting them into the latest one,
    3442              :      we must make a new PARALLEL for the latest insn
    3443              :      to hold additional the SETs.  */
    3444              : 
    3445     33220546 :   if (added_sets_0 || added_sets_1 || added_sets_2)
    3446              :     {
    3447     11018956 :       int extra_sets = added_sets_0 + added_sets_1 + added_sets_2;
    3448     11018956 :       combine_extras++;
    3449              : 
    3450     11018956 :       if (GET_CODE (newpat) == PARALLEL)
    3451              :         {
    3452      2267029 :           rtvec old = XVEC (newpat, 0);
    3453      2267029 :           total_sets = XVECLEN (newpat, 0) + extra_sets;
    3454      2267029 :           newpat = gen_rtx_PARALLEL (VOIDmode, rtvec_alloc (total_sets));
    3455      2267029 :           memcpy (XVEC (newpat, 0)->elem, &old->elem[0],
    3456      2267029 :                   sizeof (old->elem[0]) * old->num_elem);
    3457              :         }
    3458              :       else
    3459              :         {
    3460      8751927 :           rtx old = newpat;
    3461      8751927 :           total_sets = 1 + extra_sets;
    3462      8751927 :           newpat = gen_rtx_PARALLEL (VOIDmode, rtvec_alloc (total_sets));
    3463      8751927 :           XVECEXP (newpat, 0, 0) = old;
    3464              :         }
    3465              : 
    3466     11018956 :       if (added_sets_0)
    3467       509037 :         XVECEXP (newpat, 0, --total_sets) = i0pat;
    3468              : 
    3469     11018956 :       if (added_sets_1)
    3470              :         {
    3471      4014808 :           rtx t = i1pat;
    3472      4014808 :           if (i0_feeds_i1_n)
    3473       362145 :             t = subst (t, i0dest, i0src_copy ? i0src_copy : i0src,
    3474              :                        false, false, false);
    3475              : 
    3476      4014808 :           XVECEXP (newpat, 0, --total_sets) = t;
    3477              :         }
    3478     11018956 :       if (added_sets_2)
    3479              :         {
    3480      7420262 :           rtx t = i2pat;
    3481      7420262 :           if (i1_feeds_i2_n)
    3482      1491561 :             t = subst (t, i1dest, i1src_copy ? i1src_copy : i1src, false, false,
    3483              :                        i0_feeds_i1_n && i0dest_in_i0src);
    3484      7420262 :           if ((i0_feeds_i1_n && i1_feeds_i2_n) || i0_feeds_i2_n)
    3485       342708 :             t = subst (t, i0dest, i0src_copy2 ? i0src_copy2 : i0src,
    3486              :                        false, false, false);
    3487              : 
    3488      7420262 :           XVECEXP (newpat, 0, --total_sets) = t;
    3489              :         }
    3490              :     }
    3491              : 
    3492     25800284 :  validate_replacement:
    3493              : 
    3494              :   /* Note which hard regs this insn has as inputs.  */
    3495     33576465 :   mark_used_regs_combine (newpat);
    3496              : 
    3497              :   /* If recog_for_combine fails, it strips existing clobbers.  If we'll
    3498              :      consider splitting this pattern, we might need these clobbers.  */
    3499     33576465 :   if (i1 && GET_CODE (newpat) == PARALLEL
    3500      7438947 :       && GET_CODE (XVECEXP (newpat, 0, XVECLEN (newpat, 0) - 1)) == CLOBBER)
    3501              :     {
    3502      1751994 :       int len = XVECLEN (newpat, 0);
    3503              : 
    3504      1751994 :       newpat_vec_with_clobbers = rtvec_alloc (len);
    3505      7058421 :       for (i = 0; i < len; i++)
    3506      3554433 :         RTVEC_ELT (newpat_vec_with_clobbers, i) = XVECEXP (newpat, 0, i);
    3507              :     }
    3508              : 
    3509              :   /* We have recognized nothing yet.  */
    3510     33576465 :   insn_code_number = -1;
    3511              : 
    3512              :   /* See if this is a PARALLEL of two SETs where one SET's destination is
    3513              :      a register that is unused and this isn't marked as an instruction that
    3514              :      might trap in an EH region.  In that case, we just need the other SET.
    3515              :      We prefer this over the PARALLEL.
    3516              : 
    3517              :      This can occur when simplifying a divmod insn.  We *must* test for this
    3518              :      case here because the code below that splits two independent SETs doesn't
    3519              :      handle this case correctly when it updates the register status.
    3520              : 
    3521              :      It's pointless doing this if we originally had two sets, one from
    3522              :      i3, and one from i2.  Combining then splitting the parallel results
    3523              :      in the original i2 again plus an invalid insn (which we delete).
    3524              :      The net effect is only to move instructions around, which makes
    3525              :      debug info less accurate.
    3526              : 
    3527              :      If the remaining SET came from I2 its destination should not be used
    3528              :      between I2 and I3.  See PR82024.  */
    3529              : 
    3530      7420262 :   if (!(added_sets_2 && i1 == 0)
    3531     28177907 :       && is_parallel_of_n_reg_sets (newpat, 2)
    3532     35193867 :       && asm_noperands (newpat) < 0)
    3533              :     {
    3534      1616508 :       rtx set0 = XVECEXP (newpat, 0, 0);
    3535      1616508 :       rtx set1 = XVECEXP (newpat, 0, 1);
    3536      1616508 :       rtx oldpat = newpat;
    3537              : 
    3538      1616508 :       if (((REG_P (SET_DEST (set1))
    3539      1616508 :             && find_reg_note (i3, REG_UNUSED, SET_DEST (set1)))
    3540      1575027 :            || (GET_CODE (SET_DEST (set1)) == SUBREG
    3541            0 :                && find_reg_note (i3, REG_UNUSED, SUBREG_REG (SET_DEST (set1)))))
    3542        41481 :           && insn_nothrow_p (i3)
    3543      1656760 :           && !side_effects_p (SET_SRC (set1)))
    3544              :         {
    3545        40013 :           newpat = set0;
    3546        40013 :           insn_code_number = recog_for_combine (&newpat, i3, &new_i3_notes);
    3547              :         }
    3548              : 
    3549      1576495 :       else if (((REG_P (SET_DEST (set0))
    3550      1576495 :                  && find_reg_note (i3, REG_UNUSED, SET_DEST (set0)))
    3551      1551046 :                 || (GET_CODE (SET_DEST (set0)) == SUBREG
    3552            0 :                     && find_reg_note (i3, REG_UNUSED,
    3553            0 :                                       SUBREG_REG (SET_DEST (set0)))))
    3554        25449 :                && insn_nothrow_p (i3)
    3555      1601343 :                && !side_effects_p (SET_SRC (set0)))
    3556              :         {
    3557        24805 :           rtx dest = SET_DEST (set1);
    3558        24805 :           if (GET_CODE (dest) == SUBREG)
    3559            0 :             dest = SUBREG_REG (dest);
    3560        24805 :           if (!reg_used_between_p (dest, i2, i3))
    3561              :             {
    3562        24804 :               newpat = set1;
    3563        24804 :               insn_code_number = recog_for_combine (&newpat, i3, &new_i3_notes);
    3564              : 
    3565        24804 :               if (insn_code_number >= 0)
    3566              :                 changed_i3_dest = true;
    3567              :             }
    3568              :         }
    3569              : 
    3570        40013 :       if (insn_code_number < 0)
    3571      1610903 :         newpat = oldpat;
    3572              :     }
    3573              : 
    3574              :   /* Is the result of combination a valid instruction?  */
    3575      1610903 :   if (insn_code_number < 0)
    3576     33570860 :     insn_code_number = recog_for_combine (&newpat, i3, &new_i3_notes);
    3577              : 
    3578              :   /* If we were combining three insns and the result is a simple SET
    3579              :      with no ASM_OPERANDS that wasn't recognized, try to split it into two
    3580              :      insns.  There are two ways to do this.  It can be split using a
    3581              :      machine-specific method (like when you have an addition of a large
    3582              :      constant) or by combine in the function find_split_point.  */
    3583              : 
    3584     10600595 :   if (i1 && insn_code_number < 0 && GET_CODE (newpat) == SET
    3585     38401622 :       && asm_noperands (newpat) < 0)
    3586              :     {
    3587      4824598 :       rtx parallel, *split;
    3588      4824598 :       rtx_insn *m_split_insn;
    3589      4824598 :       unsigned int old_nregs, new_nregs;
    3590              : 
    3591              :       /* See if the MD file can split NEWPAT.  If it can't, see if letting it
    3592              :          use I2DEST as a scratch register will help.  In the latter case,
    3593              :          convert I2DEST to the mode of the source of NEWPAT if we can.  */
    3594              : 
    3595      4824598 :       m_split_insn = combine_split_insns (newpat, i3, &old_nregs, &new_nregs);
    3596              : 
    3597              :       /* We can only use I2DEST as a scratch reg if it doesn't overlap any
    3598              :          inputs of NEWPAT.  */
    3599              : 
    3600              :       /* ??? If I2DEST is not safe, and I1DEST exists, then it would be
    3601              :          possible to try that as a scratch reg.  This would require adding
    3602              :          more code to make it work though.  */
    3603              : 
    3604      4824598 :       if (m_split_insn == 0 && ! reg_overlap_mentioned_p (i2dest, newpat))
    3605              :         {
    3606      4682415 :           machine_mode new_mode = GET_MODE (SET_DEST (newpat));
    3607              : 
    3608              :           /* ??? Reusing i2dest without resetting the reg_stat entry for it
    3609              :              (temporarily, until we are committed to this instruction
    3610              :              combination) does not work: for example, any call to nonzero_bits
    3611              :              on the register (from a splitter in the MD file, for example)
    3612              :              will get the old information, which is invalid.
    3613              : 
    3614              :              Since nowadays we can create registers during combine just fine,
    3615              :              we should just create a new one here, not reuse i2dest.  */
    3616              : 
    3617              :           /* First try to split using the original register as a
    3618              :              scratch register.  */
    3619      4682415 :           parallel = gen_rtx_PARALLEL (VOIDmode,
    3620              :                                        gen_rtvec (2, newpat,
    3621              :                                                   gen_rtx_CLOBBER (VOIDmode,
    3622              :                                                                    i2dest)));
    3623      4682415 :           m_split_insn = combine_split_insns (parallel, i3, &old_nregs, &new_nregs);
    3624              : 
    3625              :           /* If that didn't work, try changing the mode of I2DEST if
    3626              :              we can.  */
    3627      4682415 :           if (m_split_insn == 0
    3628      4682415 :               && new_mode != GET_MODE (i2dest)
    3629      1789932 :               && new_mode != VOIDmode
    3630      5890313 :               && can_change_dest_mode (i2dest, added_sets_2, new_mode))
    3631              :             {
    3632       889592 :               machine_mode old_mode = GET_MODE (i2dest);
    3633       889592 :               rtx ni2dest;
    3634              : 
    3635       889592 :               if (REGNO (i2dest) < FIRST_PSEUDO_REGISTER)
    3636         9065 :                 ni2dest = gen_rtx_REG (new_mode, REGNO (i2dest));
    3637              :               else
    3638              :                 {
    3639       880527 :                   subst_mode (REGNO (i2dest), new_mode);
    3640       880527 :                   ni2dest = regno_reg_rtx[REGNO (i2dest)];
    3641              :                 }
    3642              : 
    3643       889592 :               parallel = (gen_rtx_PARALLEL
    3644              :                           (VOIDmode,
    3645              :                            gen_rtvec (2, newpat,
    3646              :                                       gen_rtx_CLOBBER (VOIDmode,
    3647              :                                                        ni2dest))));
    3648       889592 :               m_split_insn = combine_split_insns (parallel, i3, &old_nregs, &new_nregs);
    3649              : 
    3650       889592 :               if (m_split_insn == 0
    3651       889592 :                   && REGNO (i2dest) >= FIRST_PSEUDO_REGISTER)
    3652              :                 {
    3653       880527 :                   struct undo *buf;
    3654              : 
    3655       880527 :                   adjust_reg_mode (regno_reg_rtx[REGNO (i2dest)], old_mode);
    3656       880527 :                   buf = undobuf.undos;
    3657       880527 :                   undobuf.undos = buf->next;
    3658       880527 :                   buf->next = undobuf.frees;
    3659       880527 :                   undobuf.frees = buf;
    3660              :                 }
    3661              :             }
    3662              : 
    3663      4682415 :           i2scratch = m_split_insn != 0;
    3664              :         }
    3665              : 
    3666              :       /* If recog_for_combine has discarded clobbers, try to use them
    3667              :          again for the split.  */
    3668      4824598 :       if (m_split_insn == 0 && newpat_vec_with_clobbers)
    3669              :         {
    3670      1700357 :           parallel = gen_rtx_PARALLEL (VOIDmode, newpat_vec_with_clobbers);
    3671      1700357 :           m_split_insn = combine_split_insns (parallel, i3, &old_nregs, &new_nregs);
    3672              :         }
    3673              : 
    3674      4836320 :       if (m_split_insn && NEXT_INSN (m_split_insn) == NULL_RTX)
    3675              :         {
    3676         1670 :           rtx m_split_pat = PATTERN (m_split_insn);
    3677         1670 :           insn_code_number = recog_for_combine (&m_split_pat, i3, &new_i3_notes,
    3678              :                                                 old_nregs, new_nregs);
    3679         1670 :           if (insn_code_number >= 0)
    3680          259 :             newpat = m_split_pat;
    3681              :         }
    3682        10052 :       else if (m_split_insn && NEXT_INSN (NEXT_INSN (m_split_insn)) == NULL_RTX
    3683      4832980 :                && (next_nonnote_nondebug_insn (i2) == i3
    3684            6 :                    || !modified_between_p (PATTERN (m_split_insn), i2, i3)))
    3685              :         {
    3686        10052 :           rtx i2set, i3set;
    3687        10052 :           rtx newi3pat = PATTERN (NEXT_INSN (m_split_insn));
    3688        10052 :           newi2pat = PATTERN (m_split_insn);
    3689              : 
    3690        10052 :           i3set = single_set (NEXT_INSN (m_split_insn));
    3691        10052 :           i2set = single_set (m_split_insn);
    3692              : 
    3693        10052 :           i2_code_number = recog_for_combine (&newi2pat, i2, &new_i2_notes);
    3694              : 
    3695              :           /* If I2 or I3 has multiple SETs, we won't know how to track
    3696              :              register status, so don't use these insns.  If I2's destination
    3697              :              is used between I2 and I3, we also can't use these insns.  */
    3698              : 
    3699        10052 :           if (i2_code_number >= 0 && i2set && i3set
    3700        20104 :               && (next_nonnote_nondebug_insn (i2) == i3
    3701            6 :                   || ! reg_used_between_p (SET_DEST (i2set), i2, i3)))
    3702        10052 :             insn_code_number = recog_for_combine (&newi3pat, i3,
    3703              :                                                   &new_i3_notes,
    3704              :                                                   old_nregs, new_nregs);
    3705        10052 :           if (insn_code_number >= 0)
    3706        10052 :             newpat = newi3pat;
    3707              : 
    3708              :           /* It is possible that both insns now set the destination of I3.
    3709              :              If so, we must show an extra use of it.  */
    3710              : 
    3711        10052 :           if (insn_code_number >= 0)
    3712              :             {
    3713        10052 :               rtx new_i3_dest = SET_DEST (i3set);
    3714        10052 :               rtx new_i2_dest = SET_DEST (i2set);
    3715              : 
    3716        10052 :               while (GET_CODE (new_i3_dest) == ZERO_EXTRACT
    3717        10092 :                      || GET_CODE (new_i3_dest) == STRICT_LOW_PART
    3718        20166 :                      || GET_CODE (new_i3_dest) == SUBREG)
    3719           40 :                 new_i3_dest = XEXP (new_i3_dest, 0);
    3720              : 
    3721        10052 :               while (GET_CODE (new_i2_dest) == ZERO_EXTRACT
    3722        10052 :                      || GET_CODE (new_i2_dest) == STRICT_LOW_PART
    3723        20104 :                      || GET_CODE (new_i2_dest) == SUBREG)
    3724            0 :                 new_i2_dest = XEXP (new_i2_dest, 0);
    3725              : 
    3726        10052 :               if (REG_P (new_i3_dest)
    3727         5775 :                   && REG_P (new_i2_dest)
    3728         5775 :                   && REGNO (new_i3_dest) == REGNO (new_i2_dest)
    3729        10052 :                   && REGNO (new_i2_dest) < reg_n_sets_max)
    3730            0 :                 INC_REG_N_SETS (REGNO (new_i2_dest), 1);
    3731              :             }
    3732              :         }
    3733              : 
    3734              :       /* If we can split it and use I2DEST, go ahead and see if that
    3735              :          helps things be recognized.  Verify that none of the registers
    3736              :          are set between I2 and I3.  */
    3737         1411 :       if (insn_code_number < 0
    3738      4814287 :           && (split = find_split_point (&newpat, i3, false)) != 0
    3739              :           /* We need I2DEST in the proper mode.  If it is a hard register
    3740              :              or the only use of a pseudo, we can change its mode.
    3741              :              Make sure we don't change a hard register to have a mode that
    3742              :              isn't valid for it, or change the number of registers.  */
    3743      4544222 :           && (GET_MODE (*split) == GET_MODE (i2dest)
    3744      1686329 :               || GET_MODE (*split) == VOIDmode
    3745      1308788 :               || can_change_dest_mode (i2dest, added_sets_2,
    3746              :                                        GET_MODE (*split)))
    3747      3804387 :           && (next_nonnote_nondebug_insn (i2) == i3
    3748       589603 :               || !modified_between_p (*split, i2, i3))
    3749              :           /* We can't overwrite I2DEST if its value is still used by
    3750              :              NEWPAT.  */
    3751      3775065 :           && ! reg_referenced_p (i2dest, newpat)
    3752              :           /* We should not split a possibly trapping part when we
    3753              :              care about non-call EH and have REG_EH_REGION notes
    3754              :              to distribute.  */
    3755      8521509 :           && ! (cfun->can_throw_non_call_exceptions
    3756       396646 :                 && has_non_call_exception
    3757          121 :                 && may_trap_p (*split)))
    3758              :         {
    3759      3698201 :           rtx newdest = i2dest;
    3760      3698201 :           enum rtx_code split_code = GET_CODE (*split);
    3761      3698201 :           machine_mode split_mode = GET_MODE (*split);
    3762      3698201 :           bool subst_done = false;
    3763      3698201 :           newi2pat = NULL_RTX;
    3764              : 
    3765      3698201 :           i2scratch = true;
    3766              : 
    3767              :           /* *SPLIT may be part of I2SRC, so make sure we have the
    3768              :              original expression around for later debug processing.
    3769              :              We should not need I2SRC any more in other cases.  */
    3770      3698201 :           if (MAY_HAVE_DEBUG_BIND_INSNS)
    3771      1828753 :             i2src = copy_rtx (i2src);
    3772              :           else
    3773              :             i2src = NULL;
    3774              : 
    3775              :           /* Get NEWDEST as a register in the proper mode.  We have already
    3776              :              validated that we can do this.  */
    3777      3698201 :           if (GET_MODE (i2dest) != split_mode && split_mode != VOIDmode)
    3778              :             {
    3779       565445 :               if (REGNO (i2dest) < FIRST_PSEUDO_REGISTER)
    3780            0 :                 newdest = gen_rtx_REG (split_mode, REGNO (i2dest));
    3781              :               else
    3782              :                 {
    3783       565445 :                   subst_mode (REGNO (i2dest), split_mode);
    3784       565445 :                   newdest = regno_reg_rtx[REGNO (i2dest)];
    3785              :                 }
    3786              :             }
    3787              : 
    3788              :           /* If *SPLIT is a (mult FOO (const_int pow2)), convert it to
    3789              :              an ASHIFT.  This can occur if it was inside a PLUS and hence
    3790              :              appeared to be a memory address.  This is a kludge.  */
    3791      3698201 :           if (split_code == MULT
    3792       205955 :               && CONST_INT_P (XEXP (*split, 1))
    3793       101238 :               && INTVAL (XEXP (*split, 1)) > 0
    3794      3795032 :               && (i = exact_log2 (UINTVAL (XEXP (*split, 1)))) >= 0)
    3795              :             {
    3796        67307 :               rtx i_rtx = gen_int_shift_amount (split_mode, i);
    3797        67307 :               SUBST (*split, gen_rtx_ASHIFT (split_mode,
    3798              :                                              XEXP (*split, 0), i_rtx));
    3799              :               /* Update split_code because we may not have a multiply
    3800              :                  anymore.  */
    3801        67307 :               split_code = GET_CODE (*split);
    3802              :             }
    3803              : 
    3804              :           /* Similarly for (plus (mult FOO (const_int pow2))).  */
    3805      3698201 :           if (split_code == PLUS
    3806       719944 :               && GET_CODE (XEXP (*split, 0)) == MULT
    3807       114137 :               && CONST_INT_P (XEXP (XEXP (*split, 0), 1))
    3808        39428 :               && INTVAL (XEXP (XEXP (*split, 0), 1)) > 0
    3809      3734174 :               && (i = exact_log2 (UINTVAL (XEXP (XEXP (*split, 0), 1)))) >= 0)
    3810              :             {
    3811         6955 :               rtx nsplit = XEXP (*split, 0);
    3812         6955 :               rtx i_rtx = gen_int_shift_amount (GET_MODE (nsplit), i);
    3813         6955 :               SUBST (XEXP (*split, 0), gen_rtx_ASHIFT (GET_MODE (nsplit),
    3814              :                                                        XEXP (nsplit, 0),
    3815              :                                                        i_rtx));
    3816              :               /* Update split_code because we may not have a multiply
    3817              :                  anymore.  */
    3818         6955 :               split_code = GET_CODE (*split);
    3819              :             }
    3820              : 
    3821              : #ifdef INSN_SCHEDULING
    3822              :           /* If *SPLIT is a paradoxical SUBREG, when we split it, it should
    3823              :              be written as a ZERO_EXTEND.  */
    3824      3698201 :           if (split_code == SUBREG && MEM_P (SUBREG_REG (*split)))
    3825              :             {
    3826              :               /* Or as a SIGN_EXTEND if LOAD_EXTEND_OP says that that's
    3827              :                  what it really is.  */
    3828        10527 :               if (load_extend_op (GET_MODE (SUBREG_REG (*split)))
    3829              :                   == SIGN_EXTEND)
    3830              :                 SUBST (*split, gen_rtx_SIGN_EXTEND (split_mode,
    3831              :                                                     SUBREG_REG (*split)));
    3832              :               else
    3833        10527 :                 SUBST (*split, gen_rtx_ZERO_EXTEND (split_mode,
    3834              :                                                     SUBREG_REG (*split)));
    3835              :             }
    3836              : #endif
    3837              : 
    3838              :           /* Attempt to split binary operators using arithmetic identities.  */
    3839      3698201 :           if (BINARY_P (SET_SRC (newpat))
    3840      3097536 :               && split_mode == GET_MODE (SET_SRC (newpat))
    3841      5836091 :               && ! side_effects_p (SET_SRC (newpat)))
    3842              :             {
    3843      2122877 :               rtx setsrc = SET_SRC (newpat);
    3844      2122877 :               machine_mode mode = GET_MODE (setsrc);
    3845      2122877 :               enum rtx_code code = GET_CODE (setsrc);
    3846      2122877 :               rtx src_op0 = XEXP (setsrc, 0);
    3847      2122877 :               rtx src_op1 = XEXP (setsrc, 1);
    3848              : 
    3849              :               /* Split "X = Y op Y" as "Z = Y; X = Z op Z".  */
    3850      2122877 :               if (rtx_equal_p (src_op0, src_op1))
    3851              :                 {
    3852         1579 :                   newi2pat = gen_rtx_SET (newdest, src_op0);
    3853         1579 :                   SUBST (XEXP (setsrc, 0), newdest);
    3854         1579 :                   SUBST (XEXP (setsrc, 1), newdest);
    3855         1579 :                   subst_done = true;
    3856              :                 }
    3857              :               /* Split "((P op Q) op R) op S" where op is PLUS or MULT.  */
    3858      2121298 :               else if ((code == PLUS || code == MULT)
    3859      1112940 :                        && GET_CODE (src_op0) == code
    3860       423570 :                        && GET_CODE (XEXP (src_op0, 0)) == code
    3861       182823 :                        && (INTEGRAL_MODE_P (mode)
    3862              :                            || (FLOAT_MODE_P (mode)
    3863        99654 :                                && flag_unsafe_math_optimizations)))
    3864              :                 {
    3865        86976 :                   rtx p = XEXP (XEXP (src_op0, 0), 0);
    3866        86976 :                   rtx q = XEXP (XEXP (src_op0, 0), 1);
    3867        86976 :                   rtx r = XEXP (src_op0, 1);
    3868        86976 :                   rtx s = src_op1;
    3869              : 
    3870              :                   /* Split both "((X op Y) op X) op Y" and
    3871              :                      "((X op Y) op Y) op X" as "T op T" where T is
    3872              :                      "X op Y".  */
    3873        87221 :                   if ((rtx_equal_p (p,r) && rtx_equal_p (q,s))
    3874        87142 :                        || (rtx_equal_p (p,s) && rtx_equal_p (q,r)))
    3875              :                     {
    3876           79 :                       newi2pat = gen_rtx_SET (newdest, XEXP (src_op0, 0));
    3877           79 :                       SUBST (XEXP (setsrc, 0), newdest);
    3878           79 :                       SUBST (XEXP (setsrc, 1), newdest);
    3879           79 :                       subst_done = true;
    3880              :                     }
    3881              :                   /* Split "((X op X) op Y) op Y)" as "T op T" where
    3882              :                      T is "X op Y".  */
    3883        86897 :                   else if (rtx_equal_p (p,q) && rtx_equal_p (r,s))
    3884              :                     {
    3885           41 :                       rtx tmp = simplify_gen_binary (code, mode, p, r);
    3886           41 :                       newi2pat = gen_rtx_SET (newdest, tmp);
    3887           41 :                       SUBST (XEXP (setsrc, 0), newdest);
    3888           41 :                       SUBST (XEXP (setsrc, 1), newdest);
    3889           41 :                       subst_done = true;
    3890              :                     }
    3891              :                 }
    3892              :             }
    3893              : 
    3894         1699 :           if (!subst_done)
    3895              :             {
    3896      3696502 :               newi2pat = gen_rtx_SET (newdest, *split);
    3897      3696502 :               SUBST (*split, newdest);
    3898              :             }
    3899              : 
    3900      3698201 :           i2_code_number = recog_for_combine (&newi2pat, i2, &new_i2_notes);
    3901              : 
    3902              :           /* recog_for_combine might have added CLOBBERs to newi2pat.
    3903              :              Make sure NEWPAT does not depend on the clobbered regs.  */
    3904      3698201 :           if (GET_CODE (newi2pat) == PARALLEL)
    3905      2530470 :             for (i = XVECLEN (newi2pat, 0) - 1; i >= 0; i--)
    3906      1699807 :               if (GET_CODE (XVECEXP (newi2pat, 0, i)) == CLOBBER)
    3907              :                 {
    3908       869144 :                   rtx reg = XEXP (XVECEXP (newi2pat, 0, i), 0);
    3909       869144 :                   if (reg_overlap_mentioned_p (reg, newpat))
    3910              :                     {
    3911        24214 :                       undo_all ();
    3912        24214 :                       return 0;
    3913              :                     }
    3914              :                 }
    3915              : 
    3916              :           /* If the split point was a MULT and we didn't have one before,
    3917              :              don't use one now.  */
    3918      3673987 :           if (i2_code_number >= 0 && ! (split_code == MULT && ! have_mult))
    3919      2189322 :             insn_code_number = recog_for_combine (&newpat, i3, &new_i3_notes);
    3920              :         }
    3921              :     }
    3922              : 
    3923              :   /* Check for a case where we loaded from memory in a narrow mode and
    3924              :      then sign extended it, but we need both registers.  In that case,
    3925              :      we have a PARALLEL with both loads from the same memory location.
    3926              :      We can split this into a load from memory followed by a register-register
    3927              :      copy.  This saves at least one insn, more if register allocation can
    3928              :      eliminate the copy.
    3929              : 
    3930              :      We cannot do this if the involved modes have more than one elements,
    3931              :      like for vector or complex modes.
    3932              : 
    3933              :      We cannot do this if the destination of the first assignment is a
    3934              :      condition code register.  We eliminate this case by making sure
    3935              :      the SET_DEST and SET_SRC have the same mode.
    3936              : 
    3937              :      We cannot do this if the destination of the second assignment is
    3938              :      a register that we have already assumed is zero-extended.  Similarly
    3939              :      for a SUBREG of such a register.  */
    3940              : 
    3941      5775997 :   else if (i1 && insn_code_number < 0 && asm_noperands (newpat) < 0
    3942      5716689 :            && GET_CODE (newpat) == PARALLEL
    3943      5714835 :            && XVECLEN (newpat, 0) == 2
    3944      4739596 :            && GET_CODE (XVECEXP (newpat, 0, 0)) == SET
    3945      4739331 :            && GET_CODE (SET_SRC (XVECEXP (newpat, 0, 0))) == SIGN_EXTEND
    3946        23389 :            && (GET_MODE (SET_DEST (XVECEXP (newpat, 0, 0)))
    3947        23389 :                == GET_MODE (SET_SRC (XVECEXP (newpat, 0, 0))))
    3948        23389 :            && ! VECTOR_MODE_P (GET_MODE (SET_DEST (XVECEXP (newpat, 0, 0))))
    3949              :            && ! COMPLEX_MODE_P (GET_MODE (SET_DEST (XVECEXP (newpat, 0, 0))))
    3950        21955 :            && GET_CODE (XVECEXP (newpat, 0, 1)) == SET
    3951        21955 :            && rtx_equal_p (SET_SRC (XVECEXP (newpat, 0, 1)),
    3952        21955 :                            XEXP (SET_SRC (XVECEXP (newpat, 0, 0)), 0))
    3953         6520 :            && !modified_between_p (SET_SRC (XVECEXP (newpat, 0, 1)), i2, i3)
    3954         6520 :            && GET_CODE (SET_DEST (XVECEXP (newpat, 0, 1))) != ZERO_EXTRACT
    3955         6520 :            && GET_CODE (SET_DEST (XVECEXP (newpat, 0, 1))) != STRICT_LOW_PART
    3956         6520 :            && ! (temp_expr = SET_DEST (XVECEXP (newpat, 0, 1)),
    3957              :                  (REG_P (temp_expr)
    3958         6520 :                   && reg_stat[REGNO (temp_expr)].nonzero_bits != 0
    3959         6595 :                   && known_lt (GET_MODE_PRECISION (GET_MODE (temp_expr)),
    3960              :                                BITS_PER_WORD)
    3961         6419 :                   && known_lt (GET_MODE_PRECISION (GET_MODE (temp_expr)),
    3962              :                                HOST_BITS_PER_INT)
    3963         1142 :                   && (reg_stat[REGNO (temp_expr)].nonzero_bits
    3964         1142 :                       != GET_MODE_MASK (word_mode))))
    3965         6502 :            && ! (GET_CODE (SET_DEST (XVECEXP (newpat, 0, 1))) == SUBREG
    3966            0 :                  && (temp_expr = SUBREG_REG (SET_DEST (XVECEXP (newpat, 0, 1))),
    3967            0 :                      (REG_P (temp_expr)
    3968            0 :                       && reg_stat[REGNO (temp_expr)].nonzero_bits != 0
    3969            0 :                       && known_lt (GET_MODE_PRECISION (GET_MODE (temp_expr)),
    3970              :                                    BITS_PER_WORD)
    3971            0 :                       && known_lt (GET_MODE_PRECISION (GET_MODE (temp_expr)),
    3972              :                                    HOST_BITS_PER_INT)
    3973            0 :                       && (reg_stat[REGNO (temp_expr)].nonzero_bits
    3974            0 :                           != GET_MODE_MASK (word_mode)))))
    3975         6502 :            && ! reg_overlap_mentioned_p (SET_DEST (XVECEXP (newpat, 0, 1)),
    3976         6502 :                                          SET_SRC (XVECEXP (newpat, 0, 1)))
    3977     28758324 :            && ! find_reg_note (i3, REG_UNUSED,
    3978         6457 :                                SET_DEST (XVECEXP (newpat, 0, 0))))
    3979              :     {
    3980         6457 :       rtx ni2dest;
    3981              : 
    3982         6457 :       newi2pat = XVECEXP (newpat, 0, 0);
    3983         6457 :       ni2dest = SET_DEST (XVECEXP (newpat, 0, 0));
    3984         6457 :       newpat = XVECEXP (newpat, 0, 1);
    3985         6457 :       SUBST (SET_SRC (newpat),
    3986              :              gen_lowpart (GET_MODE (SET_SRC (newpat)), ni2dest));
    3987         6457 :       i2_code_number = recog_for_combine (&newi2pat, i2, &new_i2_notes);
    3988              : 
    3989         6457 :       if (i2_code_number >= 0)
    3990            0 :         insn_code_number = recog_for_combine (&newpat, i3, &new_i3_notes);
    3991              : 
    3992         6457 :       if (insn_code_number >= 0)
    3993              :         swap_i2i3 = 1;
    3994              :     }
    3995              : 
    3996              :   /* Similarly, check for a case where we have a PARALLEL of two independent
    3997              :      SETs but we started with three insns.  In this case, we can do the sets
    3998              :      as two separate insns.  This case occurs when some SET allows two
    3999              :      other insns to combine, but the destination of that SET is still live.
    4000              : 
    4001              :      Also do this if we started with two insns and (at least) one of the
    4002              :      resulting sets is a noop; this noop will be deleted later.
    4003              : 
    4004              :      Also do this if we started with two insns neither of which was a simple
    4005              :      move.  */
    4006              : 
    4007     24644218 :   else if (insn_code_number < 0 && asm_noperands (newpat) < 0
    4008     24625869 :            && GET_CODE (newpat) == PARALLEL
    4009     11225460 :            && XVECLEN (newpat, 0) == 2
    4010     10146460 :            && GET_CODE (XVECEXP (newpat, 0, 0)) == SET
    4011     10037373 :            && GET_CODE (XVECEXP (newpat, 0, 1)) == SET
    4012      9973343 :            && (i1
    4013      5263142 :                || set_noop_p (XVECEXP (newpat, 0, 0))
    4014      5262669 :                || set_noop_p (XVECEXP (newpat, 0, 1))
    4015      5262665 :                || (!i2_was_move && !i3_was_move))
    4016      6611436 :            && GET_CODE (SET_DEST (XVECEXP (newpat, 0, 0))) != ZERO_EXTRACT
    4017      6610743 :            && GET_CODE (SET_DEST (XVECEXP (newpat, 0, 0))) != STRICT_LOW_PART
    4018      6610600 :            && GET_CODE (SET_DEST (XVECEXP (newpat, 0, 1))) != ZERO_EXTRACT
    4019      6610019 :            && GET_CODE (SET_DEST (XVECEXP (newpat, 0, 1))) != STRICT_LOW_PART
    4020      6610005 :            && ! reg_referenced_p (SET_DEST (XVECEXP (newpat, 0, 1)),
    4021              :                                   XVECEXP (newpat, 0, 0))
    4022      5513807 :            && ! reg_referenced_p (SET_DEST (XVECEXP (newpat, 0, 0)),
    4023      5513807 :                                   XVECEXP (newpat, 0, 1))
    4024     34576887 :            && ! (contains_muldiv (SET_SRC (XVECEXP (newpat, 0, 0)))
    4025       446665 :                  && contains_muldiv (SET_SRC (XVECEXP (newpat, 0, 1)))))
    4026              :     {
    4027      5131358 :       rtx set0 = XVECEXP (newpat, 0, 0);
    4028      5131358 :       rtx set1 = XVECEXP (newpat, 0, 1);
    4029              : 
    4030              :       /* Normally, it doesn't matter which of the two is done first, but
    4031              :          one which uses any regs/memory set or used in between i2 and i3
    4032              :          can't be first.  The PARALLEL might also have been pre-existing
    4033              :          in i3, so we need to make sure that we won't wrongly hoist a SET
    4034              :          to i2 that would conflict with a death note present in there, or
    4035              :          would have its dest modified or used between i2 and i3.  */
    4036      5131358 :       if ((set_noop_p (set1)
    4037      5131358 :            || (!modified_between_p (SET_SRC (set1), i2, i3)
    4038     10224869 :                && !(REG_P (SET_DEST (set1))
    4039      5100386 :                     && find_reg_note (i2, REG_DEAD, SET_DEST (set1)))
    4040      5148282 :                && !(GET_CODE (SET_DEST (set1)) == SUBREG
    4041        24097 :                     && find_reg_note (i2, REG_DEAD,
    4042        24097 :                                       SUBREG_REG (SET_DEST (set1))))
    4043      5124185 :                && !modified_between_p (SET_DEST (set1), i2, i3)
    4044      5124185 :                && !reg_used_between_p (SET_DEST (set1), i2, i3)))
    4045              :           /* If I3 is a jump, ensure that set0 is a jump so that
    4046              :              we do not create invalid RTL.  */
    4047     10255537 :           && (!JUMP_P (i3) || SET_DEST (set0) == pc_rtx))
    4048              :         {
    4049      5124179 :           newi2pat = set1;
    4050      5124179 :           newpat = set0;
    4051              :         }
    4052         7179 :       else if ((set_noop_p (set0)
    4053         7173 :                 || (!modified_between_p (SET_SRC (set0), i2, i3)
    4054          596 :                     && !(REG_P (SET_DEST (set0))
    4055          298 :                          && find_reg_note (i2, REG_DEAD, SET_DEST (set0)))
    4056          298 :                     && !(GET_CODE (SET_DEST (set0)) == SUBREG
    4057            0 :                          && find_reg_note (i2, REG_DEAD,
    4058            0 :                                            SUBREG_REG (SET_DEST (set0))))
    4059          298 :                     && !modified_between_p (SET_DEST (set0), i2, i3)
    4060          297 :                     && !reg_used_between_p (SET_DEST (set0), i2, i3)))
    4061              :                /* If I3 is a jump, ensure that set1 is a jump so that
    4062              :                   we do not create invalid RTL.  */
    4063         7476 :                && (!JUMP_P (i3) || SET_DEST (set1) == pc_rtx))
    4064              :         {
    4065          303 :           newi2pat = set0;
    4066          303 :           newpat = set1;
    4067              :         }
    4068              :       else
    4069              :         {
    4070         6876 :           undo_all ();
    4071         6876 :           return 0;
    4072              :         }
    4073              : 
    4074      5124482 :       i2_code_number = recog_for_combine (&newi2pat, i2, &new_i2_notes);
    4075              : 
    4076      5124482 :       if (i2_code_number >= 0)
    4077              :         {
    4078              :           /* recog_for_combine might have added CLOBBERs to newi2pat.
    4079              :              Make sure NEWPAT does not depend on the clobbered regs.  */
    4080      3769529 :           if (GET_CODE (newi2pat) == PARALLEL)
    4081              :             {
    4082      1379749 :               for (i = XVECLEN (newi2pat, 0) - 1; i >= 0; i--)
    4083       923905 :                 if (GET_CODE (XVECEXP (newi2pat, 0, i)) == CLOBBER)
    4084              :                   {
    4085       468061 :                     rtx reg = XEXP (XVECEXP (newi2pat, 0, i), 0);
    4086       468061 :                     if (reg_overlap_mentioned_p (reg, newpat))
    4087              :                       {
    4088         4634 :                         undo_all ();
    4089         4634 :                         return 0;
    4090              :                       }
    4091              :                   }
    4092              :             }
    4093              : 
    4094      3764895 :           insn_code_number = recog_for_combine (&newpat, i3, &new_i3_notes);
    4095              : 
    4096              :           /* Likewise, recog_for_combine might have added clobbers to NEWPAT.
    4097              :              Checking that the SET0's SET_DEST and SET1's SET_DEST aren't
    4098              :              mentioned/clobbered, ensures NEWI2PAT's SET_DEST is live.  */
    4099      3764895 :           if (insn_code_number >= 0 && GET_CODE (newpat) == PARALLEL)
    4100              :             {
    4101        94499 :               for (i = XVECLEN (newpat, 0) - 1; i >= 0; i--)
    4102        63006 :                 if (GET_CODE (XVECEXP (newpat, 0, i)) == CLOBBER)
    4103              :                   {
    4104        31513 :                     rtx reg = XEXP (XVECEXP (newpat, 0, i), 0);
    4105        31513 :                     if (reg_overlap_mentioned_p (reg, SET_DEST (set0))
    4106        31513 :                         || reg_overlap_mentioned_p (reg, SET_DEST (set1)))
    4107              :                       {
    4108            0 :                         undo_all ();
    4109            0 :                         return 0;
    4110              :                       }
    4111              :                   }
    4112              :             }
    4113              : 
    4114              :           if (insn_code_number >= 0)
    4115              :             split_i2i3 = true;
    4116              :         }
    4117              :     }
    4118              : 
    4119              :   /* If it still isn't recognized, fail and change things back the way they
    4120              :      were.  */
    4121     29769389 :   if ((insn_code_number < 0
    4122              :        /* Is the result a reasonable ASM_OPERANDS?  */
    4123     33368268 :        && (! check_asm_operands (newpat) || added_sets_1 || added_sets_2)))
    4124              :     {
    4125     29210621 :       undo_all ();
    4126     29210621 :       return 0;
    4127              :     }
    4128              : 
    4129              :   /* If we had to change another insn, make sure it is valid also.  */
    4130      4330120 :   if (undobuf.other_insn)
    4131              :     {
    4132       232253 :       CLEAR_HARD_REG_SET (newpat_used_regs);
    4133              : 
    4134       232253 :       other_pat = PATTERN (undobuf.other_insn);
    4135       232253 :       other_code_number = recog_for_combine (&other_pat, undobuf.other_insn,
    4136              :                                              &new_other_notes);
    4137              : 
    4138       232253 :       if (other_code_number < 0 && ! check_asm_operands (other_pat))
    4139              :         {
    4140         7225 :           undo_all ();
    4141         7225 :           return 0;
    4142              :         }
    4143              :     }
    4144              : 
    4145              :   /* Reject this combination if insn_cost reports that the replacement
    4146              :      instructions are more expensive than the originals.  */
    4147      4322895 :   if (!combine_validate_cost (i0, i1, i2, i3, newpat, newi2pat, other_pat,
    4148              :                               insn_code_number, i2_code_number,
    4149              :                               other_code_number))
    4150              :     {
    4151       217741 :       undo_all ();
    4152       217741 :       return 0;
    4153              :     }
    4154              : 
    4155      4105154 :   if (MAY_HAVE_DEBUG_BIND_INSNS)
    4156              :     {
    4157      2202038 :       struct undo *undo;
    4158              : 
    4159      6613731 :       for (undo = undobuf.undos; undo; undo = undo->next)
    4160      4411693 :         if (undo->kind == UNDO_MODE)
    4161              :           {
    4162         2885 :             rtx reg = regno_reg_rtx[undo->where.regno];
    4163         2885 :             machine_mode new_mode = GET_MODE (reg);
    4164         2885 :             machine_mode old_mode = undo->old_contents.m;
    4165              : 
    4166              :             /* Temporarily revert mode back.  */
    4167         2885 :             adjust_reg_mode (reg, old_mode);
    4168              : 
    4169         2885 :             if (reg == i2dest && i2scratch)
    4170              :               {
    4171              :                 /* If we used i2dest as a scratch register with a
    4172              :                    different mode, substitute it for the original
    4173              :                    i2src while its original mode is temporarily
    4174              :                    restored, and then clear i2scratch so that we don't
    4175              :                    do it again later.  */
    4176         2885 :                 propagate_for_debug (i2, last_combined_insn, reg, i2src,
    4177              :                                      this_basic_block);
    4178         2885 :                 i2scratch = false;
    4179              :                 /* Put back the new mode.  */
    4180         2885 :                 adjust_reg_mode (reg, new_mode);
    4181              :               }
    4182              :             else
    4183              :               {
    4184            0 :                 rtx tempreg = gen_raw_REG (old_mode, REGNO (reg));
    4185            0 :                 rtx_insn *first, *last;
    4186              : 
    4187            0 :                 if (reg == i2dest)
    4188              :                   {
    4189              :                     first = i2;
    4190              :                     last = last_combined_insn;
    4191              :                   }
    4192              :                 else
    4193              :                   {
    4194            0 :                     first = i3;
    4195            0 :                     last = undobuf.other_insn;
    4196            0 :                     gcc_assert (last);
    4197            0 :                     if (DF_INSN_LUID (last)
    4198            0 :                         < DF_INSN_LUID (last_combined_insn))
    4199            0 :                       last = last_combined_insn;
    4200              :                   }
    4201              : 
    4202              :                 /* We're dealing with a reg that changed mode but not
    4203              :                    meaning, so we want to turn it into a subreg for
    4204              :                    the new mode.  However, because of REG sharing and
    4205              :                    because its mode had already changed, we have to do
    4206              :                    it in two steps.  First, replace any debug uses of
    4207              :                    reg, with its original mode temporarily restored,
    4208              :                    with this copy we have created; then, replace the
    4209              :                    copy with the SUBREG of the original shared reg,
    4210              :                    once again changed to the new mode.  */
    4211            0 :                 propagate_for_debug (first, last, reg, tempreg,
    4212              :                                      this_basic_block);
    4213            0 :                 adjust_reg_mode (reg, new_mode);
    4214            0 :                 propagate_for_debug (first, last, tempreg,
    4215              :                                      lowpart_subreg (old_mode, reg, new_mode),
    4216              :                                      this_basic_block);
    4217              :               }
    4218              :           }
    4219              :     }
    4220              : 
    4221              :   /* If we will be able to accept this, we have made a
    4222              :      change to the destination of I3.  This requires us to
    4223              :      do a few adjustments.  */
    4224              : 
    4225      4105154 :   if (changed_i3_dest)
    4226              :     {
    4227        17339 :       PATTERN (i3) = newpat;
    4228        17339 :       adjust_for_new_dest (i3);
    4229              :     }
    4230              : 
    4231      4105154 :   bool only_i3_changed = !i0 && !i1 && rtx_equal_p (newi2pat, PATTERN (i2));
    4232              : 
    4233              :   /* If only i3 has changed, any split of the combined instruction just
    4234              :      restored i2 to its original state.  No destinations moved from i3
    4235              :      to i2.  */
    4236              :   if (only_i3_changed)
    4237              :     split_i2i3 = false;
    4238              : 
    4239              :   /* We now know that we can do this combination.  Merge the insns and
    4240              :      update the status of registers and LOG_LINKS.  */
    4241              : 
    4242      4105154 :   if (undobuf.other_insn)
    4243              :     {
    4244       224863 :       rtx note, next;
    4245              : 
    4246       224863 :       PATTERN (undobuf.other_insn) = other_pat;
    4247              : 
    4248              :       /* If any of the notes in OTHER_INSN were REG_DEAD or REG_UNUSED,
    4249              :          ensure that they are still valid.  Then add any non-duplicate
    4250              :          notes added by recog_for_combine.  */
    4251       671883 :       for (note = REG_NOTES (undobuf.other_insn); note; note = next)
    4252              :         {
    4253       447020 :           next = XEXP (note, 1);
    4254              : 
    4255       447020 :           if ((REG_NOTE_KIND (note) == REG_DEAD
    4256       228037 :                && !reg_referenced_p (XEXP (note, 0),
    4257       228037 :                                      PATTERN (undobuf.other_insn)))
    4258       442812 :               ||(REG_NOTE_KIND (note) == REG_UNUSED
    4259            5 :                  && !reg_set_p (XEXP (note, 0),
    4260            5 :                                 PATTERN (undobuf.other_insn)))
    4261              :               /* Simply drop equal note since it may be no longer valid
    4262              :                  for other_insn.  It may be possible to record that CC
    4263              :                  register is changed and only discard those notes, but
    4264              :                  in practice it's unnecessary complication and doesn't
    4265              :                  give any meaningful improvement.
    4266              : 
    4267              :                  See PR78559.  */
    4268       442812 :               || REG_NOTE_KIND (note) == REG_EQUAL
    4269       889683 :               || REG_NOTE_KIND (note) == REG_EQUIV)
    4270         4357 :             remove_note (undobuf.other_insn, note);
    4271              :         }
    4272              : 
    4273       224863 :       distribute_notes  (new_other_notes, undobuf.other_insn,
    4274              :                         undobuf.other_insn, NULL, NULL_RTX, NULL_RTX,
    4275              :                         NULL_RTX);
    4276              :     }
    4277              : 
    4278      4105154 :   if (swap_i2i3)
    4279              :     {
    4280              :       /* I3 now uses what used to be its destination and which is now
    4281              :          I2's destination.  This requires us to do a few adjustments.  */
    4282            0 :       PATTERN (i3) = newpat;
    4283            0 :       adjust_for_new_dest (i3);
    4284              :     }
    4285              : 
    4286      4105154 :   if (swap_i2i3 || split_i2i3)
    4287              :     {
    4288              :       /* We might need a LOG_LINK from I3 to I2.  But then we used to
    4289              :          have one, so we still will.
    4290              : 
    4291              :          However, some later insn might be using I2's dest and have
    4292              :          a LOG_LINK pointing at I3.  We should change it to point at
    4293              :          I2 instead.  */
    4294              : 
    4295              :       /* newi2pat is usually a SET here; however, recog_for_combine might
    4296              :          have added some clobbers.  */
    4297        27828 :       rtx x = newi2pat;
    4298        27828 :       if (GET_CODE (x) == PARALLEL)
    4299          546 :         x = XVECEXP (newi2pat, 0, 0);
    4300              : 
    4301        27828 :       if (REG_P (SET_DEST (x))
    4302            8 :           || (GET_CODE (SET_DEST (x)) == SUBREG
    4303            2 :               && REG_P (SUBREG_REG (SET_DEST (x)))))
    4304              :         {
    4305        27822 :           unsigned int regno = reg_or_subregno (SET_DEST (x));
    4306              : 
    4307        27822 :           bool done = false;
    4308       406302 :           for (rtx_insn *insn = NEXT_INSN (i3);
    4309       406302 :                !done
    4310       406302 :                && insn
    4311       404903 :                && INSN_P (insn)
    4312       784782 :                && BLOCK_FOR_INSN (insn) == this_basic_block;
    4313       378480 :                insn = NEXT_INSN (insn))
    4314              :             {
    4315       378480 :               if (DEBUG_INSN_P (insn))
    4316        69588 :                 continue;
    4317       308892 :               struct insn_link *link;
    4318       586413 :               FOR_EACH_LOG_LINK (link, insn)
    4319       277531 :                 if (link->insn == i3 && link->regno == regno)
    4320              :                   {
    4321           10 :                     link->insn = i2;
    4322           10 :                     done = true;
    4323           10 :                     break;
    4324              :                   }
    4325              :             }
    4326              :         }
    4327              :     }
    4328              : 
    4329      4105154 :   {
    4330      4105154 :     rtx i3notes, i2notes, i1notes = 0, i0notes = 0;
    4331      4105154 :     struct insn_link *i3links, *i2links, *i1links = 0, *i0links = 0;
    4332      4105154 :     rtx midnotes = 0;
    4333      4105154 :     int from_luid;
    4334              :     /* Compute which registers we expect to eliminate.  newi2pat may be setting
    4335              :        either i3dest or i2dest, so we must check it.  */
    4336       110792 :     rtx elim_i2 = ((newi2pat && reg_set_p (i2dest, newi2pat))
    4337      4005945 :                    || i2dest_in_i2src || i2dest_in_i1src || i2dest_in_i0src
    4338      3919088 :                    || !i2dest_killed
    4339      8023152 :                    ? 0 : i2dest);
    4340              :     /* For i1, we need to compute both local elimination and global
    4341              :        elimination information with respect to newi2pat because i1dest
    4342              :        may be the same as i3dest, in which case newi2pat may be setting
    4343              :        i1dest.  Global information is used when distributing REG_DEAD
    4344              :        note for i2 and i3, in which case it does matter if newi2pat sets
    4345              :        i1dest or not.
    4346              : 
    4347              :        Local information is used when distributing REG_DEAD note for i1,
    4348              :        in which case it doesn't matter if newi2pat sets i1dest or not.
    4349              :        See PR62151, if we have four insns combination:
    4350              :            i0: r0 <- i0src
    4351              :            i1: r1 <- i1src (using r0)
    4352              :                      REG_DEAD (r0)
    4353              :            i2: r0 <- i2src (using r1)
    4354              :            i3: r3 <- i3src (using r0)
    4355              :            ix: using r0
    4356              :        From i1's point of view, r0 is eliminated, no matter if it is set
    4357              :        by newi2pat or not.  In other words, REG_DEAD info for r0 in i1
    4358              :        should be discarded.
    4359              : 
    4360              :        Note local information only affects cases in forms like "I1->I2->I3",
    4361              :        "I0->I1->I2->I3" or "I0&I1->I2, I2->I3".  For other cases like
    4362              :        "I0->I1, I1&I2->I3" or "I1&I2->I3", newi2pat won't set i1dest or
    4363              :        i0dest anyway.  */
    4364       103494 :     rtx local_elim_i1 = (i1 == 0 || i1dest_in_i1src || i1dest_in_i0src
    4365       103427 :                          || !i1dest_killed
    4366      4105154 :                          ? 0 : i1dest);
    4367       103426 :     rtx elim_i1 = (local_elim_i1 == 0
    4368       103426 :                    || (newi2pat && reg_set_p (i1dest, newi2pat))
    4369       103426 :                    ? 0 : i1dest);
    4370              :     /* Same case as i1.  */
    4371         4711 :     rtx local_elim_i0 = (i0 == 0 || i0dest_in_i0src || !i0dest_killed
    4372      4105154 :                          ? 0 : i0dest);
    4373         4699 :     rtx elim_i0 = (local_elim_i0 == 0
    4374         4699 :                    || (newi2pat && reg_set_p (i0dest, newi2pat))
    4375         4699 :                    ? 0 : i0dest);
    4376              : 
    4377              :     /* Get the old REG_NOTES and LOG_LINKS from all our insns and
    4378              :        clear them.  */
    4379      4105154 :     i3notes = REG_NOTES (i3), i3links = LOG_LINKS (i3);
    4380      4105154 :     i2notes = REG_NOTES (i2), i2links = LOG_LINKS (i2);
    4381      4105154 :     if (i1)
    4382       103494 :       i1notes = REG_NOTES (i1), i1links = LOG_LINKS (i1);
    4383      4105154 :     if (i0)
    4384         4711 :       i0notes = REG_NOTES (i0), i0links = LOG_LINKS (i0);
    4385              : 
    4386              :     /* Ensure that we do not have something that should not be shared but
    4387              :        occurs multiple times in the new insns.  Check this by first
    4388              :        resetting all the `used' flags and then copying anything is shared.  */
    4389              : 
    4390      4105154 :     reset_used_flags (i3notes);
    4391      4105154 :     reset_used_flags (i2notes);
    4392      4105154 :     reset_used_flags (i1notes);
    4393      4105154 :     reset_used_flags (i0notes);
    4394      4105154 :     reset_used_flags (newpat);
    4395      4105154 :     reset_used_flags (newi2pat);
    4396      4105154 :     if (undobuf.other_insn)
    4397       224863 :       reset_used_flags (PATTERN (undobuf.other_insn));
    4398              : 
    4399      4105154 :     i3notes = copy_rtx_if_shared (i3notes);
    4400      4105154 :     i2notes = copy_rtx_if_shared (i2notes);
    4401      4105154 :     i1notes = copy_rtx_if_shared (i1notes);
    4402      4105154 :     i0notes = copy_rtx_if_shared (i0notes);
    4403      4105154 :     newpat = copy_rtx_if_shared (newpat);
    4404      4105154 :     newi2pat = copy_rtx_if_shared (newi2pat);
    4405      4105154 :     if (undobuf.other_insn)
    4406       224863 :       reset_used_flags (PATTERN (undobuf.other_insn));
    4407              : 
    4408      4105154 :     INSN_CODE (i3) = insn_code_number;
    4409      4105154 :     PATTERN (i3) = newpat;
    4410              : 
    4411      4105154 :     auto call = dyn_cast<rtx_call_insn *> (i3);
    4412        81210 :     if (call && CALL_INSN_FUNCTION_USAGE (call))
    4413              :       {
    4414       244384 :         for (rtx link = CALL_INSN_FUNCTION_USAGE (call); link;
    4415       165285 :              link = XEXP (link, 1))
    4416              :           {
    4417       165285 :             if (substed_i2)
    4418              :               {
    4419              :                 /* I2SRC must still be meaningful at this point.  Some
    4420              :                    splitting operations can invalidate I2SRC, but those
    4421              :                    operations do not apply to calls.  */
    4422       165285 :                 gcc_assert (i2src);
    4423       165285 :                 XEXP (link, 0) = simplify_replace_rtx (XEXP (link, 0),
    4424              :                                                        i2dest, i2src);
    4425              :               }
    4426       165285 :             if (substed_i1)
    4427            0 :               XEXP (link, 0) = simplify_replace_rtx (XEXP (link, 0),
    4428              :                                                      i1dest, i1src);
    4429       165285 :             if (substed_i0)
    4430            0 :               XEXP (link, 0) = simplify_replace_rtx (XEXP (link, 0),
    4431              :                                                      i0dest, i0src);
    4432              :           }
    4433              :       }
    4434              : 
    4435      4105154 :     if (undobuf.other_insn)
    4436       224863 :       INSN_CODE (undobuf.other_insn) = other_code_number;
    4437              : 
    4438              :     /* We had one special case above where I2 had more than one set and
    4439              :        we replaced a destination of one of those sets with the destination
    4440              :        of I3.  In that case, we have to update LOG_LINKS of insns later
    4441              :        in this basic block.  Note that this (expensive) case is rare.
    4442              : 
    4443              :        Also, in this case, we must pretend that all REG_NOTEs for I2
    4444              :        actually came from I3, so that REG_UNUSED notes from I2 will be
    4445              :        properly handled.  */
    4446              : 
    4447      4105154 :     if (i3_subst_into_i2)
    4448              :       {
    4449       206325 :         for (i = 0; i < XVECLEN (PATTERN (i2), 0); i++)
    4450       141721 :           if ((GET_CODE (XVECEXP (PATTERN (i2), 0, i)) == SET
    4451        65860 :                || GET_CODE (XVECEXP (PATTERN (i2), 0, i)) == CLOBBER)
    4452       140884 :               && REG_P (SET_DEST (XVECEXP (PATTERN (i2), 0, i)))
    4453       126063 :               && SET_DEST (XVECEXP (PATTERN (i2), 0, i)) != i2dest
    4454       267784 :               && ! find_reg_note (i2, REG_UNUSED,
    4455       126063 :                                   SET_DEST (XVECEXP (PATTERN (i2), 0, i))))
    4456     30880036 :             for (temp_insn = NEXT_INSN (i2);
    4457              :                  temp_insn
    4458     30880036 :                  && (this_basic_block->next_bb == EXIT_BLOCK_PTR_FOR_FN (cfun)
    4459     30632465 :                      || BB_HEAD (this_basic_block) != temp_insn);
    4460     30823196 :                  temp_insn = NEXT_INSN (temp_insn))
    4461     30823196 :               if (temp_insn != i3 && NONDEBUG_INSN_P (temp_insn))
    4462     19999534 :                 FOR_EACH_LOG_LINK (link, temp_insn)
    4463      7361263 :                   if (link->insn == i2)
    4464          440 :                     link->insn = i3;
    4465              : 
    4466        64604 :         if (i3notes)
    4467              :           {
    4468              :             rtx link = i3notes;
    4469        72152 :             while (XEXP (link, 1))
    4470              :               link = XEXP (link, 1);
    4471        64604 :             XEXP (link, 1) = i2notes;
    4472              :           }
    4473              :         else
    4474              :           i3notes = i2notes;
    4475              :         i2notes = 0;
    4476              :       }
    4477              : 
    4478      4105154 :     LOG_LINKS (i3) = NULL;
    4479      4105154 :     REG_NOTES (i3) = 0;
    4480      4105154 :     LOG_LINKS (i2) = NULL;
    4481      4105154 :     REG_NOTES (i2) = 0;
    4482              : 
    4483      4105154 :     if (newi2pat)
    4484              :       {
    4485       110792 :         if (MAY_HAVE_DEBUG_BIND_INSNS && i2scratch)
    4486        10483 :           propagate_for_debug (i2, last_combined_insn, i2dest, i2src,
    4487              :                                this_basic_block);
    4488       110792 :         INSN_CODE (i2) = i2_code_number;
    4489       110792 :         PATTERN (i2) = newi2pat;
    4490              :       }
    4491              :     else
    4492              :       {
    4493      3994362 :         if (MAY_HAVE_DEBUG_BIND_INSNS && i2src)
    4494      2134941 :           propagate_for_debug (i2, last_combined_insn, i2dest, i2src,
    4495              :                                this_basic_block);
    4496      3994362 :         SET_INSN_DELETED (i2);
    4497              :       }
    4498              : 
    4499      4105154 :     if (i1)
    4500              :       {
    4501       103494 :         LOG_LINKS (i1) = NULL;
    4502       103494 :         REG_NOTES (i1) = 0;
    4503       103494 :         if (MAY_HAVE_DEBUG_BIND_INSNS)
    4504        53757 :           propagate_for_debug (i1, last_combined_insn, i1dest, i1src,
    4505              :                                this_basic_block);
    4506       103494 :         SET_INSN_DELETED (i1);
    4507              :       }
    4508              : 
    4509      4105154 :     if (i0)
    4510              :       {
    4511         4711 :         LOG_LINKS (i0) = NULL;
    4512         4711 :         REG_NOTES (i0) = 0;
    4513         4711 :         if (MAY_HAVE_DEBUG_BIND_INSNS)
    4514         2983 :           propagate_for_debug (i0, last_combined_insn, i0dest, i0src,
    4515              :                                this_basic_block);
    4516         4711 :         SET_INSN_DELETED (i0);
    4517              :       }
    4518              : 
    4519              :     /* Get death notes for everything that is now used in either I3 or
    4520              :        I2 and used to die in a previous insn.  If we built two new
    4521              :        patterns, move from I1 to I2 then I2 to I3 so that we get the
    4522              :        proper movement on registers that I2 modifies.  */
    4523              : 
    4524         4711 :     if (i0)
    4525         4711 :       from_luid = DF_INSN_LUID (i0);
    4526      4100443 :     else if (i1)
    4527        98783 :       from_luid = DF_INSN_LUID (i1);
    4528              :     else
    4529      4001660 :       from_luid = DF_INSN_LUID (i2);
    4530      4105154 :     if (newi2pat)
    4531       110792 :       move_deaths (newi2pat, NULL_RTX, from_luid, i2, &midnotes);
    4532      4105154 :     move_deaths (newpat, newi2pat, from_luid, i3, &midnotes);
    4533              : 
    4534              :     /* Distribute all the LOG_LINKS and REG_NOTES from I1, I2, and I3.  */
    4535      4105154 :     if (i3notes)
    4536      7416291 :       distribute_notes (i3notes, i3, i3, newi2pat ? i2 : NULL,
    4537              :                         elim_i2, elim_i1, elim_i0);
    4538      4105154 :     if (i2notes)
    4539      5728901 :       distribute_notes (i2notes, i2, i3, newi2pat ? i2 : NULL,
    4540              :                         elim_i2, elim_i1, elim_i0);
    4541      4105154 :     if (i1notes)
    4542        60718 :       distribute_notes (i1notes, i1, i3, newi2pat ? i2 : NULL,
    4543              :                         elim_i2, local_elim_i1, local_elim_i0);
    4544      4105154 :     if (i0notes)
    4545         3983 :       distribute_notes (i0notes, i0, i3, newi2pat ? i2 : NULL,
    4546              :                         elim_i2, elim_i1, local_elim_i0);
    4547      4105154 :     if (midnotes)
    4548      4899827 :       distribute_notes (midnotes, NULL, i3, newi2pat ? i2 : NULL,
    4549              :                         elim_i2, elim_i1, elim_i0);
    4550              : 
    4551              :     /* Distribute any notes added to I2 or I3 by recog_for_combine.  We
    4552              :        know these are REG_UNUSED and want them to go to the desired insn,
    4553              :        so we always pass it as i3.  */
    4554              : 
    4555      4105154 :     if (newi2pat && new_i2_notes)
    4556        47996 :       distribute_notes (new_i2_notes, i2, i2, NULL, NULL_RTX, NULL_RTX,
    4557              :                         NULL_RTX);
    4558              : 
    4559      4105154 :     if (new_i3_notes)
    4560       166321 :       distribute_notes (new_i3_notes, i3, i3, NULL, NULL_RTX, NULL_RTX,
    4561              :                         NULL_RTX);
    4562              : 
    4563              :     /* If I3DEST was used in I3SRC, it really died in I3.  We may need to
    4564              :        put a REG_DEAD note for it somewhere.  If NEWI2PAT exists and sets
    4565              :        I3DEST, the death must be somewhere before I2, not I3.  If we passed I3
    4566              :        in that case, it might delete I2.  Similarly for I2 and I1.
    4567              :        Show an additional death due to the REG_DEAD note we make here.  If
    4568              :        we discard it in distribute_notes, we will decrement it again.  */
    4569              : 
    4570      4105154 :     if (i3dest_killed)
    4571              :       {
    4572       341649 :         rtx new_note = alloc_reg_note (REG_DEAD, i3dest_killed, NULL_RTX);
    4573       341649 :         if (newi2pat && reg_set_p (i3dest_killed, newi2pat))
    4574          678 :           distribute_notes (new_note, NULL, i2, NULL, elim_i2,
    4575              :                             elim_i1, elim_i0);
    4576              :         else
    4577       680006 :           distribute_notes (new_note, NULL, i3, newi2pat ? i2 : NULL,
    4578              :                             elim_i2, elim_i1, elim_i0);
    4579              :       }
    4580              : 
    4581      4105154 :     if (i2dest_in_i2src)
    4582              :       {
    4583        85651 :         rtx new_note = alloc_reg_note (REG_DEAD, i2dest, NULL_RTX);
    4584        85651 :         if (newi2pat && reg_set_p (i2dest, newi2pat))
    4585          853 :           distribute_notes (new_note,  NULL, i2, NULL, NULL_RTX,
    4586              :                             NULL_RTX, NULL_RTX);
    4587              :         else
    4588       169558 :           distribute_notes (new_note, NULL, i3, newi2pat ? i2 : NULL,
    4589              :                             NULL_RTX, NULL_RTX, NULL_RTX);
    4590              :       }
    4591              : 
    4592      4105154 :     if (i1dest_in_i1src)
    4593              :       {
    4594           65 :         rtx new_note = alloc_reg_note (REG_DEAD, i1dest, NULL_RTX);
    4595           65 :         if (newi2pat && reg_set_p (i1dest, newi2pat))
    4596            5 :           distribute_notes (new_note, NULL, i2, NULL, NULL_RTX,
    4597              :                             NULL_RTX, NULL_RTX);
    4598              :         else
    4599          102 :           distribute_notes (new_note, NULL, i3, newi2pat ? i2 : NULL,
    4600              :                             NULL_RTX, NULL_RTX, NULL_RTX);
    4601              :       }
    4602              : 
    4603      4105154 :     if (i0dest_in_i0src)
    4604              :       {
    4605           12 :         rtx new_note = alloc_reg_note (REG_DEAD, i0dest, NULL_RTX);
    4606           12 :         if (newi2pat && reg_set_p (i0dest, newi2pat))
    4607            0 :           distribute_notes (new_note, NULL, i2, NULL, NULL_RTX,
    4608              :                             NULL_RTX, NULL_RTX);
    4609              :         else
    4610           24 :           distribute_notes (new_note, NULL, i3, newi2pat ? i2 : NULL,
    4611              :                             NULL_RTX, NULL_RTX, NULL_RTX);
    4612              :       }
    4613              : 
    4614      4105154 :     if (only_i3_changed)
    4615        40696 :       distribute_links (i3links, i3, param_max_combine_search_insns);
    4616              :     else
    4617              :       {
    4618      4064458 :         distribute_links (i3links);
    4619      4064458 :         distribute_links (i2links, i2);
    4620      4064458 :         distribute_links (i1links);
    4621      4064458 :         distribute_links (i0links);
    4622              :       }
    4623              : 
    4624      4105154 :     if (REG_P (i2dest))
    4625              :       {
    4626      4105154 :         struct insn_link *link;
    4627      4105154 :         rtx_insn *i2_insn = 0;
    4628      4105154 :         rtx i2_val = 0, set;
    4629              : 
    4630              :         /* The insn that used to set this register doesn't exist, and
    4631              :            this life of the register may not exist either.  See if one of
    4632              :            I3's links points to an insn that sets I2DEST.  If it does,
    4633              :            that is now the last known value for I2DEST. If we don't update
    4634              :            this and I2 set the register to a value that depended on its old
    4635              :            contents, we will get confused.  If this insn is used, thing
    4636              :            will be set correctly in combine_instructions.  */
    4637      7553535 :         FOR_EACH_LOG_LINK (link, i3)
    4638      3448381 :           if ((set = single_set (link->insn)) != 0
    4639      3448381 :               && rtx_equal_p (i2dest, SET_DEST (set)))
    4640        48050 :             i2_insn = link->insn, i2_val = SET_SRC (set);
    4641              : 
    4642      4105154 :         record_value_for_reg (i2dest, i2_insn, i2_val);
    4643              : 
    4644              :         /* If the reg formerly set in I2 died only once and that was in I3,
    4645              :            zero its use count so it won't make `reload' do any work.  */
    4646      4105154 :         if (! added_sets_2
    4647      3969401 :             && (newi2pat == 0 || ! reg_mentioned_p (i2dest, newi2pat))
    4648      3930634 :             && ! i2dest_in_i2src
    4649      7970962 :             && REGNO (i2dest) < reg_n_sets_max)
    4650      3865806 :           INC_REG_N_SETS (REGNO (i2dest), -1);
    4651              :       }
    4652              : 
    4653      4105154 :     if (i1 && REG_P (i1dest))
    4654              :       {
    4655       103494 :         struct insn_link *link;
    4656       103494 :         rtx_insn *i1_insn = 0;
    4657       103494 :         rtx i1_val = 0, set;
    4658              : 
    4659       179897 :         FOR_EACH_LOG_LINK (link, i3)
    4660        76403 :           if ((set = single_set (link->insn)) != 0
    4661        76403 :               && rtx_equal_p (i1dest, SET_DEST (set)))
    4662          495 :             i1_insn = link->insn, i1_val = SET_SRC (set);
    4663              : 
    4664       103494 :         record_value_for_reg (i1dest, i1_insn, i1_val);
    4665              : 
    4666       103494 :         if (! added_sets_1
    4667              :             && ! i1dest_in_i1src
    4668       103494 :             && REGNO (i1dest) < reg_n_sets_max)
    4669        97177 :           INC_REG_N_SETS (REGNO (i1dest), -1);
    4670              :       }
    4671              : 
    4672      4105154 :     if (i0 && REG_P (i0dest))
    4673              :       {
    4674         4711 :         struct insn_link *link;
    4675         4711 :         rtx_insn *i0_insn = 0;
    4676         4711 :         rtx i0_val = 0, set;
    4677              : 
    4678         7279 :         FOR_EACH_LOG_LINK (link, i3)
    4679         2568 :           if ((set = single_set (link->insn)) != 0
    4680         2568 :               && rtx_equal_p (i0dest, SET_DEST (set)))
    4681            0 :             i0_insn = link->insn, i0_val = SET_SRC (set);
    4682              : 
    4683         4711 :         record_value_for_reg (i0dest, i0_insn, i0_val);
    4684              : 
    4685         4711 :         if (! added_sets_0
    4686              :             && ! i0dest_in_i0src
    4687         4711 :             && REGNO (i0dest) < reg_n_sets_max)
    4688         4650 :           INC_REG_N_SETS (REGNO (i0dest), -1);
    4689              :       }
    4690              : 
    4691              :     /* Update reg_stat[].nonzero_bits et al for any changes that may have
    4692              :        been made to this insn.  The order is important, because newi2pat
    4693              :        can affect nonzero_bits of newpat.  */
    4694      4105154 :     if (newi2pat)
    4695       110792 :       note_pattern_stores (newi2pat, set_nonzero_bits_and_sign_copies, NULL);
    4696      4105154 :     note_pattern_stores (newpat, set_nonzero_bits_and_sign_copies, NULL);
    4697              :   }
    4698              : 
    4699      4105154 :   if (undobuf.other_insn != NULL_RTX)
    4700              :     {
    4701       224863 :       if (dump_file)
    4702              :         {
    4703           12 :           fprintf (dump_file, "modifying other_insn ");
    4704           12 :           dump_insn_slim (dump_file, undobuf.other_insn);
    4705              :         }
    4706       224863 :       df_insn_rescan (undobuf.other_insn);
    4707              :     }
    4708              : 
    4709      4105154 :   if (i0 && !(NOTE_P (i0) && (NOTE_KIND (i0) == NOTE_INSN_DELETED)))
    4710              :     {
    4711            0 :       if (dump_file)
    4712              :         {
    4713            0 :           fprintf (dump_file, "modifying insn i0 ");
    4714            0 :           dump_insn_slim (dump_file, i0);
    4715              :         }
    4716            0 :       df_insn_rescan (i0);
    4717              :     }
    4718              : 
    4719      4105154 :   if (i1 && !(NOTE_P (i1) && (NOTE_KIND (i1) == NOTE_INSN_DELETED)))
    4720              :     {
    4721            0 :       if (dump_file)
    4722              :         {
    4723            0 :           fprintf (dump_file, "modifying insn i1 ");
    4724            0 :           dump_insn_slim (dump_file, i1);
    4725              :         }
    4726            0 :       df_insn_rescan (i1);
    4727              :     }
    4728              : 
    4729      4105154 :   if (i2 && !(NOTE_P (i2) && (NOTE_KIND (i2) == NOTE_INSN_DELETED)))
    4730              :     {
    4731       110792 :       if (dump_file)
    4732              :         {
    4733           19 :           fprintf (dump_file, "modifying insn i2 ");
    4734           19 :           dump_insn_slim (dump_file, i2);
    4735              :         }
    4736       110792 :       df_insn_rescan (i2);
    4737              :     }
    4738              : 
    4739      4105154 :   if (i3 && !(NOTE_P (i3) && (NOTE_KIND (i3) == NOTE_INSN_DELETED)))
    4740              :     {
    4741      4105154 :       if (dump_file)
    4742              :         {
    4743          244 :           fprintf (dump_file, "modifying insn i3 ");
    4744          244 :           dump_insn_slim (dump_file, i3);
    4745              :         }
    4746      4105154 :       df_insn_rescan (i3);
    4747              :     }
    4748              : 
    4749              :   /* Set new_direct_jump_p if a new return or simple jump instruction
    4750              :      has been created.  Adjust the CFG accordingly.  */
    4751      4105154 :   if (returnjump_p (i3) || any_uncondjump_p (i3))
    4752              :     {
    4753          201 :       *new_direct_jump_p = 1;
    4754          201 :       mark_jump_label (PATTERN (i3), i3, 0);
    4755          201 :       update_cfg_for_uncondjump (i3);
    4756              :     }
    4757              : 
    4758      4105154 :   if (undobuf.other_insn != NULL_RTX
    4759      4105154 :       && (returnjump_p (undobuf.other_insn)
    4760       224863 :           || any_uncondjump_p (undobuf.other_insn)))
    4761              :     {
    4762         1963 :       *new_direct_jump_p = 1;
    4763         1963 :       update_cfg_for_uncondjump (undobuf.other_insn);
    4764              :     }
    4765              : 
    4766      4105154 :   if (GET_CODE (PATTERN (i3)) == TRAP_IF
    4767      4105154 :       && XEXP (PATTERN (i3), 0) == const1_rtx)
    4768              :     {
    4769            0 :       basic_block bb = BLOCK_FOR_INSN (i3);
    4770            0 :       gcc_assert (bb);
    4771            0 :       remove_edge (split_block (bb, i3));
    4772            0 :       emit_barrier_after_bb (bb);
    4773            0 :       *new_direct_jump_p = 1;
    4774              :     }
    4775              : 
    4776      4105154 :   if (undobuf.other_insn
    4777       224863 :       && GET_CODE (PATTERN (undobuf.other_insn)) == TRAP_IF
    4778      4105154 :       && XEXP (PATTERN (undobuf.other_insn), 0) == const1_rtx)
    4779              :     {
    4780            0 :       basic_block bb = BLOCK_FOR_INSN (undobuf.other_insn);
    4781            0 :       gcc_assert (bb);
    4782            0 :       remove_edge (split_block (bb, undobuf.other_insn));
    4783            0 :       emit_barrier_after_bb (bb);
    4784            0 :       *new_direct_jump_p = 1;
    4785              :     }
    4786              : 
    4787              :   /* A noop might also need cleaning up of CFG, if it comes from the
    4788              :      simplification of a jump.  */
    4789      4105154 :   if (JUMP_P (i3)
    4790        47056 :       && GET_CODE (newpat) == SET
    4791        35247 :       && SET_SRC (newpat) == pc_rtx
    4792          383 :       && SET_DEST (newpat) == pc_rtx)
    4793              :     {
    4794          383 :       *new_direct_jump_p = 1;
    4795          383 :       update_cfg_for_uncondjump (i3);
    4796              :     }
    4797              : 
    4798      4105154 :   if (undobuf.other_insn != NULL_RTX
    4799       224863 :       && JUMP_P (undobuf.other_insn)
    4800       218900 :       && GET_CODE (PATTERN (undobuf.other_insn)) == SET
    4801       218900 :       && SET_SRC (PATTERN (undobuf.other_insn)) == pc_rtx
    4802      4106901 :       && SET_DEST (PATTERN (undobuf.other_insn)) == pc_rtx)
    4803              :     {
    4804         1747 :       *new_direct_jump_p = 1;
    4805         1747 :       update_cfg_for_uncondjump (undobuf.other_insn);
    4806              :     }
    4807              : 
    4808      4105154 :   combine_successes++;
    4809      4105154 :   undo_commit ();
    4810              : 
    4811      4105154 :   if (only_i3_changed)
    4812              :     return i3;
    4813              : 
    4814      4064458 :   rtx_insn *ret = newi2pat ? i2 : i3;
    4815      4064458 :   if (added_links_insn && DF_INSN_LUID (added_links_insn) < DF_INSN_LUID (ret))
    4816              :     ret = added_links_insn;
    4817      4064458 :   if (added_notes_insn && DF_INSN_LUID (added_notes_insn) < DF_INSN_LUID (ret))
    4818              :     ret = added_notes_insn;
    4819              : 
    4820              :   return ret;
    4821              : }
    4822              : 
    4823              : /* Get a marker for undoing to the current state.  */
    4824              : 
    4825              : static void *
    4826     38282428 : get_undo_marker (void)
    4827              : {
    4828     38282428 :   return undobuf.undos;
    4829              : }
    4830              : 
    4831              : /* Undo the modifications up to the marker.  */
    4832              : 
    4833              : static void
    4834     45188645 : undo_to_marker (void *marker)
    4835              : {
    4836     45188645 :   struct undo *undo, *next;
    4837              : 
    4838    142254981 :   for (undo = undobuf.undos; undo != marker; undo = next)
    4839              :     {
    4840     97066336 :       gcc_assert (undo);
    4841              : 
    4842     97066336 :       next = undo->next;
    4843     97066336 :       switch (undo->kind)
    4844              :         {
    4845     89696873 :         case UNDO_RTX:
    4846     89696873 :           *undo->where.r = undo->old_contents.r;
    4847     89696873 :           break;
    4848      6737648 :         case UNDO_INT:
    4849      6737648 :           *undo->where.i = undo->old_contents.i;
    4850      6737648 :           break;
    4851       560247 :         case UNDO_MODE:
    4852       560247 :           adjust_reg_mode (regno_reg_rtx[undo->where.regno],
    4853              :                            undo->old_contents.m);
    4854       560247 :           break;
    4855        71568 :         case UNDO_LINKS:
    4856        71568 :           *undo->where.l = undo->old_contents.l;
    4857        71568 :           break;
    4858            0 :         default:
    4859            0 :           gcc_unreachable ();
    4860              :         }
    4861              : 
    4862     97066336 :       undo->next = undobuf.frees;
    4863     97066336 :       undobuf.frees = undo;
    4864              :     }
    4865              : 
    4866     45188645 :   undobuf.undos = (struct undo *) marker;
    4867     45188645 : }
    4868              : 
    4869              : /* Undo all the modifications recorded in undobuf.  */
    4870              : 
    4871              : static void
    4872     44051619 : undo_all (void)
    4873              : {
    4874     44051619 :   undo_to_marker (0);
    4875            0 : }
    4876              : 
    4877              : /* We've committed to accepting the changes we made.  Move all
    4878              :    of the undos to the free list.  */
    4879              : 
    4880              : static void
    4881      4105154 : undo_commit (void)
    4882              : {
    4883      4105154 :   struct undo *undo, *next;
    4884              : 
    4885     12124490 :   for (undo = undobuf.undos; undo; undo = next)
    4886              :     {
    4887      8019336 :       next = undo->next;
    4888      8019336 :       undo->next = undobuf.frees;
    4889      8019336 :       undobuf.frees = undo;
    4890              :     }
    4891      4105154 :   undobuf.undos = 0;
    4892      4105154 : }
    4893              : 
    4894              : /* Find the innermost point within the rtx at LOC, possibly LOC itself,
    4895              :    where we have an arithmetic expression and return that point.  LOC will
    4896              :    be inside INSN.
    4897              : 
    4898              :    try_combine will call this function to see if an insn can be split into
    4899              :    two insns.  */
    4900              : 
    4901              : static rtx *
    4902     31734618 : find_split_point (rtx *loc, rtx_insn *insn, bool set_src)
    4903              : {
    4904     32778548 :   rtx x = *loc;
    4905     32778548 :   enum rtx_code code = GET_CODE (x);
    4906     32778548 :   rtx *split;
    4907     32778548 :   unsigned HOST_WIDE_INT len = 0;
    4908     32778548 :   HOST_WIDE_INT pos = 0;
    4909     32778548 :   bool unsignedp = false;
    4910     32778548 :   rtx inner = NULL_RTX;
    4911     32778548 :   scalar_int_mode mode, inner_mode;
    4912              : 
    4913              :   /* First special-case some codes.  */
    4914     32778548 :   switch (code)
    4915              :     {
    4916      1048963 :     case SUBREG:
    4917              : #ifdef INSN_SCHEDULING
    4918              :       /* If we are making a paradoxical SUBREG invalid, it becomes a split
    4919              :          point.  */
    4920      1048963 :       if (MEM_P (SUBREG_REG (x)))
    4921              :         return loc;
    4922              : #endif
    4923      1036841 :       return find_split_point (&SUBREG_REG (x), insn, false);
    4924              : 
    4925      1546405 :     case MEM:
    4926              :       /* If we have (mem (const ..)) or (mem (symbol_ref ...)), split it
    4927              :          using LO_SUM and HIGH.  */
    4928      1546405 :       if (HAVE_lo_sum && (GET_CODE (XEXP (x, 0)) == CONST
    4929              :                           || GET_CODE (XEXP (x, 0)) == SYMBOL_REF))
    4930              :         {
    4931              :           machine_mode address_mode = get_address_mode (x);
    4932              : 
    4933              :           SUBST (XEXP (x, 0),
    4934              :                  gen_rtx_LO_SUM (address_mode,
    4935              :                                  gen_rtx_HIGH (address_mode, XEXP (x, 0)),
    4936              :                                  XEXP (x, 0)));
    4937              :           return &XEXP (XEXP (x, 0), 0);
    4938              :         }
    4939              : 
    4940              :       /* If we have a PLUS whose second operand is a constant and the
    4941              :          address is not valid, perhaps we can split it up using
    4942              :          the machine-specific way to split large constants.  We use
    4943              :          the first pseudo-reg (one of the virtual regs) as a placeholder;
    4944              :          it will not remain in the result.  */
    4945      1546405 :       if (GET_CODE (XEXP (x, 0)) == PLUS
    4946      1025864 :           && CONST_INT_P (XEXP (XEXP (x, 0), 1))
    4947      3321256 :           && ! memory_address_addr_space_p (GET_MODE (x), XEXP (x, 0),
    4948       748987 :                                             MEM_ADDR_SPACE (x)))
    4949              :         {
    4950       116616 :           rtx reg = regno_reg_rtx[FIRST_PSEUDO_REGISTER];
    4951       116616 :           unsigned int old_nregs, new_nregs;
    4952       116616 :           rtx_insn *seq = combine_split_insns (gen_rtx_SET (reg, XEXP (x, 0)),
    4953              :                                                subst_insn, &old_nregs, &new_nregs);
    4954              : 
    4955              :           /* This should have produced two insns, each of which sets our
    4956              :              placeholder.  If the source of the second is a valid address,
    4957              :              we can put both sources together and make a split point
    4958              :              in the middle.  */
    4959              : 
    4960       116616 :           if (seq
    4961           56 :               && NEXT_INSN (seq) != NULL_RTX
    4962            0 :               && NEXT_INSN (NEXT_INSN (seq)) == NULL_RTX
    4963            0 :               && NONJUMP_INSN_P (seq)
    4964            0 :               && GET_CODE (PATTERN (seq)) == SET
    4965            0 :               && SET_DEST (PATTERN (seq)) == reg
    4966            0 :               && ! reg_mentioned_p (reg,
    4967            0 :                                     SET_SRC (PATTERN (seq)))
    4968            0 :               && NONJUMP_INSN_P (NEXT_INSN (seq))
    4969            0 :               && GET_CODE (PATTERN (NEXT_INSN (seq))) == SET
    4970            0 :               && SET_DEST (PATTERN (NEXT_INSN (seq))) == reg
    4971       116616 :               && memory_address_addr_space_p
    4972       116616 :                    (GET_MODE (x), SET_SRC (PATTERN (NEXT_INSN (seq))),
    4973            0 :                     MEM_ADDR_SPACE (x)))
    4974              :             {
    4975            0 :               rtx src1 = SET_SRC (PATTERN (seq));
    4976            0 :               rtx src2 = SET_SRC (PATTERN (NEXT_INSN (seq)));
    4977              : 
    4978              :               /* Replace the placeholder in SRC2 with SRC1.  If we can
    4979              :                  find where in SRC2 it was placed, that can become our
    4980              :                  split point and we can replace this address with SRC2.
    4981              :                  Just try two obvious places.  */
    4982              : 
    4983            0 :               src2 = replace_rtx (src2, reg, src1);
    4984            0 :               split = 0;
    4985            0 :               if (XEXP (src2, 0) == src1)
    4986            0 :                 split = &XEXP (src2, 0);
    4987            0 :               else if (GET_RTX_FORMAT (GET_CODE (XEXP (src2, 0)))[0] == 'e'
    4988            0 :                        && XEXP (XEXP (src2, 0), 0) == src1)
    4989            0 :                 split = &XEXP (XEXP (src2, 0), 0);
    4990              : 
    4991            0 :               if (split)
    4992              :                 {
    4993            0 :                   SUBST (XEXP (x, 0), src2);
    4994        90432 :                   return split;
    4995              :                 }
    4996              :             }
    4997              : 
    4998              :           /* If that didn't work and we have a nested plus, like:
    4999              :              ((REG1 * CONST1) + REG2) + CONST2 and (REG1 + REG2) + CONST2
    5000              :              is valid address, try to split (REG1 * CONST1).  */
    5001       116616 :           if (GET_CODE (XEXP (XEXP (x, 0), 0)) == PLUS
    5002        75694 :               && !OBJECT_P (XEXP (XEXP (XEXP (x, 0), 0), 0))
    5003        61153 :               && OBJECT_P (XEXP (XEXP (XEXP (x, 0), 0), 1))
    5004        61142 :               && ! (GET_CODE (XEXP (XEXP (XEXP (x, 0), 0), 0)) == SUBREG
    5005           10 :                     && OBJECT_P (SUBREG_REG (XEXP (XEXP (XEXP (x, 0),
    5006              :                                                          0), 0)))))
    5007              :             {
    5008        61142 :               rtx tem = XEXP (XEXP (XEXP (x, 0), 0), 0);
    5009        61142 :               XEXP (XEXP (XEXP (x, 0), 0), 0) = reg;
    5010       122284 :               if (memory_address_addr_space_p (GET_MODE (x), XEXP (x, 0),
    5011        61142 :                                                MEM_ADDR_SPACE (x)))
    5012              :                 {
    5013        49635 :                   XEXP (XEXP (XEXP (x, 0), 0), 0) = tem;
    5014        49635 :                   return &XEXP (XEXP (XEXP (x, 0), 0), 0);
    5015              :                 }
    5016        11507 :               XEXP (XEXP (XEXP (x, 0), 0), 0) = tem;
    5017        11507 :             }
    5018        55474 :           else if (GET_CODE (XEXP (XEXP (x, 0), 0)) == PLUS
    5019        14552 :                    && OBJECT_P (XEXP (XEXP (XEXP (x, 0), 0), 0))
    5020        14541 :                    && !OBJECT_P (XEXP (XEXP (XEXP (x, 0), 0), 1))
    5021          294 :                    && ! (GET_CODE (XEXP (XEXP (XEXP (x, 0), 0), 1)) == SUBREG
    5022          294 :                          && OBJECT_P (SUBREG_REG (XEXP (XEXP (XEXP (x, 0),
    5023              :                                                               0), 1)))))
    5024              :             {
    5025            0 :               rtx tem = XEXP (XEXP (XEXP (x, 0), 0), 1);
    5026            0 :               XEXP (XEXP (XEXP (x, 0), 0), 1) = reg;
    5027            0 :               if (memory_address_addr_space_p (GET_MODE (x), XEXP (x, 0),
    5028            0 :                                                MEM_ADDR_SPACE (x)))
    5029              :                 {
    5030            0 :                   XEXP (XEXP (XEXP (x, 0), 0), 1) = tem;
    5031            0 :                   return &XEXP (XEXP (XEXP (x, 0), 0), 1);
    5032              :                 }
    5033            0 :               XEXP (XEXP (XEXP (x, 0), 0), 1) = tem;
    5034              :             }
    5035              : 
    5036              :           /* If that didn't work, perhaps the first operand is complex and
    5037              :              needs to be computed separately, so make a split point there.
    5038              :              This will occur on machines that just support REG + CONST
    5039              :              and have a constant moved through some previous computation.  */
    5040        66981 :           if (!OBJECT_P (XEXP (XEXP (x, 0), 0))
    5041        40797 :               && ! (GET_CODE (XEXP (XEXP (x, 0), 0)) == SUBREG
    5042            0 :                     && OBJECT_P (SUBREG_REG (XEXP (XEXP (x, 0), 0)))))
    5043        40797 :             return &XEXP (XEXP (x, 0), 0);
    5044              :         }
    5045              : 
    5046              :       /* If we have a PLUS whose first operand is complex, try computing it
    5047              :          separately by making a split there.  */
    5048      1455973 :       if (GET_CODE (XEXP (x, 0)) == PLUS
    5049      2557674 :           && ! memory_address_addr_space_p (GET_MODE (x), XEXP (x, 0),
    5050       935432 :                                             MEM_ADDR_SPACE (x))
    5051       166269 :           && ! OBJECT_P (XEXP (XEXP (x, 0), 0))
    5052      1567618 :           && ! (GET_CODE (XEXP (XEXP (x, 0), 0)) == SUBREG
    5053          606 :                 && OBJECT_P (SUBREG_REG (XEXP (XEXP (x, 0), 0)))))
    5054       111641 :         return &XEXP (XEXP (x, 0), 0);
    5055              :       break;
    5056              : 
    5057      4814287 :     case SET:
    5058              :       /* See if we can split SET_SRC as it stands.  */
    5059      4814287 :       split = find_split_point (&SET_SRC (x), insn, true);
    5060      4814287 :       if (split && split != &SET_SRC (x))
    5061              :         return split;
    5062              : 
    5063              :       /* See if we can split SET_DEST as it stands.  */
    5064       513658 :       split = find_split_point (&SET_DEST (x), insn, false);
    5065       513658 :       if (split && split != &SET_DEST (x))
    5066              :         return split;
    5067              : 
    5068              :       /* See if this is a bitfield assignment with everything constant.  If
    5069              :          so, this is an IOR of an AND, so split it into that.  */
    5070       483028 :       if (GET_CODE (SET_DEST (x)) == ZERO_EXTRACT
    5071         4117 :           && is_a <scalar_int_mode> (GET_MODE (XEXP (SET_DEST (x), 0)),
    5072              :                                      &inner_mode)
    5073         4117 :           && HWI_COMPUTABLE_MODE_P (inner_mode)
    5074         4117 :           && CONST_INT_P (XEXP (SET_DEST (x), 1))
    5075         4117 :           && CONST_INT_P (XEXP (SET_DEST (x), 2))
    5076         3955 :           && CONST_INT_P (SET_SRC (x))
    5077          418 :           && ((INTVAL (XEXP (SET_DEST (x), 1))
    5078          418 :                + INTVAL (XEXP (SET_DEST (x), 2)))
    5079          418 :               <= GET_MODE_PRECISION (inner_mode))
    5080       483446 :           && ! side_effects_p (XEXP (SET_DEST (x), 0)))
    5081              :         {
    5082          401 :           HOST_WIDE_INT pos = INTVAL (XEXP (SET_DEST (x), 2));
    5083          401 :           unsigned HOST_WIDE_INT len = INTVAL (XEXP (SET_DEST (x), 1));
    5084          401 :           rtx dest = XEXP (SET_DEST (x), 0);
    5085          401 :           unsigned HOST_WIDE_INT mask = (HOST_WIDE_INT_1U << len) - 1;
    5086          401 :           unsigned HOST_WIDE_INT src = INTVAL (SET_SRC (x)) & mask;
    5087          401 :           rtx or_mask;
    5088              : 
    5089          401 :           if (BITS_BIG_ENDIAN)
    5090              :             pos = GET_MODE_PRECISION (inner_mode) - len - pos;
    5091              : 
    5092          401 :           or_mask = gen_int_mode (src << pos, inner_mode);
    5093          401 :           if (src == mask)
    5094            0 :             SUBST (SET_SRC (x),
    5095              :                    simplify_gen_binary (IOR, inner_mode, dest, or_mask));
    5096              :           else
    5097              :             {
    5098          401 :               rtx negmask = gen_int_mode (~(mask << pos), inner_mode);
    5099          401 :               SUBST (SET_SRC (x),
    5100              :                      simplify_gen_binary (IOR, inner_mode,
    5101              :                                           simplify_gen_binary (AND, inner_mode,
    5102              :                                                                dest, negmask),
    5103              :                                           or_mask));
    5104              :             }
    5105              : 
    5106          401 :           SUBST (SET_DEST (x), dest);
    5107              : 
    5108          401 :           split = find_split_point (&SET_SRC (x), insn, true);
    5109          401 :           if (split && split != &SET_SRC (x))
    5110              :             return split;
    5111              :         }
    5112              : 
    5113              :       /* Otherwise, see if this is an operation that we can split into two.
    5114              :          If so, try to split that.  */
    5115       482627 :       code = GET_CODE (SET_SRC (x));
    5116              : 
    5117       482627 :       switch (code)
    5118              :         {
    5119        17166 :         case AND:
    5120              :           /* If we are AND'ing with a large constant that is only a single
    5121              :              bit and the result is only being used in a context where we
    5122              :              need to know if it is zero or nonzero, replace it with a bit
    5123              :              extraction.  This will avoid the large constant, which might
    5124              :              have taken more than one insn to make.  If the constant were
    5125              :              not a valid argument to the AND but took only one insn to make,
    5126              :              this is no worse, but if it took more than one insn, it will
    5127              :              be better.  */
    5128              : 
    5129        17166 :           if (CONST_INT_P (XEXP (SET_SRC (x), 1))
    5130        11477 :               && REG_P (XEXP (SET_SRC (x), 0))
    5131          450 :               && (pos = exact_log2 (UINTVAL (XEXP (SET_SRC (x), 1)))) >= 7
    5132            2 :               && REG_P (SET_DEST (x))
    5133            1 :               && (split = find_single_use (SET_DEST (x), insn, NULL)) != 0
    5134            1 :               && (GET_CODE (*split) == EQ || GET_CODE (*split) == NE)
    5135            0 :               && XEXP (*split, 0) == SET_DEST (x)
    5136        17166 :               && XEXP (*split, 1) == const0_rtx)
    5137              :             {
    5138            0 :               rtx extraction = make_extraction (GET_MODE (SET_DEST (x)),
    5139            0 :                                                 XEXP (SET_SRC (x), 0),
    5140              :                                                 pos, NULL_RTX, 1,
    5141              :                                                 true, false, false);
    5142            0 :               if (extraction != 0)
    5143              :                 {
    5144            0 :                   SUBST (SET_SRC (x), extraction);
    5145            0 :                   return find_split_point (loc, insn, false);
    5146              :                 }
    5147              :             }
    5148              :           break;
    5149              : 
    5150              :         case NE:
    5151              :           /* If STORE_FLAG_VALUE is -1, this is (NE X 0) and only one bit of X
    5152              :              is known to be on, this can be converted into a NEG of a shift.  */
    5153              :           if (STORE_FLAG_VALUE == -1 && XEXP (SET_SRC (x), 1) == const0_rtx
    5154              :               && GET_MODE (SET_SRC (x)) == GET_MODE (XEXP (SET_SRC (x), 0))
    5155              :               && ((pos = exact_log2 (nonzero_bits (XEXP (SET_SRC (x), 0),
    5156              :                                                    GET_MODE (XEXP (SET_SRC (x),
    5157              :                                                              0))))) >= 1))
    5158              :             {
    5159              :               machine_mode mode = GET_MODE (XEXP (SET_SRC (x), 0));
    5160              :               rtx pos_rtx = gen_int_shift_amount (mode, pos);
    5161              :               SUBST (SET_SRC (x),
    5162              :                      gen_rtx_NEG (mode,
    5163              :                                   gen_rtx_LSHIFTRT (mode,
    5164              :                                                     XEXP (SET_SRC (x), 0),
    5165              :                                                     pos_rtx)));
    5166              : 
    5167              :               split = find_split_point (&SET_SRC (x), insn, true);
    5168              :               if (split && split != &SET_SRC (x))
    5169              :                 return split;
    5170              :             }
    5171              :           break;
    5172              : 
    5173          523 :         case SIGN_EXTEND:
    5174          523 :           inner = XEXP (SET_SRC (x), 0);
    5175              : 
    5176              :           /* We can't optimize if either mode is a partial integer
    5177              :              mode as we don't know how many bits are significant
    5178              :              in those modes.  */
    5179          523 :           if (!is_int_mode (GET_MODE (inner), &inner_mode)
    5180          517 :               || GET_MODE_CLASS (GET_MODE (SET_SRC (x))) == MODE_PARTIAL_INT)
    5181              :             break;
    5182              : 
    5183          517 :           pos = 0;
    5184          517 :           len = GET_MODE_PRECISION (inner_mode);
    5185          517 :           unsignedp = false;
    5186          517 :           break;
    5187              : 
    5188        12882 :         case SIGN_EXTRACT:
    5189        12882 :         case ZERO_EXTRACT:
    5190        12882 :           if (is_a <scalar_int_mode> (GET_MODE (XEXP (SET_SRC (x), 0)),
    5191              :                                       &inner_mode)
    5192        12578 :               && CONST_INT_P (XEXP (SET_SRC (x), 1))
    5193        12578 :               && CONST_INT_P (XEXP (SET_SRC (x), 2)))
    5194              :             {
    5195        12138 :               inner = XEXP (SET_SRC (x), 0);
    5196        12138 :               len = INTVAL (XEXP (SET_SRC (x), 1));
    5197        12138 :               pos = INTVAL (XEXP (SET_SRC (x), 2));
    5198              : 
    5199        12138 :               if (BITS_BIG_ENDIAN)
    5200              :                 pos = GET_MODE_PRECISION (inner_mode) - len - pos;
    5201        12138 :               unsignedp = (code == ZERO_EXTRACT);
    5202              :             }
    5203              :           break;
    5204              : 
    5205              :         default:
    5206              :           break;
    5207              :         }
    5208              : 
    5209       482627 :       if (len
    5210        12655 :           && known_subrange_p (pos, len,
    5211        12655 :                                0, GET_MODE_PRECISION (GET_MODE (inner)))
    5212       495282 :           && is_a <scalar_int_mode> (GET_MODE (SET_SRC (x)), &mode))
    5213              :         {
    5214              :           /* For unsigned, we have a choice of a shift followed by an
    5215              :              AND or two shifts.  Use two shifts for field sizes where the
    5216              :              constant might be too large.  We assume here that we can
    5217              :              always at least get 8-bit constants in an AND insn, which is
    5218              :              true for every current RISC.  */
    5219              : 
    5220        12655 :           if (unsignedp && len <= 8)
    5221              :             {
    5222         5275 :               unsigned HOST_WIDE_INT mask
    5223         5275 :                 = (HOST_WIDE_INT_1U << len) - 1;
    5224         5275 :               rtx pos_rtx = gen_int_shift_amount (mode, pos);
    5225         5275 :               SUBST (SET_SRC (x),
    5226              :                      gen_rtx_AND (mode,
    5227              :                                   gen_rtx_LSHIFTRT
    5228              :                                   (mode, gen_lowpart (mode, inner), pos_rtx),
    5229              :                                   gen_int_mode (mask, mode)));
    5230              : 
    5231         5275 :               split = find_split_point (&SET_SRC (x), insn, true);
    5232         5275 :               if (split && split != &SET_SRC (x))
    5233     31734618 :                 return split;
    5234              :             }
    5235              :           else
    5236              :             {
    5237         7380 :               int left_bits = GET_MODE_PRECISION (mode) - len - pos;
    5238         7380 :               int right_bits = GET_MODE_PRECISION (mode) - len;
    5239        14760 :               SUBST (SET_SRC (x),
    5240              :                      gen_rtx_fmt_ee
    5241              :                      (unsignedp ? LSHIFTRT : ASHIFTRT, mode,
    5242              :                       gen_rtx_ASHIFT (mode,
    5243              :                                       gen_lowpart (mode, inner),
    5244              :                                       gen_int_shift_amount (mode, left_bits)),
    5245              :                       gen_int_shift_amount (mode, right_bits)));
    5246              : 
    5247         7380 :               split = find_split_point (&SET_SRC (x), insn, true);
    5248         7380 :               if (split && split != &SET_SRC (x))
    5249     31734618 :                 return split;
    5250              :             }
    5251              :         }
    5252              : 
    5253              :       /* See if this is a simple operation with a constant as the second
    5254              :          operand.  It might be that this constant is out of range and hence
    5255              :          could be used as a split point.  */
    5256       469972 :       if (BINARY_P (SET_SRC (x))
    5257       205740 :           && CONSTANT_P (XEXP (SET_SRC (x), 1))
    5258       114268 :           && (OBJECT_P (XEXP (SET_SRC (x), 0))
    5259        37633 :               || (GET_CODE (XEXP (SET_SRC (x), 0)) == SUBREG
    5260        11257 :                   && OBJECT_P (SUBREG_REG (XEXP (SET_SRC (x), 0))))))
    5261        78468 :         return &XEXP (SET_SRC (x), 1);
    5262              : 
    5263              :       /* Finally, see if this is a simple operation with its first operand
    5264              :          not in a register.  The operation might require this operand in a
    5265              :          register, so return it as a split point.  We can always do this
    5266              :          because if the first operand were another operation, we would have
    5267              :          already found it as a split point.  */
    5268       391504 :       if ((BINARY_P (SET_SRC (x)) || UNARY_P (SET_SRC (x)))
    5269       391504 :           && ! register_operand (XEXP (SET_SRC (x), 0), VOIDmode))
    5270       121439 :         return &XEXP (SET_SRC (x), 0);
    5271              : 
    5272              :       return 0;
    5273              : 
    5274      1015082 :     case AND:
    5275      1015082 :     case IOR:
    5276              :       /* We write NOR as (and (not A) (not B)), but if we don't have a NOR,
    5277              :          it is better to write this as (not (ior A B)) so we can split it.
    5278              :          Similarly for IOR.  */
    5279      1015082 :       if (GET_CODE (XEXP (x, 0)) == NOT && GET_CODE (XEXP (x, 1)) == NOT)
    5280              :         {
    5281         1808 :           SUBST (*loc,
    5282              :                  gen_rtx_NOT (GET_MODE (x),
    5283              :                               gen_rtx_fmt_ee (code == IOR ? AND : IOR,
    5284              :                                               GET_MODE (x),
    5285              :                                               XEXP (XEXP (x, 0), 0),
    5286              :                                               XEXP (XEXP (x, 1), 0))));
    5287          904 :           return find_split_point (loc, insn, set_src);
    5288              :         }
    5289              : 
    5290              :       /* Many RISC machines have a large set of logical insns.  If the
    5291              :          second operand is a NOT, put it first so we will try to split the
    5292              :          other operand first.  */
    5293      1014178 :       if (GET_CODE (XEXP (x, 1)) == NOT)
    5294              :         {
    5295         5414 :           rtx tem = XEXP (x, 0);
    5296         5414 :           SUBST (XEXP (x, 0), XEXP (x, 1));
    5297         5414 :           SUBST (XEXP (x, 1), tem);
    5298              :         }
    5299              :       /* Many targets have a `(and (not X) Y)` and/or `(ior (not X) Y)` instructions.
    5300              :          Split at that insns.  However if this is
    5301              :          the SET_SRC, we likely do not have such an instruction and it's
    5302              :          worthless to try this split.  */
    5303      1014178 :       if (!set_src && GET_CODE (XEXP (x, 0)) == NOT)
    5304              :         return loc;
    5305              :       break;
    5306              : 
    5307      3423697 :     case PLUS:
    5308      3423697 :     case MINUS:
    5309              :       /* Canonicalization can produce (minus A (mult B C)), where C is a
    5310              :          constant.  It may be better to try splitting (plus (mult B -C) A)
    5311              :          instead if this isn't a multiply by a power of two.  */
    5312       200349 :       if (set_src && code == MINUS && GET_CODE (XEXP (x, 1)) == MULT
    5313        21980 :           && GET_CODE (XEXP (XEXP (x, 1), 1)) == CONST_INT
    5314      3429882 :           && !pow2p_hwi (INTVAL (XEXP (XEXP (x, 1), 1))))
    5315              :         {
    5316         6185 :           machine_mode mode = GET_MODE (x);
    5317         6185 :           unsigned HOST_WIDE_INT this_int = INTVAL (XEXP (XEXP (x, 1), 1));
    5318         6185 :           HOST_WIDE_INT other_int = trunc_int_for_mode (-this_int, mode);
    5319         6185 :           SUBST (*loc, gen_rtx_PLUS (mode,
    5320              :                                      gen_rtx_MULT (mode,
    5321              :                                                    XEXP (XEXP (x, 1), 0),
    5322              :                                                    gen_int_mode (other_int,
    5323              :                                                                  mode)),
    5324              :                                      XEXP (x, 0)));
    5325         6185 :           return find_split_point (loc, insn, set_src);
    5326              :         }
    5327              : 
    5328              :       /* Split at a multiply-accumulate instruction.  However if this is
    5329              :          the SET_SRC, we likely do not have such an instruction and it's
    5330              :          worthless to try this split.  */
    5331      3417512 :       if (!set_src
    5332      2030727 :           && (GET_CODE (XEXP (x, 0)) == MULT
    5333      1915406 :               || (GET_CODE (XEXP (x, 0)) == ASHIFT
    5334       137878 :                   && GET_CODE (XEXP (XEXP (x, 0), 1)) == CONST_INT)))
    5335              :         return loc;
    5336              : 
    5337              :     default:
    5338              :       break;
    5339              :     }
    5340              : 
    5341              :   /* Otherwise, select our actions depending on our rtx class.  */
    5342     26447240 :   switch (GET_RTX_CLASS (code))
    5343              :     {
    5344      1512452 :     case RTX_BITFIELD_OPS:              /* This is ZERO_EXTRACT and SIGN_EXTRACT.  */
    5345      1512452 :     case RTX_TERNARY:
    5346      1512452 :       split = find_split_point (&XEXP (x, 2), insn, false);
    5347      1512452 :       if (split)
    5348              :         return split;
    5349              :       /* fall through */
    5350     10420923 :     case RTX_BIN_ARITH:
    5351     10420923 :     case RTX_COMM_ARITH:
    5352     10420923 :     case RTX_COMPARE:
    5353     10420923 :     case RTX_COMM_COMPARE:
    5354     10420923 :       split = find_split_point (&XEXP (x, 1), insn, false);
    5355     10420923 :       if (split)
    5356              :         return split;
    5357              :       /* fall through */
    5358     10026372 :     case RTX_UNARY:
    5359              :       /* Some machines have (and (shift ...) ...) insns.  If X is not
    5360              :          an AND, but XEXP (X, 0) is, use it as our split point.  */
    5361     10026372 :       if (GET_CODE (x) != AND && GET_CODE (XEXP (x, 0)) == AND)
    5362       380417 :         return &XEXP (x, 0);
    5363              : 
    5364      9645955 :       split = find_split_point (&XEXP (x, 0), insn, false);
    5365      9645955 :       if (split)
    5366      5915133 :         return split;
    5367              :       return loc;
    5368              : 
    5369              :     default:
    5370              :       /* Otherwise, we don't have a split point.  */
    5371              :       return 0;
    5372              :     }
    5373              : }
    5374              : 
    5375              : /* Throughout X, replace FROM with TO, and return the result.
    5376              :    The result is TO if X is FROM;
    5377              :    otherwise the result is X, but its contents may have been modified.
    5378              :    If they were modified, a record was made in undobuf so that
    5379              :    undo_all will (among other things) return X to its original state.
    5380              : 
    5381              :    If the number of changes necessary is too much to record to undo,
    5382              :    the excess changes are not made, so the result is invalid.
    5383              :    The changes already made can still be undone.
    5384              :    undobuf.num_undo is incremented for such changes, so by testing that
    5385              :    the caller can tell whether the result is valid.
    5386              : 
    5387              :    `n_occurrences' is incremented each time FROM is replaced.
    5388              : 
    5389              :    IN_DEST is true if we are processing the SET_DEST of a SET.
    5390              : 
    5391              :    IN_COND is true if we are at the top level of a condition.
    5392              : 
    5393              :    UNIQUE_COPY is true if each substitution must be unique.  We do this
    5394              :    by copying if `n_occurrences' is nonzero.  */
    5395              : 
    5396              : static rtx
    5397    425526913 : subst (rtx x, rtx from, rtx to, bool in_dest, bool in_cond, bool unique_copy)
    5398              : {
    5399    425526913 :   enum rtx_code code = GET_CODE (x);
    5400    425526913 :   machine_mode op0_mode = VOIDmode;
    5401    425526913 :   const char *fmt;
    5402    425526913 :   int len, i;
    5403    425526913 :   rtx new_rtx;
    5404              : 
    5405              : /* Two expressions are equal if they are identical copies of a shared
    5406              :    RTX or if they are both registers with the same register number
    5407              :    and mode.  */
    5408              : 
    5409              : #define COMBINE_RTX_EQUAL_P(X,Y)                        \
    5410              :   ((X) == (Y)                                           \
    5411              :    || (REG_P (X) && REG_P (Y)   \
    5412              :        && REGNO (X) == REGNO (Y) && GET_MODE (X) == GET_MODE (Y)))
    5413              : 
    5414              :   /* Do not substitute into clobbers of regs -- this will never result in
    5415              :      valid RTL.  */
    5416    425526913 :   if (GET_CODE (x) == CLOBBER && REG_P (XEXP (x, 0)))
    5417              :     return x;
    5418              : 
    5419    414739827 :   if (! in_dest && COMBINE_RTX_EQUAL_P (x, from))
    5420              :     {
    5421            0 :       n_occurrences++;
    5422            0 :       return (unique_copy && n_occurrences > 1 ? copy_rtx (to) : to);
    5423              :     }
    5424              : 
    5425              :   /* If X and FROM are the same register but different modes, they
    5426              :      will not have been seen as equal above.  However, the log links code
    5427              :      will make a LOG_LINKS entry for that case.  If we do nothing, we
    5428              :      will try to rerecognize our original insn and, when it succeeds,
    5429              :      we will delete the feeding insn, which is incorrect.
    5430              : 
    5431              :      So force this insn not to match in this (rare) case.  */
    5432     92323588 :   if (! in_dest && code == REG && REG_P (from)
    5433    447897744 :       && reg_overlap_mentioned_p (x, from))
    5434         4689 :     return gen_rtx_CLOBBER (GET_MODE (x), const0_rtx);
    5435              : 
    5436              :   /* If this is an object, we are done unless it is a MEM or LO_SUM, both
    5437              :      of which may contain things that can be combined.  */
    5438    414735138 :   if (code != MEM && code != LO_SUM && OBJECT_P (x))
    5439              :     return x;
    5440              : 
    5441              :   /* It is possible to have a subexpression appear twice in the insn.
    5442              :      Suppose that FROM is a register that appears within TO.
    5443              :      Then, after that subexpression has been scanned once by `subst',
    5444              :      the second time it is scanned, TO may be found.  If we were
    5445              :      to scan TO here, we would find FROM within it and create a
    5446              :      self-referent rtl structure which is completely wrong.  */
    5447    222261213 :   if (COMBINE_RTX_EQUAL_P (x, to))
    5448              :     return to;
    5449              : 
    5450              :   /* Parallel asm_operands need special attention because all of the
    5451              :      inputs are shared across the arms.  Furthermore, unsharing the
    5452              :      rtl results in recognition failures.  Failure to handle this case
    5453              :      specially can result in circular rtl.
    5454              : 
    5455              :      Solve this by doing a normal pass across the first entry of the
    5456              :      parallel, and only processing the SET_DESTs of the subsequent
    5457              :      entries.  Ug.  */
    5458              : 
    5459    222117236 :   if (code == PARALLEL
    5460     13392919 :       && GET_CODE (XVECEXP (x, 0, 0)) == SET
    5461     11377853 :       && GET_CODE (SET_SRC (XVECEXP (x, 0, 0))) == ASM_OPERANDS)
    5462              :     {
    5463        21499 :       new_rtx = subst (XVECEXP (x, 0, 0), from, to, false, false, unique_copy);
    5464              : 
    5465              :       /* If this substitution failed, this whole thing fails.  */
    5466        21499 :       if (GET_CODE (new_rtx) == CLOBBER
    5467            0 :           && XEXP (new_rtx, 0) == const0_rtx)
    5468              :         return new_rtx;
    5469              : 
    5470        21499 :       SUBST (XVECEXP (x, 0, 0), new_rtx);
    5471              : 
    5472       104134 :       for (i = XVECLEN (x, 0) - 1; i >= 1; i--)
    5473              :         {
    5474        82635 :           rtx dest = SET_DEST (XVECEXP (x, 0, i));
    5475              : 
    5476        82635 :           if (!REG_P (dest) && GET_CODE (dest) != PC)
    5477              :             {
    5478         3188 :               new_rtx = subst (dest, from, to, false, false, unique_copy);
    5479              : 
    5480              :               /* If this substitution failed, this whole thing fails.  */
    5481         3188 :               if (GET_CODE (new_rtx) == CLOBBER
    5482            0 :                   && XEXP (new_rtx, 0) == const0_rtx)
    5483              :                 return new_rtx;
    5484              : 
    5485         3188 :               SUBST (SET_DEST (XVECEXP (x, 0, i)), new_rtx);
    5486              :             }
    5487              :         }
    5488              :     }
    5489              :   else
    5490              :     {
    5491    222095737 :       len = GET_RTX_LENGTH (code);
    5492    222095737 :       fmt = GET_RTX_FORMAT (code);
    5493              : 
    5494              :       /* We don't need to process a SET_DEST that is a register or PC, so
    5495              :          set up to skip this common case.  All other cases where we want
    5496              :          to suppress replacing something inside a SET_SRC are handled via
    5497              :          the IN_DEST operand.  */
    5498    222095737 :       if (code == SET
    5499     48468644 :           && (REG_P (SET_DEST (x))
    5500     48468644 :               || GET_CODE (SET_DEST (x)) == PC))
    5501    222095737 :         fmt = "ie";
    5502              : 
    5503              :       /* Trying to simplify the operands of a widening MULT is not likely
    5504              :          to create RTL matching a machine insn.  */
    5505    222095737 :       if (code == MULT
    5506      4977965 :           && (GET_CODE (XEXP (x, 0)) == ZERO_EXTEND
    5507      4977965 :               || GET_CODE (XEXP (x, 0)) == SIGN_EXTEND)
    5508       312735 :           && (GET_CODE (XEXP (x, 1)) == ZERO_EXTEND
    5509       312735 :               || GET_CODE (XEXP (x, 1)) == SIGN_EXTEND)
    5510       235134 :           && REG_P (XEXP (XEXP (x, 0), 0))
    5511       112402 :           && REG_P (XEXP (XEXP (x, 1), 0))
    5512        99767 :           && from == to)
    5513              :         return x;
    5514              : 
    5515              : 
    5516              :       /* Get the mode of operand 0 in case X is now a SIGN_EXTEND of a
    5517              :          constant.  */
    5518    222034680 :       if (fmt[0] == 'e')
    5519    163632200 :         op0_mode = GET_MODE (XEXP (x, 0));
    5520              : 
    5521    657477104 :       for (i = 0; i < len; i++)
    5522              :         {
    5523    436422758 :           if (fmt[i] == 'E')
    5524              :             {
    5525     15872654 :               int j;
    5526     50411457 :               for (j = XVECLEN (x, i) - 1; j >= 0; j--)
    5527              :                 {
    5528     34671855 :                   if (COMBINE_RTX_EQUAL_P (XVECEXP (x, i, j), from))
    5529              :                     {
    5530         1692 :                       new_rtx = (unique_copy && n_occurrences
    5531       307074 :                              ? copy_rtx (to) : to);
    5532       307055 :                       n_occurrences++;
    5533              :                     }
    5534              :                   else
    5535              :                     {
    5536     34364800 :                       new_rtx = subst (XVECEXP (x, i, j), from, to,
    5537              :                                        false, false, unique_copy);
    5538              : 
    5539              :                       /* If this substitution failed, this whole thing
    5540              :                          fails.  */
    5541     34364800 :                       if (GET_CODE (new_rtx) == CLOBBER
    5542     11238660 :                           && XEXP (new_rtx, 0) == const0_rtx)
    5543              :                         return new_rtx;
    5544              :                     }
    5545              : 
    5546     34538803 :                   SUBST (XVECEXP (x, i, j), new_rtx);
    5547              :                 }
    5548              :             }
    5549    420550104 :           else if (fmt[i] == 'e')
    5550              :             {
    5551              :               /* If this is a register being set, ignore it.  */
    5552    343028173 :               new_rtx = XEXP (x, i);
    5553    343028173 :               if (in_dest
    5554    343028173 :                   && i == 0
    5555      5972637 :                   && (((code == SUBREG || code == ZERO_EXTRACT)
    5556       367256 :                        && REG_P (new_rtx))
    5557      5607834 :                       || code == STRICT_LOW_PART))
    5558              :                 ;
    5559              : 
    5560    342652662 :               else if (COMBINE_RTX_EQUAL_P (XEXP (x, i), from))
    5561              :                 {
    5562              :                   /* In general, don't install a subreg involving two
    5563              :                      modes not tieable.  It can worsen register
    5564              :                      allocation, and can even make invalid reload
    5565              :                      insns, since the reg inside may need to be copied
    5566              :                      from in the outside mode, and that may be invalid
    5567              :                      if it is an fp reg copied in integer mode.
    5568              : 
    5569              :                      We allow an exception to this: It is valid if
    5570              :                      it is inside another SUBREG and the mode of that
    5571              :                      SUBREG and the mode of the inside of TO is
    5572              :                      tieable.  */
    5573              : 
    5574     48665632 :                   if (GET_CODE (to) == SUBREG
    5575       606987 :                       && !targetm.modes_tieable_p (GET_MODE (to),
    5576       606987 :                                                    GET_MODE (SUBREG_REG (to)))
    5577     48997162 :                       && ! (code == SUBREG
    5578        28516 :                             && (targetm.modes_tieable_p
    5579        28516 :                                 (GET_MODE (x), GET_MODE (SUBREG_REG (to))))))
    5580       300757 :                     return gen_rtx_CLOBBER (VOIDmode, const0_rtx);
    5581              : 
    5582     48364875 :                   if (code == SUBREG
    5583      2581401 :                       && REG_P (to)
    5584        98053 :                       && REGNO (to) < FIRST_PSEUDO_REGISTER
    5585     48364880 :                       && simplify_subreg_regno (REGNO (to), GET_MODE (to),
    5586            5 :                                                 SUBREG_BYTE (x),
    5587            5 :                                                 GET_MODE (x)) < 0)
    5588            0 :                     return gen_rtx_CLOBBER (VOIDmode, const0_rtx);
    5589              : 
    5590     48364875 :                   new_rtx = (unique_copy && n_occurrences ? copy_rtx (to) : to);
    5591     48364875 :                   n_occurrences++;
    5592              :                 }
    5593              :               else
    5594              :                 /* If we are in a SET_DEST, suppress most cases unless we
    5595              :                    have gone inside a MEM, in which case we want to
    5596              :                    simplify the address.  We assume here that things that
    5597              :                    are actually part of the destination have their inner
    5598              :                    parts in the first expression.  This is true for SUBREG,
    5599              :                    STRICT_LOW_PART, and ZERO_EXTRACT, which are the only
    5600              :                    things aside from REG and MEM that should appear in a
    5601              :                    SET_DEST.  */
    5602    338010502 :                 new_rtx = subst (XEXP (x, i), from, to,
    5603              :                              (((in_dest
    5604      5316292 :                                 && (code == SUBREG || code == STRICT_LOW_PART
    5605      5316292 :                                     || code == ZERO_EXTRACT))
    5606    293979134 :                                || code == SET)
    5607              :                               && i == 0),
    5608    293987030 :                                  code == IF_THEN_ELSE && i == 0,
    5609              :                                  unique_copy);
    5610              : 
    5611              :               /* If we found that we will have to reject this combination,
    5612              :                  indicate that by returning the CLOBBER ourselves, rather than
    5613              :                  an expression containing it.  This will speed things up as
    5614              :                  well as prevent accidents where two CLOBBERs are considered
    5615              :                  to be equal, thus producing an incorrect simplification.  */
    5616              : 
    5617    342727416 :               if (GET_CODE (new_rtx) == CLOBBER && XEXP (new_rtx, 0) == const0_rtx)
    5618              :                 return new_rtx;
    5619              : 
    5620    342181143 :               if (GET_CODE (x) == SUBREG && CONST_SCALAR_INT_P (new_rtx))
    5621              :                 {
    5622        31864 :                   machine_mode mode = GET_MODE (x);
    5623              : 
    5624        63728 :                   x = simplify_subreg (GET_MODE (x), new_rtx,
    5625        31864 :                                        GET_MODE (SUBREG_REG (x)),
    5626        31864 :                                        SUBREG_BYTE (x));
    5627        31864 :                   if (! x)
    5628            2 :                     x = gen_rtx_CLOBBER (mode, const0_rtx);
    5629              :                 }
    5630    342149279 :               else if (CONST_SCALAR_INT_P (new_rtx)
    5631              :                        && (GET_CODE (x) == ZERO_EXTEND
    5632     61983469 :                            || GET_CODE (x) == SIGN_EXTEND
    5633              :                            || GET_CODE (x) == FLOAT
    5634              :                            || GET_CODE (x) == UNSIGNED_FLOAT))
    5635              :                 {
    5636       139582 :                   x = simplify_unary_operation (GET_CODE (x), GET_MODE (x),
    5637              :                                                 new_rtx,
    5638        69791 :                                                 GET_MODE (XEXP (x, 0)));
    5639        69791 :                   if (!x)
    5640          252 :                     return gen_rtx_CLOBBER (VOIDmode, const0_rtx);
    5641              :                 }
    5642              :               /* CONST_INTs shouldn't be substituted into PRE_DEC, PRE_MODIFY
    5643              :                  etc. arguments, otherwise we can ICE before trying to recog
    5644              :                  it.  See PR104446.  */
    5645    342079488 :               else if (CONST_SCALAR_INT_P (new_rtx)
    5646     61913678 :                        && GET_RTX_CLASS (GET_CODE (x)) == RTX_AUTOINC)
    5647            0 :                 return gen_rtx_CLOBBER (VOIDmode, const0_rtx);
    5648              :               else
    5649    342079488 :                 SUBST (XEXP (x, i), new_rtx);
    5650              :             }
    5651              :         }
    5652              :     }
    5653              : 
    5654              :   /* Check if we are loading something from the constant pool via float
    5655              :      extension; in this case we would undo compress_float_constant
    5656              :      optimization and degenerate constant load to an immediate value.  */
    5657    221075845 :   if (GET_CODE (x) == FLOAT_EXTEND
    5658       309941 :       && MEM_P (XEXP (x, 0))
    5659    221139991 :       && MEM_READONLY_P (XEXP (x, 0)))
    5660              :     {
    5661        36746 :       rtx tmp = avoid_constant_pool_reference (x);
    5662        36746 :       if (x != tmp)
    5663              :         return x;
    5664              :     }
    5665              : 
    5666              :   /* Try to simplify X.  If the simplification changed the code, it is likely
    5667              :      that further simplification will help, so loop, but limit the number
    5668              :      of repetitions that will be performed.  */
    5669              : 
    5670    229273539 :   for (i = 0; i < 4; i++)
    5671              :     {
    5672              :       /* If X is sufficiently simple, don't bother trying to do anything
    5673              :          with it.  */
    5674    229217940 :       if (code != CONST_INT && code != REG && code != CLOBBER)
    5675    228539208 :         x = combine_simplify_rtx (x, op0_mode, in_dest, in_cond);
    5676              : 
    5677    229217940 :       if (GET_CODE (x) == code)
    5678              :         break;
    5679              : 
    5680      8234310 :       code = GET_CODE (x);
    5681              : 
    5682              :       /* We no longer know the original mode of operand 0 since we
    5683              :          have changed the form of X)  */
    5684      8234310 :       op0_mode = VOIDmode;
    5685              :     }
    5686              : 
    5687              :   return x;
    5688              : }
    5689              : 
    5690              : /* If X is a commutative operation whose operands are not in the canonical
    5691              :    order, use substitutions to swap them.  */
    5692              : 
    5693              : static void
    5694    667757164 : maybe_swap_commutative_operands (rtx x)
    5695              : {
    5696    667757164 :   if (COMMUTATIVE_ARITH_P (x)
    5697    667757164 :       && swap_commutative_operands_p (XEXP (x, 0), XEXP (x, 1)))
    5698              :     {
    5699      3559960 :       rtx temp = XEXP (x, 0);
    5700      3559960 :       SUBST (XEXP (x, 0), XEXP (x, 1));
    5701      3559960 :       SUBST (XEXP (x, 1), temp);
    5702              :     }
    5703              : 
    5704              :   /* Canonicalize (vec_merge (fma op2 op1 op3) op1 mask) to
    5705              :      (vec_merge (fma op1 op2 op3) op1 mask).  */
    5706    667757164 :   if (GET_CODE (x) == VEC_MERGE
    5707      1119528 :       && GET_CODE (XEXP (x, 0)) == FMA)
    5708              :     {
    5709        26537 :       rtx fma_op1 = XEXP (XEXP (x, 0), 0);
    5710        26537 :       rtx fma_op2 = XEXP (XEXP (x, 0), 1);
    5711        26537 :       rtx masked_op = XEXP (x, 1);
    5712        26537 :       if (rtx_equal_p (masked_op, fma_op2))
    5713              :         {
    5714          192 :           if (GET_CODE (fma_op1) == NEG)
    5715              :             {
    5716              :               /* Keep the negate canonicalized to the first operand.  */
    5717          132 :               fma_op1 = XEXP (fma_op1, 0);
    5718          132 :               SUBST (XEXP (XEXP (XEXP (x, 0), 0), 0), fma_op2);
    5719          132 :               SUBST (XEXP (XEXP (x, 0), 1), fma_op1);
    5720              :             }
    5721              :           else
    5722              :             {
    5723           60 :               SUBST (XEXP (XEXP (x, 0), 0), fma_op2);
    5724           60 :               SUBST (XEXP (XEXP (x, 0), 1), fma_op1);
    5725              :             }
    5726              :         }
    5727              :     }
    5728              : 
    5729    667757164 :   unsigned n_elts = 0;
    5730    667757164 :   if (GET_CODE (x) == VEC_MERGE
    5731      1119528 :       && CONST_INT_P (XEXP (x, 2))
    5732      1471784 :       && GET_MODE_NUNITS (GET_MODE (x)).is_constant (&n_elts)
    5733    668493056 :       && (swap_commutative_operands_p (XEXP (x, 0), XEXP (x, 1))
    5734              :           /* Two operands have same precedence, then
    5735              :              first bit of mask select first operand.  */
    5736       702063 :           || (!swap_commutative_operands_p (XEXP (x, 1), XEXP (x, 0))
    5737       159475 :               && !(UINTVAL (XEXP (x, 2)) & 1))))
    5738              :     {
    5739        66380 :       rtx temp = XEXP (x, 0);
    5740        66380 :       unsigned HOST_WIDE_INT sel = UINTVAL (XEXP (x, 2));
    5741        66380 :       unsigned HOST_WIDE_INT mask = HOST_WIDE_INT_1U;
    5742        66380 :       if (n_elts == HOST_BITS_PER_WIDE_INT)
    5743              :         mask = -1;
    5744              :       else
    5745        66239 :         mask = (HOST_WIDE_INT_1U << n_elts) - 1;
    5746        66380 :       SUBST (XEXP (x, 0), XEXP (x, 1));
    5747        66380 :       SUBST (XEXP (x, 1), temp);
    5748        66380 :       SUBST (XEXP (x, 2), GEN_INT (~sel & mask));
    5749              :     }
    5750    667757164 : }
    5751              : 
    5752              : /* Simplify X, a piece of RTL.  We just operate on the expression at the
    5753              :    outer level; call `subst' to simplify recursively.  Return the new
    5754              :    expression.
    5755              : 
    5756              :    OP0_MODE is the original mode of XEXP (x, 0).  IN_DEST is true
    5757              :    if we are inside a SET_DEST.  IN_COND is true if we are at the top level
    5758              :    of a condition.  */
    5759              : 
    5760              : static rtx
    5761    229505642 : combine_simplify_rtx (rtx x, machine_mode op0_mode, bool in_dest, bool in_cond)
    5762              : {
    5763    229505642 :   enum rtx_code code = GET_CODE (x);
    5764    229505642 :   machine_mode mode = GET_MODE (x);
    5765    229505642 :   scalar_int_mode int_mode;
    5766    229505642 :   rtx temp;
    5767    229505642 :   int i;
    5768              : 
    5769              :   /* If this is a commutative operation, put a constant last and a complex
    5770              :      expression first.  We don't need to do this for comparisons here.  */
    5771    229505642 :   maybe_swap_commutative_operands (x);
    5772              : 
    5773              :   /* Try to fold this expression in case we have constants that weren't
    5774              :      present before.  */
    5775    229505642 :   temp = 0;
    5776    229505642 :   switch (GET_RTX_CLASS (code))
    5777              :     {
    5778      7667743 :     case RTX_UNARY:
    5779      7667743 :       if (op0_mode == VOIDmode)
    5780       167374 :         op0_mode = GET_MODE (XEXP (x, 0));
    5781      7667743 :       temp = simplify_unary_operation (code, mode, XEXP (x, 0), op0_mode);
    5782      7667743 :       break;
    5783     18435969 :     case RTX_COMPARE:
    5784     18435969 :     case RTX_COMM_COMPARE:
    5785     18435969 :       {
    5786     18435969 :         machine_mode cmp_mode = GET_MODE (XEXP (x, 0));
    5787     18435969 :         if (cmp_mode == VOIDmode)
    5788              :           {
    5789        48954 :             cmp_mode = GET_MODE (XEXP (x, 1));
    5790        48954 :             if (cmp_mode == VOIDmode)
    5791         8177 :               cmp_mode = op0_mode;
    5792              :           }
    5793     18435969 :         temp = simplify_relational_operation (code, mode, cmp_mode,
    5794              :                                               XEXP (x, 0), XEXP (x, 1));
    5795              :       }
    5796     18435969 :       break;
    5797     90779557 :     case RTX_COMM_ARITH:
    5798     90779557 :     case RTX_BIN_ARITH:
    5799     90779557 :       temp = simplify_binary_operation (code, mode, XEXP (x, 0), XEXP (x, 1));
    5800     90779557 :       break;
    5801     14779279 :     case RTX_BITFIELD_OPS:
    5802     14779279 :     case RTX_TERNARY:
    5803     14779279 :       temp = simplify_ternary_operation (code, mode, op0_mode, XEXP (x, 0),
    5804              :                                          XEXP (x, 1), XEXP (x, 2));
    5805     14779279 :       break;
    5806              :     default:
    5807              :       break;
    5808              :     }
    5809              : 
    5810    131662548 :   if (temp)
    5811              :     {
    5812     16973379 :       x = temp;
    5813     16973379 :       code = GET_CODE (temp);
    5814     16973379 :       op0_mode = VOIDmode;
    5815     16973379 :       mode = GET_MODE (temp);
    5816              :     }
    5817              : 
    5818              :   /* If this is a simple operation applied to an IF_THEN_ELSE, try
    5819              :      applying it to the arms of the IF_THEN_ELSE.  This often simplifies
    5820              :      things.  Check for cases where both arms are testing the same
    5821              :      condition.
    5822              : 
    5823              :      Don't do anything if all operands are very simple.  */
    5824              : 
    5825    229505642 :   if ((BINARY_P (x)
    5826    108954093 :        && ((!OBJECT_P (XEXP (x, 0))
    5827     43110120 :             && ! (GET_CODE (XEXP (x, 0)) == SUBREG
    5828      5310746 :                   && OBJECT_P (SUBREG_REG (XEXP (x, 0)))))
    5829     68843036 :            || (!OBJECT_P (XEXP (x, 1))
    5830      4922218 :                && ! (GET_CODE (XEXP (x, 1)) == SUBREG
    5831      1970392 :                      && OBJECT_P (SUBREG_REG (XEXP (x, 1)))))))
    5832    186165124 :       || (UNARY_P (x)
    5833      7547171 :           && (!OBJECT_P (XEXP (x, 0))
    5834      3428442 :                && ! (GET_CODE (XEXP (x, 0)) == SUBREG
    5835       912327 :                      && OBJECT_P (SUBREG_REG (XEXP (x, 0)))))))
    5836              :     {
    5837     45948806 :       rtx cond, true_rtx, false_rtx;
    5838              : 
    5839     45948806 :       cond = if_then_else_cond (x, &true_rtx, &false_rtx);
    5840     45948806 :       if (cond != 0
    5841              :           /* If everything is a comparison, what we have is highly unlikely
    5842              :              to be simpler, so don't use it.  */
    5843      4658450 :           && ! (COMPARISON_P (x)
    5844      1276204 :                 && (COMPARISON_P (true_rtx) || COMPARISON_P (false_rtx)))
    5845              :           /* Similarly, if we end up with one of the expressions the same
    5846              :              as the original, it is certainly not simpler.  */
    5847      4481587 :           && ! rtx_equal_p (x, true_rtx)
    5848     50430393 :           && ! rtx_equal_p (x, false_rtx))
    5849              :         {
    5850      4481587 :           rtx cop1 = const0_rtx;
    5851      4481587 :           enum rtx_code cond_code = simplify_comparison (NE, &cond, &cop1);
    5852              : 
    5853      4481587 :           if (cond_code == NE && COMPARISON_P (cond))
    5854       673172 :             return x;
    5855              : 
    5856              :           /* Simplify the alternative arms; this may collapse the true and
    5857              :              false arms to store-flag values.  Be careful to use copy_rtx
    5858              :              here since true_rtx or false_rtx might share RTL with x as a
    5859              :              result of the if_then_else_cond call above.  */
    5860      3808415 :           true_rtx = subst (copy_rtx (true_rtx), pc_rtx, pc_rtx,
    5861              :                             false, false, false);
    5862      3808415 :           false_rtx = subst (copy_rtx (false_rtx), pc_rtx, pc_rtx,
    5863              :                              false, false, false);
    5864              : 
    5865              :           /* If true_rtx and false_rtx are not general_operands, an if_then_else
    5866              :              is unlikely to be simpler.  */
    5867      3808415 :           if (general_operand (true_rtx, VOIDmode)
    5868      3808415 :               && general_operand (false_rtx, VOIDmode))
    5869              :             {
    5870      1424083 :               enum rtx_code reversed;
    5871              : 
    5872              :               /* Restarting if we generate a store-flag expression will cause
    5873              :                  us to loop.  Just drop through in this case.  */
    5874              : 
    5875              :               /* If the result values are STORE_FLAG_VALUE and zero, we can
    5876              :                  just make the comparison operation.  */
    5877      1424083 :               if (true_rtx == const_true_rtx && false_rtx == const0_rtx)
    5878       628499 :                 x = simplify_gen_relational (cond_code, mode, VOIDmode,
    5879              :                                              cond, cop1);
    5880       519389 :               else if (true_rtx == const0_rtx && false_rtx == const_true_rtx
    5881       795584 :                        && ((reversed = reversed_comparison_code_parts
    5882       460757 :                                         (cond_code, cond, cop1, NULL))
    5883              :                            != UNKNOWN))
    5884       460757 :                 x = simplify_gen_relational (reversed, mode, VOIDmode,
    5885              :                                              cond, cop1);
    5886              : 
    5887              :               /* Likewise, we can make the negate of a comparison operation
    5888              :                  if the result values are - STORE_FLAG_VALUE and zero.  */
    5889       334827 :               else if (CONST_INT_P (true_rtx)
    5890       233895 :                        && INTVAL (true_rtx) == - STORE_FLAG_VALUE
    5891        46106 :                        && false_rtx == const0_rtx)
    5892        44388 :                 x = simplify_gen_unary (NEG, mode,
    5893              :                                         simplify_gen_relational (cond_code,
    5894              :                                                                  mode, VOIDmode,
    5895              :                                                                  cond, cop1),
    5896              :                                         mode);
    5897       290439 :               else if (CONST_INT_P (false_rtx)
    5898       227596 :                        && INTVAL (false_rtx) == - STORE_FLAG_VALUE
    5899        24071 :                        && true_rtx == const0_rtx
    5900       290439 :                        && ((reversed = reversed_comparison_code_parts
    5901        21611 :                                         (cond_code, cond, cop1, NULL))
    5902              :                            != UNKNOWN))
    5903        21608 :                 x = simplify_gen_unary (NEG, mode,
    5904              :                                         simplify_gen_relational (reversed,
    5905              :                                                                  mode, VOIDmode,
    5906              :                                                                  cond, cop1),
    5907              :                                         mode);
    5908              : 
    5909      1424083 :               code = GET_CODE (x);
    5910      1424083 :               op0_mode = VOIDmode;
    5911              :             }
    5912              :         }
    5913              :     }
    5914              : 
    5915              :   /* First see if we can apply the inverse distributive law.  */
    5916    228832470 :   if (code == PLUS || code == MINUS
    5917    228832470 :       || code == AND || code == IOR || code == XOR)
    5918              :     {
    5919     51727095 :       x = apply_distributive_law (x);
    5920     51727095 :       code = GET_CODE (x);
    5921     51727095 :       op0_mode = VOIDmode;
    5922              :     }
    5923              : 
    5924              :   /* If CODE is an associative operation not otherwise handled, see if we
    5925              :      can associate some operands.  This can win if they are constants or
    5926              :      if they are logically related (i.e. (a & b) & a).  */
    5927    228832470 :   if ((code == PLUS || code == MINUS || code == MULT || code == DIV
    5928              :        || code == AND || code == IOR || code == XOR
    5929              :        || code == SMAX || code == SMIN || code == UMAX || code == UMIN)
    5930     56313513 :       && ((INTEGRAL_MODE_P (mode) && code != DIV)
    5931      4779133 :           || (flag_associative_math && FLOAT_MODE_P (mode))))
    5932              :     {
    5933     52176747 :       if (GET_CODE (XEXP (x, 0)) == code)
    5934              :         {
    5935      4100506 :           rtx other = XEXP (XEXP (x, 0), 0);
    5936      4100506 :           rtx inner_op0 = XEXP (XEXP (x, 0), 1);
    5937      4100506 :           rtx inner_op1 = XEXP (x, 1);
    5938      4100506 :           rtx inner;
    5939              : 
    5940              :           /* Make sure we pass the constant operand if any as the second
    5941              :              one if this is a commutative operation.  */
    5942      4100506 :           if (CONSTANT_P (inner_op0) && COMMUTATIVE_ARITH_P (x))
    5943              :             std::swap (inner_op0, inner_op1);
    5944      4100506 :           inner = simplify_binary_operation (code == MINUS ? PLUS
    5945      3997105 :                                              : code == DIV ? MULT
    5946              :                                              : code,
    5947              :                                              mode, inner_op0, inner_op1);
    5948              : 
    5949              :           /* For commutative operations, try the other pair if that one
    5950              :              didn't simplify.  */
    5951      4100506 :           if (inner == 0 && COMMUTATIVE_ARITH_P (x))
    5952              :             {
    5953      3967755 :               other = XEXP (XEXP (x, 0), 1);
    5954      3967755 :               inner = simplify_binary_operation (code, mode,
    5955              :                                                  XEXP (XEXP (x, 0), 0),
    5956              :                                                  XEXP (x, 1));
    5957              :             }
    5958              : 
    5959      4068053 :           if (inner)
    5960       247268 :             return simplify_gen_binary (code, mode, other, inner);
    5961              :         }
    5962              :     }
    5963              : 
    5964              :   /* A little bit of algebraic simplification here.  */
    5965    228585202 :   switch (code)
    5966              :     {
    5967     22896305 :     case PREFETCH:
    5968              :       /* A prefetch reaches memory through an address, and targets recognize
    5969              :          that address with the same predicates they use for a MEM, so it
    5970              :          needs the same treatment.  */
    5971     22896305 :     case MEM:
    5972              :       /* Ensure that our address has any ASHIFTs converted to MULT in case
    5973              :          address-recognizing predicates are called later.  */
    5974     22896305 :       temp = make_compound_operation (XEXP (x, 0), MEM);
    5975     22896305 :       SUBST (XEXP (x, 0), temp);
    5976     22896305 :       break;
    5977              : 
    5978      9756481 :     case SUBREG:
    5979      9756481 :       if (op0_mode == VOIDmode)
    5980       160800 :         op0_mode = GET_MODE (SUBREG_REG (x));
    5981              : 
    5982              :       /* See if this can be moved to simplify_subreg.  */
    5983      9756481 :       if (CONSTANT_P (SUBREG_REG (x))
    5984        46096 :           && known_eq (subreg_lowpart_offset (mode, op0_mode), SUBREG_BYTE (x))
    5985              :              /* Don't call gen_lowpart if the inner mode
    5986              :                 is VOIDmode and we cannot simplify it, as SUBREG without
    5987              :                 inner mode is invalid.  */
    5988      9779473 :           && (GET_MODE (SUBREG_REG (x)) != VOIDmode
    5989            0 :               || gen_lowpart_common (mode, SUBREG_REG (x))))
    5990        22992 :         return gen_lowpart (mode, SUBREG_REG (x));
    5991              : 
    5992      9733489 :       if (GET_MODE_CLASS (GET_MODE (SUBREG_REG (x))) == MODE_CC)
    5993              :         break;
    5994      9733489 :       {
    5995      9733489 :         rtx temp;
    5996     19466978 :         temp = simplify_subreg (mode, SUBREG_REG (x), op0_mode,
    5997      9733489 :                                 SUBREG_BYTE (x));
    5998      9733489 :         if (temp)
    5999    229505642 :           return temp;
    6000              : 
    6001              :         /* If op is known to have all lower bits zero, the result is zero.  */
    6002      9080280 :         scalar_int_mode int_mode, int_op0_mode;
    6003      9080280 :         if (!in_dest
    6004      5536754 :             && is_a <scalar_int_mode> (mode, &int_mode)
    6005      5412264 :             && is_a <scalar_int_mode> (op0_mode, &int_op0_mode)
    6006      5412264 :             && (GET_MODE_PRECISION (int_mode)
    6007      5412264 :                 < GET_MODE_PRECISION (int_op0_mode))
    6008      4865189 :             && known_eq (subreg_lowpart_offset (int_mode, int_op0_mode),
    6009              :                          SUBREG_BYTE (x))
    6010      4354428 :             && HWI_COMPUTABLE_MODE_P (int_op0_mode)
    6011      4141066 :             && ((nonzero_bits (SUBREG_REG (x), int_op0_mode)
    6012      4141066 :                  & GET_MODE_MASK (int_mode)) == 0)
    6013      9081270 :             && !side_effects_p (SUBREG_REG (x)))
    6014          990 :           return CONST0_RTX (int_mode);
    6015              :       }
    6016              : 
    6017              :       /* Don't change the mode of the MEM if that would change the meaning
    6018              :          of the address.  */
    6019      9079290 :       if (MEM_P (SUBREG_REG (x))
    6020      9079290 :           && (MEM_VOLATILE_P (SUBREG_REG (x))
    6021        70390 :               || mode_dependent_address_p (XEXP (SUBREG_REG (x), 0),
    6022        70424 :                                            MEM_ADDR_SPACE (SUBREG_REG (x)))))
    6023        45608 :         return gen_rtx_CLOBBER (mode, const0_rtx);
    6024              : 
    6025              :       /* Note that we cannot do any narrowing for non-constants since
    6026              :          we might have been counting on using the fact that some bits were
    6027              :          zero.  We now do this in the SET.  */
    6028              : 
    6029              :       break;
    6030              : 
    6031       395634 :     case NEG:
    6032       395634 :       temp = expand_compound_operation (XEXP (x, 0));
    6033              : 
    6034              :       /* For C equal to the width of MODE minus 1, (neg (ashiftrt X C)) can be
    6035              :          replaced by (lshiftrt X C).  This will convert
    6036              :          (neg (sign_extract X 1 Y)) to (zero_extract X 1 Y).  */
    6037              : 
    6038       395634 :       if (GET_CODE (temp) == ASHIFTRT
    6039        16573 :           && CONST_INT_P (XEXP (temp, 1))
    6040       428706 :           && INTVAL (XEXP (temp, 1)) == GET_MODE_UNIT_PRECISION (mode) - 1)
    6041            0 :         return simplify_shift_const (NULL_RTX, LSHIFTRT, mode, XEXP (temp, 0),
    6042            0 :                                      INTVAL (XEXP (temp, 1)));
    6043              : 
    6044              :       /* If X has only a single bit that might be nonzero, say, bit I, convert
    6045              :          (neg X) to (ashiftrt (ashift X C-I) C-I) where C is the bitsize of
    6046              :          MODE minus 1.  This will convert (neg (zero_extract X 1 Y)) to
    6047              :          (sign_extract X 1 Y).  But only do this if TEMP isn't a register
    6048              :          or a SUBREG of one since we'd be making the expression more
    6049              :          complex if it was just a register.  */
    6050              : 
    6051       395634 :       if (!REG_P (temp)
    6052       190307 :           && ! (GET_CODE (temp) == SUBREG
    6053        19907 :                 && REG_P (SUBREG_REG (temp)))
    6054    229649981 :           && is_a <scalar_int_mode> (mode, &int_mode)
    6055       539973 :           && (i = exact_log2 (nonzero_bits (temp, int_mode))) >= 0)
    6056              :         {
    6057        67004 :           rtx temp1 = simplify_shift_const
    6058        67004 :             (NULL_RTX, ASHIFTRT, int_mode,
    6059              :              simplify_shift_const (NULL_RTX, ASHIFT, int_mode, temp,
    6060        67004 :                                    GET_MODE_PRECISION (int_mode) - 1 - i),
    6061        67004 :              GET_MODE_PRECISION (int_mode) - 1 - i);
    6062              : 
    6063              :           /* If all we did was surround TEMP with the two shifts, we
    6064              :              haven't improved anything, so don't use it.  Otherwise,
    6065              :              we are better off with TEMP1.  */
    6066        67004 :           if (GET_CODE (temp1) != ASHIFTRT
    6067        66431 :               || GET_CODE (XEXP (temp1, 0)) != ASHIFT
    6068        66431 :               || XEXP (XEXP (temp1, 0), 0) != temp)
    6069         6804 :             return temp1;
    6070              :         }
    6071              :       break;
    6072              : 
    6073         9837 :     case TRUNCATE:
    6074              :       /* We can't handle truncation to a partial integer mode here
    6075              :          because we don't know the real bitsize of the partial
    6076              :          integer mode.  */
    6077         9837 :       if (GET_MODE_CLASS (mode) == MODE_PARTIAL_INT)
    6078              :         break;
    6079              : 
    6080         9837 :       if (HWI_COMPUTABLE_MODE_P (mode))
    6081            0 :         SUBST (XEXP (x, 0),
    6082              :                force_to_mode (XEXP (x, 0), GET_MODE (XEXP (x, 0)),
    6083              :                               GET_MODE_MASK (mode), false));
    6084              : 
    6085              :       /* We can truncate a constant value and return it.  */
    6086         9837 :       {
    6087         9837 :         poly_int64 c;
    6088         9837 :         if (poly_int_rtx_p (XEXP (x, 0), &c))
    6089            0 :           return gen_int_mode (c, mode);
    6090              :       }
    6091              : 
    6092              :       /* Similarly to what we do in simplify-rtx.cc, a truncate of a register
    6093              :          whose value is a comparison can be replaced with a subreg if
    6094              :          STORE_FLAG_VALUE permits.  */
    6095         9837 :       if (HWI_COMPUTABLE_MODE_P (mode)
    6096            0 :           && (STORE_FLAG_VALUE & ~GET_MODE_MASK (mode)) == 0
    6097            0 :           && (temp = get_last_value (XEXP (x, 0)))
    6098            0 :           && COMPARISON_P (temp)
    6099         9837 :           && TRULY_NOOP_TRUNCATION_MODES_P (mode, GET_MODE (XEXP (x, 0))))
    6100            0 :         return gen_lowpart (mode, XEXP (x, 0));
    6101              :       break;
    6102              : 
    6103         5402 :     case CONST:
    6104              :       /* (const (const X)) can become (const X).  Do it this way rather than
    6105              :          returning the inner CONST since CONST can be shared with a
    6106              :          REG_EQUAL note.  */
    6107         5402 :       if (GET_CODE (XEXP (x, 0)) == CONST)
    6108            0 :         SUBST (XEXP (x, 0), XEXP (XEXP (x, 0), 0));
    6109              :       break;
    6110              : 
    6111              :     case LO_SUM:
    6112              :       /* Convert (lo_sum (high FOO) FOO) to FOO.  This is necessary so we
    6113              :          can add in an offset.  find_split_point will split this address up
    6114              :          again if it doesn't match.  */
    6115              :       if (HAVE_lo_sum && GET_CODE (XEXP (x, 0)) == HIGH
    6116              :           && rtx_equal_p (XEXP (XEXP (x, 0), 0), XEXP (x, 1)))
    6117              :         return XEXP (x, 1);
    6118              :       break;
    6119              : 
    6120     35118701 :     case PLUS:
    6121              :       /* (plus (xor (and <foo> (const_int pow2 - 1)) <c>) <-c>)
    6122              :          when c is (const_int (pow2 + 1) / 2) is a sign extension of a
    6123              :          bit-field and can be replaced by either a sign_extend or a
    6124              :          sign_extract.  The `and' may be a zero_extend and the two
    6125              :          <c>, -<c> constants may be reversed.  */
    6126     35118701 :       if (GET_CODE (XEXP (x, 0)) == XOR
    6127     35118701 :           && is_a <scalar_int_mode> (mode, &int_mode)
    6128        13839 :           && CONST_INT_P (XEXP (x, 1))
    6129         3839 :           && CONST_INT_P (XEXP (XEXP (x, 0), 1))
    6130         3180 :           && INTVAL (XEXP (x, 1)) == -INTVAL (XEXP (XEXP (x, 0), 1))
    6131           77 :           && ((i = exact_log2 (UINTVAL (XEXP (XEXP (x, 0), 1)))) >= 0
    6132            2 :               || (i = exact_log2 (UINTVAL (XEXP (x, 1)))) >= 0)
    6133           39 :           && HWI_COMPUTABLE_MODE_P (int_mode)
    6134     35118740 :           && ((GET_CODE (XEXP (XEXP (x, 0), 0)) == AND
    6135            0 :                && CONST_INT_P (XEXP (XEXP (XEXP (x, 0), 0), 1))
    6136            0 :                && (UINTVAL (XEXP (XEXP (XEXP (x, 0), 0), 1))
    6137            0 :                    == (HOST_WIDE_INT_1U << (i + 1)) - 1))
    6138           39 :               || (GET_CODE (XEXP (XEXP (x, 0), 0)) == ZERO_EXTEND
    6139            0 :                   && known_eq ((GET_MODE_PRECISION
    6140              :                                 (GET_MODE (XEXP (XEXP (XEXP (x, 0), 0), 0)))),
    6141              :                                (unsigned int) i + 1))))
    6142            0 :         return simplify_shift_const
    6143            0 :           (NULL_RTX, ASHIFTRT, int_mode,
    6144              :            simplify_shift_const (NULL_RTX, ASHIFT, int_mode,
    6145              :                                  XEXP (XEXP (XEXP (x, 0), 0), 0),
    6146            0 :                                  GET_MODE_PRECISION (int_mode) - (i + 1)),
    6147            0 :            GET_MODE_PRECISION (int_mode) - (i + 1));
    6148              : 
    6149              :       /* If only the low-order bit of X is possibly nonzero, (plus x -1)
    6150              :          can become (ashiftrt (ashift (xor x 1) C) C) where C is
    6151              :          the bitsize of the mode - 1.  This allows simplification of
    6152              :          "a = (b & 8) == 0;"  */
    6153     35118701 :       if (XEXP (x, 1) == constm1_rtx
    6154       744429 :           && !REG_P (XEXP (x, 0))
    6155       328779 :           && ! (GET_CODE (XEXP (x, 0)) == SUBREG
    6156        34050 :                 && REG_P (SUBREG_REG (XEXP (x, 0))))
    6157     35409781 :           && is_a <scalar_int_mode> (mode, &int_mode)
    6158     35419712 :           && nonzero_bits (XEXP (x, 0), int_mode) == 1)
    6159         9931 :         return simplify_shift_const
    6160         9931 :           (NULL_RTX, ASHIFTRT, int_mode,
    6161              :            simplify_shift_const (NULL_RTX, ASHIFT, int_mode,
    6162              :                                  gen_rtx_XOR (int_mode, XEXP (x, 0),
    6163              :                                               const1_rtx),
    6164         9931 :                                  GET_MODE_PRECISION (int_mode) - 1),
    6165        19862 :            GET_MODE_PRECISION (int_mode) - 1);
    6166              : 
    6167              :       /* If we are adding two things that have no bits in common, convert
    6168              :          the addition into an IOR.  This will often be further simplified,
    6169              :          for example in cases like ((a & 1) + (a & 2)), which can
    6170              :          become a & 3.  */
    6171              : 
    6172     35108770 :       if (HWI_COMPUTABLE_MODE_P (mode)
    6173     31219618 :           && (nonzero_bits (XEXP (x, 0), mode)
    6174     31219618 :               & nonzero_bits (XEXP (x, 1), mode)) == 0)
    6175              :         {
    6176              :           /* Try to simplify the expression further.  */
    6177       966434 :           rtx tor = simplify_gen_binary (IOR, mode, XEXP (x, 0), XEXP (x, 1));
    6178       966434 :           temp = combine_simplify_rtx (tor, VOIDmode, in_dest, false);
    6179              : 
    6180              :           /* If we could, great.  If not, do not go ahead with the IOR
    6181              :              replacement, since PLUS appears in many special purpose
    6182              :              address arithmetic instructions.  */
    6183       966434 :           if (GET_CODE (temp) != CLOBBER
    6184       966434 :               && (GET_CODE (temp) != IOR
    6185       960969 :                   || ((XEXP (temp, 0) != XEXP (x, 0)
    6186       950258 :                        || XEXP (temp, 1) != XEXP (x, 1))
    6187        10711 :                       && (XEXP (temp, 0) != XEXP (x, 1)
    6188           50 :                           || XEXP (temp, 1) != XEXP (x, 0)))))
    6189              :             return temp;
    6190              :         }
    6191              : 
    6192              :       /* Canonicalize x + x into x << 1.  */
    6193     35092644 :       if (GET_MODE_CLASS (mode) == MODE_INT
    6194     31531751 :           && rtx_equal_p (XEXP (x, 0), XEXP (x, 1))
    6195     35095840 :           && !side_effects_p (XEXP (x, 0)))
    6196         3183 :         return simplify_gen_binary (ASHIFT, mode, XEXP (x, 0), const1_rtx);
    6197              : 
    6198              :       break;
    6199              : 
    6200      4168471 :     case MINUS:
    6201              :       /* (minus <foo> (and <foo> (const_int -pow2))) becomes
    6202              :          (and <foo> (const_int pow2-1))  */
    6203      4168471 :       if (is_a <scalar_int_mode> (mode, &int_mode)
    6204      3518228 :           && GET_CODE (XEXP (x, 1)) == AND
    6205        97263 :           && CONST_INT_P (XEXP (XEXP (x, 1), 1))
    6206        94551 :           && pow2p_hwi (-UINTVAL (XEXP (XEXP (x, 1), 1)))
    6207        48677 :           && rtx_equal_p (XEXP (XEXP (x, 1), 0), XEXP (x, 0)))
    6208            0 :         return simplify_and_const_int (NULL_RTX, int_mode, XEXP (x, 0),
    6209            0 :                                        -INTVAL (XEXP (XEXP (x, 1), 1)) - 1);
    6210              :       break;
    6211              : 
    6212      3295037 :     case MULT:
    6213              :       /* If we have (mult (plus A B) C), apply the distributive law and then
    6214              :          the inverse distributive law to see if things simplify.  This
    6215              :          occurs mostly in addresses, often when unrolling loops.  */
    6216              : 
    6217      3295037 :       if (GET_CODE (XEXP (x, 0)) == PLUS)
    6218              :         {
    6219       282875 :           rtx result = distribute_and_simplify_rtx (x, 0);
    6220       282875 :           if (result)
    6221              :             return result;
    6222              :         }
    6223              : 
    6224              :       /* Try simplify a*(b/c) as (a*b)/c.  */
    6225      3294215 :       if (FLOAT_MODE_P (mode) && flag_associative_math
    6226       203880 :           && GET_CODE (XEXP (x, 0)) == DIV)
    6227              :         {
    6228          244 :           rtx tem = simplify_binary_operation (MULT, mode,
    6229              :                                                XEXP (XEXP (x, 0), 0),
    6230              :                                                XEXP (x, 1));
    6231          244 :           if (tem)
    6232           32 :             return simplify_gen_binary (DIV, mode, tem, XEXP (XEXP (x, 0), 1));
    6233              :         }
    6234              :       break;
    6235              : 
    6236       118794 :     case UDIV:
    6237              :       /* If this is a divide by a power of two, treat it as a shift if
    6238              :          its first operand is a shift.  */
    6239       118794 :       if (is_a <scalar_int_mode> (mode, &int_mode)
    6240       118794 :           && CONST_INT_P (XEXP (x, 1))
    6241         2006 :           && (i = exact_log2 (UINTVAL (XEXP (x, 1)))) >= 0
    6242            0 :           && (GET_CODE (XEXP (x, 0)) == ASHIFT
    6243            0 :               || GET_CODE (XEXP (x, 0)) == LSHIFTRT
    6244            0 :               || GET_CODE (XEXP (x, 0)) == ASHIFTRT
    6245            0 :               || GET_CODE (XEXP (x, 0)) == ROTATE
    6246            0 :               || GET_CODE (XEXP (x, 0)) == ROTATERT))
    6247            0 :         return simplify_shift_const (NULL_RTX, LSHIFTRT, int_mode,
    6248            0 :                                      XEXP (x, 0), i);
    6249              :       break;
    6250              : 
    6251     18366116 :     case EQ:  case NE:
    6252     18366116 :     case GT:  case GTU:  case GE:  case GEU:
    6253     18366116 :     case LT:  case LTU:  case LE:  case LEU:
    6254     18366116 :     case UNEQ:  case LTGT:
    6255     18366116 :     case UNGT:  case UNGE:
    6256     18366116 :     case UNLT:  case UNLE:
    6257     18366116 :     case UNORDERED: case ORDERED:
    6258              :       /* If the first operand is a condition code, we can't do anything
    6259              :          with it.  */
    6260     18366116 :       if (GET_CODE (XEXP (x, 0)) == COMPARE
    6261     18366116 :           || GET_MODE_CLASS (GET_MODE (XEXP (x, 0))) != MODE_CC)
    6262              :         {
    6263     13802910 :           rtx op0 = XEXP (x, 0);
    6264     13802910 :           rtx op1 = XEXP (x, 1);
    6265     13802910 :           enum rtx_code new_code;
    6266              : 
    6267     13802910 :           if (GET_CODE (op0) == COMPARE)
    6268            0 :             op1 = XEXP (op0, 1), op0 = XEXP (op0, 0);
    6269              : 
    6270              :           /* Simplify our comparison, if possible.  */
    6271     13802910 :           new_code = simplify_comparison (code, &op0, &op1);
    6272              : 
    6273              :           /* If STORE_FLAG_VALUE is 1, we can convert (ne x 0) to simply X
    6274              :              if only the low-order bit is possibly nonzero in X (such as when
    6275              :              X is a ZERO_EXTRACT of one bit).  Similarly, we can convert EQ to
    6276              :              (xor X 1) or (minus 1 X); we use the former.  Finally, if X is
    6277              :              known to be either 0 or -1, NE becomes a NEG and EQ becomes
    6278              :              (plus X 1).
    6279              : 
    6280              :              Remove any ZERO_EXTRACT we made when thinking this was a
    6281              :              comparison.  It may now be simpler to use, e.g., an AND.  If a
    6282              :              ZERO_EXTRACT is indeed appropriate, it will be placed back by
    6283              :              the call to make_compound_operation in the SET case.
    6284              : 
    6285              :              Don't apply these optimizations if the caller would
    6286              :              prefer a comparison rather than a value.
    6287              :              E.g., for the condition in an IF_THEN_ELSE most targets need
    6288              :              an explicit comparison.  */
    6289              : 
    6290     13802910 :           if (in_cond)
    6291              :             ;
    6292              : 
    6293      2168953 :           else if (STORE_FLAG_VALUE == 1
    6294              :                    && new_code == NE
    6295      2615576 :                    && is_int_mode (mode, &int_mode)
    6296       446838 :                    && op1 == const0_rtx
    6297       232412 :                    && int_mode == GET_MODE (op0)
    6298      2265327 :                    && nonzero_bits (op0, int_mode) == 1)
    6299          215 :             return gen_lowpart (int_mode,
    6300       471613 :                                 expand_compound_operation (op0));
    6301              : 
    6302      2168738 :           else if (STORE_FLAG_VALUE == 1
    6303              :                    && new_code == NE
    6304      2614334 :                    && is_int_mode (mode, &int_mode)
    6305       446623 :                    && op1 == const0_rtx
    6306       232197 :                    && int_mode == GET_MODE (op0)
    6307      2264897 :                    && (num_sign_bit_copies (op0, int_mode)
    6308        96159 :                        == GET_MODE_PRECISION (int_mode)))
    6309              :             {
    6310         1027 :               op0 = expand_compound_operation (op0);
    6311         1027 :               return simplify_gen_unary (NEG, int_mode,
    6312         1027 :                                          gen_lowpart (int_mode, op0),
    6313         1027 :                                          int_mode);
    6314              :             }
    6315              : 
    6316      2167711 :           else if (STORE_FLAG_VALUE == 1
    6317              :                    && new_code == EQ
    6318      2506353 :                    && is_int_mode (mode, &int_mode)
    6319       340635 :                    && op1 == const0_rtx
    6320       136394 :                    && int_mode == GET_MODE (op0)
    6321      2212038 :                    && nonzero_bits (op0, int_mode) == 1)
    6322              :             {
    6323         1993 :               op0 = expand_compound_operation (op0);
    6324         1993 :               return simplify_gen_binary (XOR, int_mode,
    6325         1993 :                                           gen_lowpart (int_mode, op0),
    6326         1993 :                                           const1_rtx);
    6327              :             }
    6328              : 
    6329      2165718 :           else if (STORE_FLAG_VALUE == 1
    6330              :                    && new_code == EQ
    6331     14137753 :                    && is_int_mode (mode, &int_mode)
    6332       338642 :                    && op1 == const0_rtx
    6333       134401 :                    && int_mode == GET_MODE (op0)
    6334      2208052 :                    && (num_sign_bit_copies (op0, int_mode)
    6335        42334 :                        == GET_MODE_PRECISION (int_mode)))
    6336              :             {
    6337          564 :               op0 = expand_compound_operation (op0);
    6338          564 :               return plus_constant (int_mode, gen_lowpart (int_mode, op0), 1);
    6339              :             }
    6340              : 
    6341              :           /* If STORE_FLAG_VALUE is -1, we have cases similar to
    6342              :              those above.  */
    6343     13799111 :           if (in_cond)
    6344              :             ;
    6345              : 
    6346     13799111 :           else if (STORE_FLAG_VALUE == -1
    6347              :                    && new_code == NE
    6348              :                    && is_int_mode (mode, &int_mode)
    6349              :                    && op1 == const0_rtx
    6350              :                    && int_mode == GET_MODE (op0)
    6351              :                    && (num_sign_bit_copies (op0, int_mode)
    6352              :                        == GET_MODE_PRECISION (int_mode)))
    6353              :             return gen_lowpart (int_mode, expand_compound_operation (op0));
    6354              : 
    6355     13799111 :           else if (STORE_FLAG_VALUE == -1
    6356              :                    && new_code == NE
    6357              :                    && is_int_mode (mode, &int_mode)
    6358              :                    && op1 == const0_rtx
    6359              :                    && int_mode == GET_MODE (op0)
    6360              :                    && nonzero_bits (op0, int_mode) == 1)
    6361              :             {
    6362              :               op0 = expand_compound_operation (op0);
    6363              :               return simplify_gen_unary (NEG, int_mode,
    6364              :                                          gen_lowpart (int_mode, op0),
    6365              :                                          int_mode);
    6366              :             }
    6367              : 
    6368     13799111 :           else if (STORE_FLAG_VALUE == -1
    6369              :                    && new_code == EQ
    6370              :                    && is_int_mode (mode, &int_mode)
    6371              :                    && op1 == const0_rtx
    6372              :                    && int_mode == GET_MODE (op0)
    6373              :                    && (num_sign_bit_copies (op0, int_mode)
    6374              :                        == GET_MODE_PRECISION (int_mode)))
    6375              :             {
    6376              :               op0 = expand_compound_operation (op0);
    6377              :               return simplify_gen_unary (NOT, int_mode,
    6378              :                                          gen_lowpart (int_mode, op0),
    6379              :                                          int_mode);
    6380              :             }
    6381              : 
    6382              :           /* If X is 0/1, (eq X 0) is X-1.  */
    6383     13799111 :           else if (STORE_FLAG_VALUE == -1
    6384              :                    && new_code == EQ
    6385              :                    && is_int_mode (mode, &int_mode)
    6386              :                    && op1 == const0_rtx
    6387              :                    && int_mode == GET_MODE (op0)
    6388              :                    && nonzero_bits (op0, int_mode) == 1)
    6389              :             {
    6390              :               op0 = expand_compound_operation (op0);
    6391              :               return plus_constant (int_mode, gen_lowpart (int_mode, op0), -1);
    6392              :             }
    6393              : 
    6394              :           /* If STORE_FLAG_VALUE says to just test the sign bit and X has just
    6395              :              one bit that might be nonzero, we can convert (ne x 0) to
    6396              :              (ashift x c) where C puts the bit in the sign bit.  Remove any
    6397              :              AND with STORE_FLAG_VALUE when we are done, since we are only
    6398              :              going to test the sign bit.  */
    6399     13799111 :           if (new_code == NE
    6400     14240369 :               && is_int_mode (mode, &int_mode)
    6401       445666 :               && HWI_COMPUTABLE_MODE_P (int_mode)
    6402       441258 :               && val_signbit_p (int_mode, STORE_FLAG_VALUE)
    6403            0 :               && op1 == const0_rtx
    6404            0 :               && int_mode == GET_MODE (op0)
    6405     13799111 :               && (i = exact_log2 (nonzero_bits (op0, int_mode))) >= 0)
    6406              :             {
    6407            0 :               x = simplify_shift_const (NULL_RTX, ASHIFT, int_mode,
    6408              :                                         expand_compound_operation (op0),
    6409            0 :                                         GET_MODE_PRECISION (int_mode) - 1 - i);
    6410            0 :               if (GET_CODE (x) == AND && XEXP (x, 1) == const_true_rtx)
    6411            0 :                 return XEXP (x, 0);
    6412              :               else
    6413              :                 return x;
    6414              :             }
    6415              : 
    6416              :           /* If the code changed, return a whole new comparison.
    6417              :              We also need to avoid using SUBST in cases where
    6418              :              simplify_comparison has widened a comparison with a CONST_INT,
    6419              :              since in that case the wider CONST_INT may fail the sanity
    6420              :              checks in do_SUBST.  */
    6421     13799111 :           if (new_code != code
    6422     13339246 :               || (CONST_INT_P (op1)
    6423      7611197 :                   && GET_MODE (op0) != GET_MODE (XEXP (x, 0))
    6424         9338 :                   && GET_MODE (op0) != GET_MODE (XEXP (x, 1))))
    6425       467814 :             return gen_rtx_fmt_ee (new_code, mode, op0, op1);
    6426              : 
    6427              :           /* Otherwise, keep this operation, but maybe change its operands.
    6428              :              This also converts (ne (compare FOO BAR) 0) to (ne FOO BAR).  */
    6429     13331297 :           SUBST (XEXP (x, 0), op0);
    6430     13331297 :           SUBST (XEXP (x, 1), op1);
    6431              :         }
    6432              :       break;
    6433              : 
    6434     13487332 :     case IF_THEN_ELSE:
    6435     13487332 :       return simplify_if_then_else (x);
    6436              : 
    6437      4924210 :     case ZERO_EXTRACT:
    6438      4924210 :     case SIGN_EXTRACT:
    6439      4924210 :     case ZERO_EXTEND:
    6440      4924210 :     case SIGN_EXTEND:
    6441              :       /* If we are processing SET_DEST, we are done.  */
    6442      4924210 :       if (in_dest)
    6443              :         return x;
    6444              : 
    6445      4921415 :       return expand_compound_operation (x);
    6446              : 
    6447     48110788 :     case SET:
    6448     48110788 :       return simplify_set (x);
    6449              : 
    6450     11531057 :     case AND:
    6451     11531057 :     case IOR:
    6452     11531057 :       return simplify_logical (x);
    6453              : 
    6454     13663437 :     case ASHIFT:
    6455     13663437 :     case LSHIFTRT:
    6456     13663437 :     case ASHIFTRT:
    6457     13663437 :     case ROTATE:
    6458     13663437 :     case ROTATERT:
    6459              :       /* If this is a shift by a constant amount, simplify it.  */
    6460     13663437 :       if (CONST_INT_P (XEXP (x, 1)))
    6461     13160239 :         return simplify_shift_const (x, code, mode, XEXP (x, 0),
    6462     13160239 :                                      INTVAL (XEXP (x, 1)));
    6463              : 
    6464              :       else if (SHIFT_COUNT_TRUNCATED && !REG_P (XEXP (x, 1)))
    6465              :         SUBST (XEXP (x, 1),
    6466              :                force_to_mode (XEXP (x, 1), GET_MODE (XEXP (x, 1)),
    6467              :                               (HOST_WIDE_INT_1U
    6468              :                                << exact_log2 (GET_MODE_UNIT_BITSIZE
    6469              :                                               (GET_MODE (x)))) - 1, false));
    6470              :       break;
    6471      2039899 :     case VEC_SELECT:
    6472      2039899 :       {
    6473      2039899 :         rtx trueop0 = XEXP (x, 0);
    6474      2039899 :         mode = GET_MODE (trueop0);
    6475      2039899 :         rtx trueop1 = XEXP (x, 1);
    6476              :         /* If we select a low-part subreg, return that.  */
    6477      2039899 :         if (vec_series_lowpart_p (GET_MODE (x), mode, trueop1))
    6478              :           {
    6479         1173 :             rtx new_rtx = lowpart_subreg (GET_MODE (x), trueop0, mode);
    6480         1173 :             if (new_rtx != NULL_RTX)
    6481         1173 :               return new_rtx;
    6482              :           }
    6483              :       }
    6484              : 
    6485              :     default:
    6486              :       break;
    6487              :     }
    6488              : 
    6489              :   return x;
    6490              : }
    6491              : 
    6492              : /* Simplify X, an IF_THEN_ELSE expression.  Return the new expression.  */
    6493              : 
    6494              : static rtx
    6495     13487332 : simplify_if_then_else (rtx x)
    6496              : {
    6497     13487332 :   machine_mode mode = GET_MODE (x);
    6498     13487332 :   rtx cond = XEXP (x, 0);
    6499     13487332 :   rtx true_rtx = XEXP (x, 1);
    6500     13487332 :   rtx false_rtx = XEXP (x, 2);
    6501     13487332 :   enum rtx_code true_code = GET_CODE (cond);
    6502     13487332 :   bool comparison_p = COMPARISON_P (cond);
    6503     13487332 :   rtx temp;
    6504     13487332 :   int i;
    6505     13487332 :   enum rtx_code false_code;
    6506     13487332 :   rtx reversed;
    6507     13487332 :   scalar_int_mode int_mode, inner_mode;
    6508              : 
    6509              :   /* Simplify storing of the truth value.  */
    6510     13487332 :   if (comparison_p && true_rtx == const_true_rtx && false_rtx == const0_rtx)
    6511            0 :     return simplify_gen_relational (true_code, mode, VOIDmode,
    6512            0 :                                     XEXP (cond, 0), XEXP (cond, 1));
    6513              : 
    6514              :   /* Also when the truth value has to be reversed.  */
    6515     13486787 :   if (comparison_p
    6516     13486787 :       && true_rtx == const0_rtx && false_rtx == const_true_rtx
    6517            0 :       && (reversed = reversed_comparison (cond, mode)))
    6518              :     return reversed;
    6519              : 
    6520              :   /* Sometimes we can simplify the arm of an IF_THEN_ELSE if a register used
    6521              :      in it is being compared against certain values.  Get the true and false
    6522              :      comparisons and see if that says anything about the value of each arm.  */
    6523              : 
    6524     13487332 :   if (comparison_p
    6525     13486787 :       && ((false_code = reversed_comparison_code (cond, NULL))
    6526              :           != UNKNOWN)
    6527     26813287 :       && REG_P (XEXP (cond, 0)))
    6528              :     {
    6529      8339951 :       HOST_WIDE_INT nzb;
    6530      8339951 :       rtx from = XEXP (cond, 0);
    6531      8339951 :       rtx true_val = XEXP (cond, 1);
    6532      8339951 :       rtx false_val = true_val;
    6533      8339951 :       bool swapped = false;
    6534              : 
    6535              :       /* If FALSE_CODE is EQ, swap the codes and arms.  */
    6536              : 
    6537      8339951 :       if (false_code == EQ)
    6538              :         {
    6539      3051531 :           swapped = true, true_code = EQ, false_code = NE;
    6540      3051531 :           std::swap (true_rtx, false_rtx);
    6541              :         }
    6542              : 
    6543      8339951 :       scalar_int_mode from_mode;
    6544      8339951 :       if (is_a <scalar_int_mode> (GET_MODE (from), &from_mode))
    6545              :         {
    6546              :           /* If we are comparing against zero and the expression being
    6547              :              tested has only a single bit that might be nonzero, that is
    6548              :              its value when it is not equal to zero.  Similarly if it is
    6549              :              known to be -1 or 0.  */
    6550      6900910 :           if (true_code == EQ
    6551      5022755 :               && true_val == const0_rtx
    6552      8987520 :               && pow2p_hwi (nzb = nonzero_bits (from, from_mode)))
    6553              :             {
    6554       240554 :               false_code = EQ;
    6555       240554 :               false_val = gen_int_mode (nzb, from_mode);
    6556              :             }
    6557      6660356 :           else if (true_code == EQ
    6558      4782201 :                    && true_val == const0_rtx
    6559      8506412 :                    && (num_sign_bit_copies (from, from_mode)
    6560      1846056 :                        == GET_MODE_PRECISION (from_mode)))
    6561              :             {
    6562          725 :               false_code = EQ;
    6563          725 :               false_val = constm1_rtx;
    6564              :             }
    6565              :         }
    6566              : 
    6567              :       /* Now simplify an arm if we know the value of the register in the
    6568              :          branch and it is used in the arm.  Be careful due to the potential
    6569              :          of locally-shared RTL.  */
    6570              : 
    6571      8339951 :       if (reg_mentioned_p (from, true_rtx))
    6572       320089 :         true_rtx = subst (known_cond (copy_rtx (true_rtx), true_code,
    6573              :                                       from, true_val),
    6574              :                           pc_rtx, pc_rtx, false, false, false);
    6575      8339951 :       if (reg_mentioned_p (from, false_rtx))
    6576       136317 :         false_rtx = subst (known_cond (copy_rtx (false_rtx), false_code,
    6577              :                                        from, false_val),
    6578              :                            pc_rtx, pc_rtx, false, false, false);
    6579              : 
    6580     13628371 :       SUBST (XEXP (x, 1), swapped ? false_rtx : true_rtx);
    6581     13628371 :       SUBST (XEXP (x, 2), swapped ? true_rtx : false_rtx);
    6582              : 
    6583      8339951 :       true_rtx = XEXP (x, 1);
    6584      8339951 :       false_rtx = XEXP (x, 2);
    6585      8339951 :       true_code = GET_CODE (cond);
    6586              :     }
    6587              : 
    6588              :   /* If we have (if_then_else FOO (pc) (label_ref BAR)) and FOO can be
    6589              :      reversed, do so to avoid needing two sets of patterns for
    6590              :      subtract-and-branch insns.  Similarly if we have a constant in the true
    6591              :      arm, the false arm is the same as the first operand of the comparison, or
    6592              :      the false arm is more complicated than the true arm.  */
    6593              : 
    6594     13487332 :   if (comparison_p
    6595     13486787 :       && reversed_comparison_code (cond, NULL) != UNKNOWN
    6596     26813287 :       && (true_rtx == pc_rtx
    6597     13325955 :           || (CONSTANT_P (true_rtx)
    6598     10905763 :               && !CONST_INT_P (false_rtx) && false_rtx != pc_rtx)
    6599     13284979 :           || true_rtx == const0_rtx
    6600     13281985 :           || (OBJECT_P (true_rtx) && !OBJECT_P (false_rtx))
    6601     13214108 :           || (GET_CODE (true_rtx) == SUBREG && OBJECT_P (SUBREG_REG (true_rtx))
    6602        19249 :               && !OBJECT_P (false_rtx))
    6603     13210334 :           || reg_mentioned_p (true_rtx, false_rtx)
    6604     13210238 :           || rtx_equal_p (false_rtx, XEXP (cond, 0))))
    6605              :     {
    6606       170028 :       SUBST (XEXP (x, 0), reversed_comparison (cond, GET_MODE (cond)));
    6607       170028 :       SUBST (XEXP (x, 1), false_rtx);
    6608       170028 :       SUBST (XEXP (x, 2), true_rtx);
    6609              : 
    6610       170028 :       std::swap (true_rtx, false_rtx);
    6611       170028 :       cond = XEXP (x, 0);
    6612              : 
    6613              :       /* It is possible that the conditional has been simplified out.  */
    6614       170028 :       true_code = GET_CODE (cond);
    6615       170028 :       comparison_p = COMPARISON_P (cond);
    6616              :     }
    6617              : 
    6618              :   /* If the two arms are identical, we don't need the comparison.  */
    6619              : 
    6620     13487332 :   if (rtx_equal_p (true_rtx, false_rtx) && ! side_effects_p (cond))
    6621              :     return true_rtx;
    6622              : 
    6623              :   /* Convert a == b ? b : a to "a".  */
    6624      3938910 :   if (true_code == EQ && ! side_effects_p (cond)
    6625      3919675 :       && !HONOR_NANS (mode)
    6626      3916597 :       && rtx_equal_p (XEXP (cond, 0), false_rtx)
    6627     13487793 :       && rtx_equal_p (XEXP (cond, 1), true_rtx))
    6628              :     return false_rtx;
    6629      4916155 :   else if (true_code == NE && ! side_effects_p (cond)
    6630      4867029 :            && !HONOR_NANS (mode)
    6631      4713262 :            && rtx_equal_p (XEXP (cond, 0), true_rtx)
    6632     13553963 :            && rtx_equal_p (XEXP (cond, 1), false_rtx))
    6633              :     return true_rtx;
    6634              : 
    6635              :   /* Look for cases where we have (abs x) or (neg (abs X)).  */
    6636              : 
    6637     13487314 :   if (GET_MODE_CLASS (mode) == MODE_INT
    6638      2228940 :       && comparison_p
    6639      2228927 :       && XEXP (cond, 1) == const0_rtx
    6640      1704837 :       && GET_CODE (false_rtx) == NEG
    6641          159 :       && rtx_equal_p (true_rtx, XEXP (false_rtx, 0))
    6642           24 :       && rtx_equal_p (true_rtx, XEXP (cond, 0))
    6643     13487338 :       && ! side_effects_p (true_rtx))
    6644           24 :     switch (true_code)
    6645              :       {
    6646           24 :       case GT:
    6647           24 :       case GE:
    6648           24 :         return simplify_gen_unary (ABS, mode, true_rtx, mode);
    6649            0 :       case LT:
    6650            0 :       case LE:
    6651            0 :         return
    6652            0 :           simplify_gen_unary (NEG, mode,
    6653              :                               simplify_gen_unary (ABS, mode, true_rtx, mode),
    6654            0 :                               mode);
    6655              :       default:
    6656              :         break;
    6657              :       }
    6658              : 
    6659              :   /* Look for MIN or MAX.  */
    6660              : 
    6661     13487290 :   if ((! FLOAT_MODE_P (mode)
    6662       170662 :        || (flag_unsafe_math_optimizations
    6663          438 :            && !HONOR_NANS (mode)
    6664          438 :            && !HONOR_SIGNED_ZEROS (mode)))
    6665     13317066 :       && comparison_p
    6666     13316696 :       && rtx_equal_p (XEXP (cond, 0), true_rtx)
    6667       138066 :       && rtx_equal_p (XEXP (cond, 1), false_rtx)
    6668        13653 :       && ! side_effects_p (cond))
    6669        13649 :     switch (true_code)
    6670              :       {
    6671         5152 :       case GE:
    6672         5152 :       case GT:
    6673         5152 :         return simplify_gen_binary (SMAX, mode, true_rtx, false_rtx);
    6674         4777 :       case LE:
    6675         4777 :       case LT:
    6676         4777 :         return simplify_gen_binary (SMIN, mode, true_rtx, false_rtx);
    6677         2813 :       case GEU:
    6678         2813 :       case GTU:
    6679         2813 :         return simplify_gen_binary (UMAX, mode, true_rtx, false_rtx);
    6680          907 :       case LEU:
    6681          907 :       case LTU:
    6682          907 :         return simplify_gen_binary (UMIN, mode, true_rtx, false_rtx);
    6683              :       default:
    6684              :         break;
    6685              :       }
    6686              : 
    6687              :   /* If we have (if_then_else COND (OP Z C1) Z) and OP is an identity when its
    6688              :      second operand is zero, this can be done as (OP Z (mult COND C2)) where
    6689              :      C2 = C1 * STORE_FLAG_VALUE. Similarly if OP has an outer ZERO_EXTEND or
    6690              :      SIGN_EXTEND as long as Z is already extended (so we don't destroy it).
    6691              :      We can do this kind of thing in some cases when STORE_FLAG_VALUE is
    6692              :      neither 1 or -1, but it isn't worth checking for.  */
    6693              : 
    6694     13473641 :   if ((STORE_FLAG_VALUE == 1 || STORE_FLAG_VALUE == -1)
    6695              :       && comparison_p
    6696     15601714 :       && is_int_mode (mode, &int_mode)
    6697     15688917 :       && ! side_effects_p (x))
    6698              :     {
    6699      2210979 :       rtx t = make_compound_operation (true_rtx, SET);
    6700      2210979 :       rtx f = make_compound_operation (false_rtx, SET);
    6701      2210979 :       rtx cond_op0 = XEXP (cond, 0);
    6702      2210979 :       rtx cond_op1 = XEXP (cond, 1);
    6703      2210979 :       enum rtx_code op = UNKNOWN, extend_op = UNKNOWN;
    6704      2210979 :       scalar_int_mode m = int_mode;
    6705      2210979 :       rtx z = 0, c1 = NULL_RTX;
    6706              : 
    6707      2210979 :       if ((GET_CODE (t) == PLUS || GET_CODE (t) == MINUS
    6708              :            || GET_CODE (t) == IOR || GET_CODE (t) == XOR
    6709              :            || GET_CODE (t) == ASHIFT
    6710              :            || GET_CODE (t) == LSHIFTRT || GET_CODE (t) == ASHIFTRT)
    6711       239902 :           && rtx_equal_p (XEXP (t, 0), f))
    6712        76385 :         c1 = XEXP (t, 1), op = GET_CODE (t), z = f;
    6713              : 
    6714              :       /* If an identity-zero op is commutative, check whether there
    6715              :          would be a match if we swapped the operands.  */
    6716      2048351 :       else if ((GET_CODE (t) == PLUS || GET_CODE (t) == IOR
    6717      2037959 :                 || GET_CODE (t) == XOR)
    6718      2148035 :                && rtx_equal_p (XEXP (t, 1), f))
    6719        10818 :         c1 = XEXP (t, 0), op = GET_CODE (t), z = f;
    6720      2123776 :       else if (GET_CODE (t) == SIGN_EXTEND
    6721         4619 :                && is_a <scalar_int_mode> (GET_MODE (XEXP (t, 0)), &inner_mode)
    6722         4619 :                && (GET_CODE (XEXP (t, 0)) == PLUS
    6723         4619 :                    || GET_CODE (XEXP (t, 0)) == MINUS
    6724              :                    || GET_CODE (XEXP (t, 0)) == IOR
    6725              :                    || GET_CODE (XEXP (t, 0)) == XOR
    6726              :                    || GET_CODE (XEXP (t, 0)) == ASHIFT
    6727              :                    || GET_CODE (XEXP (t, 0)) == LSHIFTRT
    6728              :                    || GET_CODE (XEXP (t, 0)) == ASHIFTRT)
    6729          164 :                && GET_CODE (XEXP (XEXP (t, 0), 0)) == SUBREG
    6730           86 :                && subreg_lowpart_p (XEXP (XEXP (t, 0), 0))
    6731           86 :                && rtx_equal_p (SUBREG_REG (XEXP (XEXP (t, 0), 0)), f)
    6732      2123776 :                && (num_sign_bit_copies (f, GET_MODE (f))
    6733            0 :                    > (unsigned int)
    6734            0 :                      (GET_MODE_PRECISION (int_mode)
    6735            0 :                       - GET_MODE_PRECISION (inner_mode))))
    6736              :         {
    6737            0 :           c1 = XEXP (XEXP (t, 0), 1); z = f; op = GET_CODE (XEXP (t, 0));
    6738            0 :           extend_op = SIGN_EXTEND;
    6739            0 :           m = inner_mode;
    6740              :         }
    6741      2123776 :       else if (GET_CODE (t) == SIGN_EXTEND
    6742         4619 :                && is_a <scalar_int_mode> (GET_MODE (XEXP (t, 0)), &inner_mode)
    6743         4619 :                && (GET_CODE (XEXP (t, 0)) == PLUS
    6744         4483 :                    || GET_CODE (XEXP (t, 0)) == IOR
    6745         4479 :                    || GET_CODE (XEXP (t, 0)) == XOR)
    6746          140 :                && GET_CODE (XEXP (XEXP (t, 0), 1)) == SUBREG
    6747            4 :                && subreg_lowpart_p (XEXP (XEXP (t, 0), 1))
    6748            4 :                && rtx_equal_p (SUBREG_REG (XEXP (XEXP (t, 0), 1)), f)
    6749      2123780 :                && (num_sign_bit_copies (f, GET_MODE (f))
    6750            4 :                    > (unsigned int)
    6751            4 :                      (GET_MODE_PRECISION (int_mode)
    6752            4 :                       - GET_MODE_PRECISION (inner_mode))))
    6753              :         {
    6754            0 :           c1 = XEXP (XEXP (t, 0), 0); z = f; op = GET_CODE (XEXP (t, 0));
    6755            0 :           extend_op = SIGN_EXTEND;
    6756            0 :           m = inner_mode;
    6757              :         }
    6758      2123776 :       else if (GET_CODE (t) == ZERO_EXTEND
    6759         4968 :                && is_a <scalar_int_mode> (GET_MODE (XEXP (t, 0)), &inner_mode)
    6760         4968 :                && (GET_CODE (XEXP (t, 0)) == PLUS
    6761         4968 :                    || GET_CODE (XEXP (t, 0)) == MINUS
    6762              :                    || GET_CODE (XEXP (t, 0)) == IOR
    6763              :                    || GET_CODE (XEXP (t, 0)) == XOR
    6764              :                    || GET_CODE (XEXP (t, 0)) == ASHIFT
    6765              :                    || GET_CODE (XEXP (t, 0)) == LSHIFTRT
    6766              :                    || GET_CODE (XEXP (t, 0)) == ASHIFTRT)
    6767         1323 :                && GET_CODE (XEXP (XEXP (t, 0), 0)) == SUBREG
    6768           38 :                && HWI_COMPUTABLE_MODE_P (int_mode)
    6769           38 :                && subreg_lowpart_p (XEXP (XEXP (t, 0), 0))
    6770           38 :                && rtx_equal_p (SUBREG_REG (XEXP (XEXP (t, 0), 0)), f)
    6771      2123776 :                && ((nonzero_bits (f, GET_MODE (f))
    6772            0 :                     & ~GET_MODE_MASK (inner_mode))
    6773              :                    == 0))
    6774              :         {
    6775            0 :           c1 = XEXP (XEXP (t, 0), 1); z = f; op = GET_CODE (XEXP (t, 0));
    6776            0 :           extend_op = ZERO_EXTEND;
    6777            0 :           m = inner_mode;
    6778              :         }
    6779      2123776 :       else if (GET_CODE (t) == ZERO_EXTEND
    6780         4968 :                && is_a <scalar_int_mode> (GET_MODE (XEXP (t, 0)), &inner_mode)
    6781         4968 :                && (GET_CODE (XEXP (t, 0)) == PLUS
    6782         4138 :                    || GET_CODE (XEXP (t, 0)) == IOR
    6783         4138 :                    || GET_CODE (XEXP (t, 0)) == XOR)
    6784          830 :                && GET_CODE (XEXP (XEXP (t, 0), 1)) == SUBREG
    6785           18 :                && HWI_COMPUTABLE_MODE_P (int_mode)
    6786           18 :                && subreg_lowpart_p (XEXP (XEXP (t, 0), 1))
    6787           18 :                && rtx_equal_p (SUBREG_REG (XEXP (XEXP (t, 0), 1)), f)
    6788      2123776 :                && ((nonzero_bits (f, GET_MODE (f))
    6789            0 :                     & ~GET_MODE_MASK (inner_mode))
    6790              :                    == 0))
    6791              :         {
    6792            0 :           c1 = XEXP (XEXP (t, 0), 0); z = f; op = GET_CODE (XEXP (t, 0));
    6793            0 :           extend_op = ZERO_EXTEND;
    6794            0 :           m = inner_mode;
    6795              :         }
    6796              : 
    6797        87203 :       if (z)
    6798              :         {
    6799        87203 :           machine_mode cm = m;
    6800        87203 :           if ((op == ASHIFT || op == LSHIFTRT || op == ASHIFTRT)
    6801         2491 :               && GET_MODE (c1) != VOIDmode)
    6802         1855 :             cm = GET_MODE (c1);
    6803        87203 :           temp = subst (simplify_gen_relational (true_code, cm, VOIDmode,
    6804              :                                                  cond_op0, cond_op1),
    6805              :                         pc_rtx, pc_rtx, false, false, false);
    6806        87203 :           temp = simplify_gen_binary (MULT, cm, temp,
    6807              :                                       simplify_gen_binary (MULT, cm, c1,
    6808              :                                                            const_true_rtx));
    6809        87203 :           temp = subst (temp, pc_rtx, pc_rtx, false, false, false);
    6810        87203 :           temp = simplify_gen_binary (op, m, gen_lowpart (m, z), temp);
    6811              : 
    6812        87203 :           if (extend_op != UNKNOWN)
    6813            0 :             temp = simplify_gen_unary (extend_op, int_mode, temp, m);
    6814              : 
    6815     13487332 :           return temp;
    6816              :         }
    6817              :     }
    6818              : 
    6819              :   /* If we have (if_then_else (ne A 0) C1 0) and either A is known to be 0 or
    6820              :      1 and C1 is a single bit or A is known to be 0 or -1 and C1 is the
    6821              :      negation of a single bit, we can convert this operation to a shift.  We
    6822              :      can actually do this more generally, but it doesn't seem worth it.  */
    6823              : 
    6824     13386438 :   if (true_code == NE
    6825     13386438 :       && is_a <scalar_int_mode> (mode, &int_mode)
    6826       450660 :       && XEXP (cond, 1) == const0_rtx
    6827       335895 :       && false_rtx == const0_rtx
    6828        51981 :       && CONST_INT_P (true_rtx)
    6829     13388474 :       && ((nonzero_bits (XEXP (cond, 0), int_mode) == 1
    6830            0 :            && (i = exact_log2 (UINTVAL (true_rtx))) >= 0)
    6831         2036 :           || ((num_sign_bit_copies (XEXP (cond, 0), int_mode)
    6832         2036 :                == GET_MODE_PRECISION (int_mode))
    6833            0 :               && (i = exact_log2 (-UINTVAL (true_rtx))) >= 0)))
    6834            0 :     return
    6835            0 :       simplify_shift_const (NULL_RTX, ASHIFT, int_mode,
    6836            0 :                             gen_lowpart (int_mode, XEXP (cond, 0)), i);
    6837              : 
    6838              :   /* (IF_THEN_ELSE (NE A 0) C1 0) is A or a zero-extend of A if the only
    6839              :      non-zero bit in A is C1.  */
    6840      4898582 :   if (true_code == NE && XEXP (cond, 1) == const0_rtx
    6841      2305899 :       && false_rtx == const0_rtx && CONST_INT_P (true_rtx)
    6842     13489368 :       && is_a <scalar_int_mode> (mode, &int_mode)
    6843         2036 :       && is_a <scalar_int_mode> (GET_MODE (XEXP (cond, 0)), &inner_mode)
    6844            5 :       && (UINTVAL (true_rtx) & GET_MODE_MASK (int_mode))
    6845            5 :           == nonzero_bits (XEXP (cond, 0), inner_mode)
    6846     13386438 :       && (i = exact_log2 (UINTVAL (true_rtx) & GET_MODE_MASK (int_mode))) >= 0)
    6847              :     {
    6848            0 :       rtx val = XEXP (cond, 0);
    6849            0 :       if (inner_mode == int_mode)
    6850              :         return val;
    6851            0 :       else if (GET_MODE_PRECISION (inner_mode) < GET_MODE_PRECISION (int_mode))
    6852            0 :         return simplify_gen_unary (ZERO_EXTEND, int_mode, val, inner_mode);
    6853              :     }
    6854              : 
    6855              :   return x;
    6856              : }
    6857              : 
    6858              : /* Simplify X, a SET expression.  Return the new expression.  */
    6859              : 
    6860              : static rtx
    6861     48110788 : simplify_set (rtx x)
    6862              : {
    6863     48110788 :   rtx src = SET_SRC (x);
    6864     48110788 :   rtx dest = SET_DEST (x);
    6865    107624146 :   machine_mode mode
    6866     48110788 :     = GET_MODE (src) != VOIDmode ? GET_MODE (src) : GET_MODE (dest);
    6867     48110788 :   rtx_insn *other_insn;
    6868     48110788 :   rtx *cc_use;
    6869     48110788 :   scalar_int_mode int_mode;
    6870              : 
    6871              :   /* (set (pc) (return)) gets written as (return).  */
    6872     48110788 :   if (GET_CODE (dest) == PC && ANY_RETURN_P (src))
    6873              :     return src;
    6874              : 
    6875              :   /* Now that we know for sure which bits of SRC we are using, see if we can
    6876              :      simplify the expression for the object knowing that we only need the
    6877              :      low-order bits.  */
    6878              : 
    6879     48110788 :   if (GET_MODE_CLASS (mode) == MODE_INT && HWI_COMPUTABLE_MODE_P (mode))
    6880              :     {
    6881     21531992 :       src = force_to_mode (src, mode, HOST_WIDE_INT_M1U, false);
    6882     21531992 :       SUBST (SET_SRC (x), src);
    6883              :     }
    6884              : 
    6885              :   /* If the source is a COMPARE, look for the use of the comparison result
    6886              :      and try to simplify it unless we already have used undobuf.other_insn.  */
    6887     41326129 :   if ((GET_MODE_CLASS (mode) == MODE_CC || GET_CODE (src) == COMPARE)
    6888      6784659 :       && (cc_use = find_single_use (dest, subst_insn, &other_insn)) != 0
    6889      6162605 :       && (undobuf.other_insn == 0 || other_insn == undobuf.other_insn)
    6890      6162605 :       && COMPARISON_P (*cc_use)
    6891     54272877 :       && rtx_equal_p (XEXP (*cc_use, 0), dest))
    6892              :     {
    6893      6160248 :       enum rtx_code old_code = GET_CODE (*cc_use);
    6894      6160248 :       enum rtx_code new_code;
    6895      6160248 :       rtx op0, op1, tmp;
    6896      6160248 :       bool other_changed = false;
    6897      6160248 :       rtx inner_compare = NULL_RTX;
    6898      6160248 :       machine_mode compare_mode = GET_MODE (dest);
    6899              : 
    6900      6160248 :       if (GET_CODE (src) == COMPARE)
    6901              :         {
    6902      5709359 :           op0 = XEXP (src, 0), op1 = XEXP (src, 1);
    6903      5709359 :           if (GET_CODE (op0) == COMPARE && op1 == const0_rtx)
    6904              :             {
    6905            0 :               inner_compare = op0;
    6906            0 :               op0 = XEXP (inner_compare, 0), op1 = XEXP (inner_compare, 1);
    6907              :             }
    6908              :         }
    6909              :       else
    6910       450889 :         op0 = src, op1 = CONST0_RTX (GET_MODE (src));
    6911              : 
    6912      6160248 :       tmp = simplify_relational_operation (old_code, compare_mode, VOIDmode,
    6913              :                                            op0, op1);
    6914      6160248 :       if (!tmp)
    6915              :         new_code = old_code;
    6916       489840 :       else if (!CONSTANT_P (tmp))
    6917              :         {
    6918       485079 :           new_code = GET_CODE (tmp);
    6919       485079 :           op0 = XEXP (tmp, 0);
    6920       485079 :           op1 = XEXP (tmp, 1);
    6921              :         }
    6922              :       else
    6923              :         {
    6924         4761 :           rtx pat = PATTERN (other_insn);
    6925         4761 :           undobuf.other_insn = other_insn;
    6926         4761 :           SUBST (*cc_use, tmp);
    6927              : 
    6928              :           /* Attempt to simplify CC user.  */
    6929         4761 :           if (GET_CODE (pat) == SET)
    6930              :             {
    6931         4241 :               rtx new_rtx = simplify_rtx (SET_SRC (pat));
    6932         4241 :               if (new_rtx != NULL_RTX)
    6933         3804 :                 SUBST (SET_SRC (pat), new_rtx);
    6934              :             }
    6935              : 
    6936              :           /* Convert X into a no-op move.  */
    6937         4761 :           SUBST (SET_DEST (x), pc_rtx);
    6938         4761 :           SUBST (SET_SRC (x), pc_rtx);
    6939         4761 :           return x;
    6940              :         }
    6941              : 
    6942              :       /* Simplify our comparison, if possible.  */
    6943      6155487 :       new_code = simplify_comparison (new_code, &op0, &op1);
    6944              : 
    6945              : #ifdef SELECT_CC_MODE
    6946              :       /* If this machine has CC modes other than CCmode, check to see if we
    6947              :          need to use a different CC mode here.  */
    6948      6155487 :       if (GET_MODE_CLASS (GET_MODE (op0)) == MODE_CC)
    6949       689325 :         compare_mode = GET_MODE (op0);
    6950      5466162 :       else if (inner_compare
    6951            0 :                && GET_MODE_CLASS (GET_MODE (inner_compare)) == MODE_CC
    6952            0 :                && new_code == old_code
    6953            0 :                && op0 == XEXP (inner_compare, 0)
    6954            0 :                && op1 == XEXP (inner_compare, 1))
    6955            0 :         compare_mode = GET_MODE (inner_compare);
    6956              :       else
    6957      5466162 :         compare_mode = SELECT_CC_MODE (new_code, op0, op1);
    6958              : 
    6959              :       /* If the mode changed, we have to change SET_DEST, the mode in the
    6960              :          compare, and the mode in the place SET_DEST is used.  If SET_DEST is
    6961              :          a hard register, just build new versions with the proper mode.  If it
    6962              :          is a pseudo, we lose unless it is only time we set the pseudo, in
    6963              :          which case we can safely change its mode.  */
    6964      6155487 :       if (compare_mode != GET_MODE (dest))
    6965              :         {
    6966       213802 :           if (can_change_dest_mode (dest, 0, compare_mode))
    6967              :             {
    6968       213802 :               unsigned int regno = REGNO (dest);
    6969       213802 :               rtx new_dest;
    6970              : 
    6971       213802 :               if (regno < FIRST_PSEUDO_REGISTER)
    6972       213802 :                 new_dest = gen_rtx_REG (compare_mode, regno);
    6973              :               else
    6974              :                 {
    6975            0 :                   subst_mode (regno, compare_mode);
    6976            0 :                   new_dest = regno_reg_rtx[regno];
    6977              :                 }
    6978              : 
    6979       213802 :               SUBST (SET_DEST (x), new_dest);
    6980       213802 :               SUBST (XEXP (*cc_use, 0), new_dest);
    6981       213802 :               other_changed = true;
    6982              : 
    6983       213802 :               dest = new_dest;
    6984              :             }
    6985              :         }
    6986              : #endif  /* SELECT_CC_MODE */
    6987              : 
    6988              :       /* If the code changed, we have to build a new comparison in
    6989              :          undobuf.other_insn.  */
    6990      6155487 :       if (new_code != old_code)
    6991              :         {
    6992       620470 :           bool other_changed_previously = other_changed;
    6993       620470 :           unsigned HOST_WIDE_INT mask;
    6994       620470 :           rtx old_cc_use = *cc_use;
    6995              : 
    6996       620470 :           SUBST (*cc_use, gen_rtx_fmt_ee (new_code, GET_MODE (*cc_use),
    6997              :                                           dest, const0_rtx));
    6998       620470 :           other_changed = true;
    6999              : 
    7000              :           /* If the only change we made was to change an EQ into an NE or
    7001              :              vice versa, OP0 has only one bit that might be nonzero, and OP1
    7002              :              is zero, check if changing the user of the condition code will
    7003              :              produce a valid insn.  If it won't, we can keep the original code
    7004              :              in that insn by surrounding our operation with an XOR.  */
    7005              : 
    7006       620470 :           if (((old_code == NE && new_code == EQ)
    7007       580000 :                || (old_code == EQ && new_code == NE))
    7008        90353 :               && ! other_changed_previously && op1 == const0_rtx
    7009        87598 :               && HWI_COMPUTABLE_MODE_P (GET_MODE (op0))
    7010       630415 :               && pow2p_hwi (mask = nonzero_bits (op0, GET_MODE (op0))))
    7011              :             {
    7012         9934 :               rtx pat = PATTERN (other_insn), note = 0;
    7013              : 
    7014         9934 :               if ((recog_for_combine (&pat, other_insn, &note) < 0
    7015         9934 :                    && ! check_asm_operands (pat)))
    7016              :                 {
    7017            6 :                   *cc_use = old_cc_use;
    7018            6 :                   other_changed = false;
    7019              : 
    7020            6 :                   op0 = simplify_gen_binary (XOR, GET_MODE (op0), op0,
    7021            6 :                                              gen_int_mode (mask,
    7022            6 :                                                            GET_MODE (op0)));
    7023              :                 }
    7024              :             }
    7025              :         }
    7026              : 
    7027      5544951 :       if (other_changed)
    7028       637846 :         undobuf.other_insn = other_insn;
    7029              : 
    7030              :       /* Don't generate a compare of a CC with 0, just use that CC.  */
    7031      6155487 :       if (GET_MODE (op0) == compare_mode && op1 == const0_rtx)
    7032              :         {
    7033       689325 :           SUBST (SET_SRC (x), op0);
    7034       689325 :           src = SET_SRC (x);
    7035              :         }
    7036              :       /* Otherwise, if we didn't previously have the same COMPARE we
    7037              :          want, create it from scratch.  */
    7038      5466162 :       else if (GET_CODE (src) != COMPARE || GET_MODE (src) != compare_mode
    7039      5343975 :                || XEXP (src, 0) != op0 || XEXP (src, 1) != op1)
    7040              :         {
    7041      1377144 :           SUBST (SET_SRC (x), gen_rtx_COMPARE (compare_mode, op0, op1));
    7042      1377144 :           src = SET_SRC (x);
    7043              :         }
    7044              :     }
    7045              :   else
    7046              :     {
    7047              :       /* Get SET_SRC in a form where we have placed back any
    7048              :          compound expressions.  Then do the checks below.  */
    7049     41950540 :       src = make_compound_operation (src, SET);
    7050     41950540 :       SUBST (SET_SRC (x), src);
    7051              :     }
    7052              : 
    7053              :   /* If we have (set x (subreg:m1 (op:m2 ...) 0)) with OP being some operation,
    7054              :      and X being a REG or (subreg (reg)), we may be able to convert this to
    7055              :      (set (subreg:m2 x) (op)).
    7056              : 
    7057              :      We can always do this if M1 is narrower than M2 because that means that
    7058              :      we only care about the low bits of the result.
    7059              : 
    7060              :      However, on machines without WORD_REGISTER_OPERATIONS defined, we cannot
    7061              :      perform a narrower operation than requested since the high-order bits will
    7062              :      be undefined.  On machine where it is defined, this transformation is safe
    7063              :      as long as M1 and M2 have the same number of words.  */
    7064              : 
    7065       434427 :   if (GET_CODE (src) == SUBREG && subreg_lowpart_p (src)
    7066       418412 :       && !OBJECT_P (SUBREG_REG (src))
    7067              :       && (known_equal_after_align_up
    7068       269679 :           (GET_MODE_SIZE (GET_MODE (src)),
    7069       539358 :            GET_MODE_SIZE (GET_MODE (SUBREG_REG (src))),
    7070       269679 :            UNITS_PER_WORD))
    7071       239787 :       && (WORD_REGISTER_OPERATIONS || !paradoxical_subreg_p (src))
    7072       226481 :       && ! (REG_P (dest) && REGNO (dest) < FIRST_PSEUDO_REGISTER
    7073          233 :             && !REG_CAN_CHANGE_MODE_P (REGNO (dest),
    7074              :                                        GET_MODE (SUBREG_REG (src)),
    7075              :                                        GET_MODE (src)))
    7076     48332275 :       && (REG_P (dest)
    7077       125133 :           || (GET_CODE (dest) == SUBREG
    7078          313 :               && REG_P (SUBREG_REG (dest)))))
    7079              :     {
    7080       101428 :       SUBST (SET_DEST (x),
    7081              :              gen_lowpart (GET_MODE (SUBREG_REG (src)),
    7082              :                                       dest));
    7083       101428 :       SUBST (SET_SRC (x), SUBREG_REG (src));
    7084              : 
    7085       101428 :       src = SET_SRC (x), dest = SET_DEST (x);
    7086              :     }
    7087              : 
    7088              :   /* If we have (set FOO (subreg:M (mem:N BAR) 0)) with M wider than N, this
    7089              :      would require a paradoxical subreg.  Replace the subreg with a
    7090              :      zero_extend to avoid the reload that would otherwise be required.
    7091              :      Don't do this unless we have a scalar integer mode, otherwise the
    7092              :      transformation is incorrect.  */
    7093              : 
    7094     48106027 :   enum rtx_code extend_op;
    7095     48106027 :   if (paradoxical_subreg_p (src)
    7096              :       && MEM_P (SUBREG_REG (src))
    7097              :       && SCALAR_INT_MODE_P (GET_MODE (src))
    7098              :       && (extend_op = load_extend_op (GET_MODE (SUBREG_REG (src)))) != UNKNOWN)
    7099              :     {
    7100              :       SUBST (SET_SRC (x),
    7101              :              gen_rtx_fmt_e (extend_op, GET_MODE (src), SUBREG_REG (src)));
    7102              : 
    7103              :       src = SET_SRC (x);
    7104              :     }
    7105              : 
    7106              :   /* If we don't have a conditional move, SET_SRC is an IF_THEN_ELSE, and we
    7107              :      are comparing an item known to be 0 or -1 against 0, use a logical
    7108              :      operation instead. Check for one of the arms being an IOR of the other
    7109              :      arm with some value.  We compute three terms to be IOR'ed together.  In
    7110              :      practice, at most two will be nonzero.  Then we do the IOR's.  */
    7111              : 
    7112     48106027 :   if (GET_CODE (dest) != PC
    7113     37137968 :       && GET_CODE (src) == IF_THEN_ELSE
    7114      1352515 :       && is_int_mode (GET_MODE (src), &int_mode)
    7115      1221602 :       && (GET_CODE (XEXP (src, 0)) == EQ || GET_CODE (XEXP (src, 0)) == NE)
    7116       484015 :       && XEXP (XEXP (src, 0), 1) == const0_rtx
    7117       357584 :       && int_mode == GET_MODE (XEXP (XEXP (src, 0), 0))
    7118       118930 :       && (!HAVE_conditional_move
    7119       118930 :           || ! can_conditionally_move_p (int_mode))
    7120            0 :       && (num_sign_bit_copies (XEXP (XEXP (src, 0), 0), int_mode)
    7121            0 :           == GET_MODE_PRECISION (int_mode))
    7122     48106027 :       && ! side_effects_p (src))
    7123              :     {
    7124            0 :       rtx true_rtx = (GET_CODE (XEXP (src, 0)) == NE
    7125            0 :                       ? XEXP (src, 1) : XEXP (src, 2));
    7126            0 :       rtx false_rtx = (GET_CODE (XEXP (src, 0)) == NE
    7127            0 :                    ? XEXP (src, 2) : XEXP (src, 1));
    7128            0 :       rtx term1 = const0_rtx, term2, term3;
    7129              : 
    7130            0 :       if (GET_CODE (true_rtx) == IOR
    7131            0 :           && rtx_equal_p (XEXP (true_rtx, 0), false_rtx))
    7132            0 :         term1 = false_rtx, true_rtx = XEXP (true_rtx, 1), false_rtx = const0_rtx;
    7133            0 :       else if (GET_CODE (true_rtx) == IOR
    7134            0 :                && rtx_equal_p (XEXP (true_rtx, 1), false_rtx))
    7135            0 :         term1 = false_rtx, true_rtx = XEXP (true_rtx, 0), false_rtx = const0_rtx;
    7136            0 :       else if (GET_CODE (false_rtx) == IOR
    7137            0 :                && rtx_equal_p (XEXP (false_rtx, 0), true_rtx))
    7138            0 :         term1 = true_rtx, false_rtx = XEXP (false_rtx, 1), true_rtx = const0_rtx;
    7139            0 :       else if (GET_CODE (false_rtx) == IOR
    7140            0 :                && rtx_equal_p (XEXP (false_rtx, 1), true_rtx))
    7141            0 :         term1 = true_rtx, false_rtx = XEXP (false_rtx, 0), true_rtx = const0_rtx;
    7142              : 
    7143            0 :       term2 = simplify_gen_binary (AND, int_mode,
    7144            0 :                                    XEXP (XEXP (src, 0), 0), true_rtx);
    7145            0 :       term3 = simplify_gen_binary (AND, int_mode,
    7146              :                                    simplify_gen_unary (NOT, int_mode,
    7147            0 :                                                        XEXP (XEXP (src, 0), 0),
    7148              :                                                        int_mode),
    7149              :                                    false_rtx);
    7150              : 
    7151            0 :       SUBST (SET_SRC (x),
    7152              :              simplify_gen_binary (IOR, int_mode,
    7153              :                                   simplify_gen_binary (IOR, int_mode,
    7154              :                                                        term1, term2),
    7155              :                                   term3));
    7156              : 
    7157            0 :       src = SET_SRC (x);
    7158              :     }
    7159              : 
    7160              :   /* If either SRC or DEST is a CLOBBER of (const_int 0), make this
    7161              :      whole thing fail.  */
    7162     48106027 :   if (GET_CODE (src) == CLOBBER && XEXP (src, 0) == const0_rtx)
    7163              :     return src;
    7164     48106009 :   else if (GET_CODE (dest) == CLOBBER && XEXP (dest, 0) == const0_rtx)
    7165              :     return dest;
    7166              :   else
    7167              :     /* Convert this into a field assignment operation, if possible.  */
    7168     48105967 :     return make_field_assignment (x);
    7169              : }
    7170              : 
    7171              : /* Simplify, X, and AND, IOR, or XOR operation, and return the simplified
    7172              :    result.  */
    7173              : 
    7174              : static rtx
    7175     11531057 : simplify_logical (rtx x)
    7176              : {
    7177     11531057 :   rtx op0 = XEXP (x, 0);
    7178     11531057 :   rtx op1 = XEXP (x, 1);
    7179     11531057 :   scalar_int_mode mode;
    7180              : 
    7181     11531057 :   switch (GET_CODE (x))
    7182              :     {
    7183      7332797 :     case AND:
    7184              :       /* We can call simplify_and_const_int only if we don't lose
    7185              :          any (sign) bits when converting INTVAL (op1) to
    7186              :          "unsigned HOST_WIDE_INT".  */
    7187      7332797 :       if (is_a <scalar_int_mode> (GET_MODE (x), &mode)
    7188      6793116 :           && CONST_INT_P (op1)
    7189      5317464 :           && (HWI_COMPUTABLE_MODE_P (mode)
    7190         6729 :               || INTVAL (op1) > 0))
    7191              :         {
    7192      5314124 :           x = simplify_and_const_int (x, mode, op0, INTVAL (op1));
    7193      5314124 :           if (GET_CODE (x) != AND)
    7194              :             return x;
    7195              : 
    7196      5288229 :           op0 = XEXP (x, 0);
    7197      5288229 :           op1 = XEXP (x, 1);
    7198              :         }
    7199              : 
    7200              :       /* If we have any of (and (ior A B) C) or (and (xor A B) C),
    7201              :          apply the distributive law and then the inverse distributive
    7202              :          law to see if things simplify.  */
    7203      7306902 :       if (GET_CODE (op0) == IOR || GET_CODE (op0) == XOR)
    7204              :         {
    7205       125158 :           rtx result = distribute_and_simplify_rtx (x, 0);
    7206       125158 :           if (result)
    7207              :             return result;
    7208              :         }
    7209      7292996 :       if (GET_CODE (op1) == IOR || GET_CODE (op1) == XOR)
    7210              :         {
    7211         1938 :           rtx result = distribute_and_simplify_rtx (x, 1);
    7212         1938 :           if (result)
    7213            0 :             return result;
    7214              :         }
    7215              :       break;
    7216              : 
    7217      4198260 :     case IOR:
    7218              :       /* If we have (ior (and A B) C), apply the distributive law and then
    7219              :          the inverse distributive law to see if things simplify.  */
    7220              : 
    7221      4198260 :       if (GET_CODE (op0) == AND)
    7222              :         {
    7223      1267265 :           rtx result = distribute_and_simplify_rtx (x, 0);
    7224      1267265 :           if (result)
    7225              :             return result;
    7226              :         }
    7227              : 
    7228      4195506 :       if (GET_CODE (op1) == AND)
    7229              :         {
    7230        58169 :           rtx result = distribute_and_simplify_rtx (x, 1);
    7231        58169 :           if (result)
    7232            4 :             return result;
    7233              :         }
    7234              :       break;
    7235              : 
    7236            0 :     default:
    7237            0 :       gcc_unreachable ();
    7238              :     }
    7239              : 
    7240              :   return x;
    7241              : }
    7242              : 
    7243              : /* We consider ZERO_EXTRACT, SIGN_EXTRACT, and SIGN_EXTEND as "compound
    7244              :    operations" because they can be replaced with two more basic operations.
    7245              :    ZERO_EXTEND is also considered "compound" because it can be replaced with
    7246              :    an AND operation, which is simpler, though only one operation.
    7247              : 
    7248              :    The function expand_compound_operation is called with an rtx expression
    7249              :    and will convert it to the appropriate shifts and AND operations,
    7250              :    simplifying at each stage.
    7251              : 
    7252              :    The function make_compound_operation is called to convert an expression
    7253              :    consisting of shifts and ANDs into the equivalent compound expression.
    7254              :    It is the inverse of this function, loosely speaking.  */
    7255              : 
    7256              : static rtx
    7257     29994842 : expand_compound_operation (rtx x)
    7258              : {
    7259     29994842 :   unsigned HOST_WIDE_INT pos = 0, len;
    7260     29994842 :   bool unsignedp = false;
    7261     29994842 :   unsigned int modewidth;
    7262     29994842 :   rtx tem;
    7263     29994842 :   scalar_int_mode inner_mode;
    7264              : 
    7265     29994842 :   switch (GET_CODE (x))
    7266              :     {
    7267      4999932 :     case ZERO_EXTEND:
    7268      4999932 :       unsignedp = true;
    7269              :       /* FALLTHRU */
    7270      6446502 :     case SIGN_EXTEND:
    7271              :       /* We can't necessarily use a const_int for a multiword mode;
    7272              :          it depends on implicitly extending the value.
    7273              :          Since we don't know the right way to extend it,
    7274              :          we can't tell whether the implicit way is right.
    7275              : 
    7276              :          Even for a mode that is no wider than a const_int,
    7277              :          we can't win, because we need to sign extend one of its bits through
    7278              :          the rest of it, and we don't know which bit.  */
    7279      6446502 :       if (CONST_INT_P (XEXP (x, 0)))
    7280              :         return x;
    7281              : 
    7282              :       /* Reject modes that aren't scalar integers because turning vector
    7283              :          or complex modes into shifts causes problems.  */
    7284      6446502 :       if (!is_a <scalar_int_mode> (GET_MODE (XEXP (x, 0)), &inner_mode))
    7285              :         return x;
    7286              : 
    7287              :       /* Return if (subreg:MODE FROM 0) is not a safe replacement for
    7288              :          (zero_extend:MODE FROM) or (sign_extend:MODE FROM).  It is for any MEM
    7289              :          because (SUBREG (MEM...)) is guaranteed to cause the MEM to be
    7290              :          reloaded. If not for that, MEM's would very rarely be safe.
    7291              : 
    7292              :          Reject modes bigger than a word, because we might not be able
    7293              :          to reference a two-register group starting with an arbitrary register
    7294              :          (and currently gen_lowpart might crash for a SUBREG).  */
    7295              : 
    7296     13085437 :       if (GET_MODE_SIZE (inner_mode) > UNITS_PER_WORD)
    7297              :         return x;
    7298              : 
    7299      6086833 :       len = GET_MODE_PRECISION (inner_mode);
    7300              :       /* If the inner object has VOIDmode (the only way this can happen
    7301              :          is if it is an ASM_OPERANDS), we can't do anything since we don't
    7302              :          know how much masking to do.  */
    7303      6086833 :       if (len == 0)
    7304              :         return x;
    7305              : 
    7306              :       break;
    7307              : 
    7308       959837 :     case ZERO_EXTRACT:
    7309       959837 :       unsignedp = true;
    7310              : 
    7311              :       /* fall through */
    7312              : 
    7313       991694 :     case SIGN_EXTRACT:
    7314              :       /* If the operand is a CLOBBER, just return it.  */
    7315       991694 :       if (GET_CODE (XEXP (x, 0)) == CLOBBER)
    7316              :         return XEXP (x, 0);
    7317              : 
    7318       991688 :       if (!CONST_INT_P (XEXP (x, 1))
    7319       991565 :           || !CONST_INT_P (XEXP (x, 2)))
    7320              :         return x;
    7321              : 
    7322              :       /* Reject modes that aren't scalar integers because turning vector
    7323              :          or complex modes into shifts causes problems.  */
    7324       916956 :       if (!is_a <scalar_int_mode> (GET_MODE (XEXP (x, 0)), &inner_mode))
    7325              :         return x;
    7326              : 
    7327       916954 :       len = INTVAL (XEXP (x, 1));
    7328       916954 :       pos = INTVAL (XEXP (x, 2));
    7329              : 
    7330              :       /* This should stay within the object being extracted, fail otherwise.  */
    7331       916954 :       if (len + pos > GET_MODE_PRECISION (inner_mode))
    7332              :         return x;
    7333              : 
    7334              :       if (BITS_BIG_ENDIAN)
    7335              :         pos = GET_MODE_PRECISION (inner_mode) - len - pos;
    7336              : 
    7337              :       break;
    7338              : 
    7339              :     default:
    7340              :       return x;
    7341              :     }
    7342              : 
    7343              :   /* We've rejected non-scalar operations by now.  */
    7344      7003738 :   scalar_int_mode mode = as_a <scalar_int_mode> (GET_MODE (x));
    7345              : 
    7346              :   /* Convert sign extension to zero extension, if we know that the high
    7347              :      bit is not set, as this is easier to optimize.  It will be converted
    7348              :      back to cheaper alternative in make_extraction.  */
    7349      7003738 :   if (GET_CODE (x) == SIGN_EXTEND
    7350      1290152 :       && HWI_COMPUTABLE_MODE_P (mode)
    7351      8179232 :       && ((nonzero_bits (XEXP (x, 0), inner_mode)
    7352      1175494 :            & ~(((unsigned HOST_WIDE_INT) GET_MODE_MASK (inner_mode)) >> 1))
    7353              :           == 0))
    7354              :     {
    7355          383 :       rtx temp = gen_rtx_ZERO_EXTEND (mode, XEXP (x, 0));
    7356          383 :       rtx temp2 = expand_compound_operation (temp);
    7357              : 
    7358              :       /* Make sure this is a profitable operation.  */
    7359          383 :       if (set_src_cost (x, mode, optimize_this_for_speed_p)
    7360          383 :           > set_src_cost (temp2, mode, optimize_this_for_speed_p))
    7361              :        return temp2;
    7362          339 :       else if (set_src_cost (x, mode, optimize_this_for_speed_p)
    7363          339 :                > set_src_cost (temp, mode, optimize_this_for_speed_p))
    7364              :        return temp;
    7365              :       else
    7366           18 :        return x;
    7367              :     }
    7368              : 
    7369              :   /* We can optimize some special cases of ZERO_EXTEND.  */
    7370      7003355 :   if (GET_CODE (x) == ZERO_EXTEND)
    7371              :     {
    7372              :       /* (zero_extend:DI (truncate:SI foo:DI)) is just foo:DI if we
    7373              :          know that the last value didn't have any inappropriate bits
    7374              :          set.  */
    7375      4796681 :       if (GET_CODE (XEXP (x, 0)) == TRUNCATE
    7376          248 :           && GET_MODE (XEXP (XEXP (x, 0), 0)) == mode
    7377          248 :           && HWI_COMPUTABLE_MODE_P (mode)
    7378      4796929 :           && (nonzero_bits (XEXP (XEXP (x, 0), 0), mode)
    7379          248 :               & ~GET_MODE_MASK (inner_mode)) == 0)
    7380           49 :         return XEXP (XEXP (x, 0), 0);
    7381              : 
    7382              :       /* Likewise for (zero_extend:DI (subreg:SI foo:DI 0)).  */
    7383      4796632 :       if (GET_CODE (XEXP (x, 0)) == SUBREG
    7384       780465 :           && GET_MODE (SUBREG_REG (XEXP (x, 0))) == mode
    7385       708939 :           && subreg_lowpart_p (XEXP (x, 0))
    7386       342787 :           && HWI_COMPUTABLE_MODE_P (mode)
    7387      5111765 :           && (nonzero_bits (SUBREG_REG (XEXP (x, 0)), mode)
    7388       315133 :               & ~GET_MODE_MASK (inner_mode)) == 0)
    7389           65 :         return SUBREG_REG (XEXP (x, 0));
    7390              : 
    7391              :       /* (zero_extend:DI (truncate:SI foo:DI)) is just foo:DI when foo
    7392              :          is a comparison and STORE_FLAG_VALUE permits.  This is like
    7393              :          the first case, but it works even when MODE is larger
    7394              :          than HOST_WIDE_INT.  */
    7395      4796567 :       if (GET_CODE (XEXP (x, 0)) == TRUNCATE
    7396          199 :           && GET_MODE (XEXP (XEXP (x, 0), 0)) == mode
    7397          199 :           && COMPARISON_P (XEXP (XEXP (x, 0), 0))
    7398            0 :           && GET_MODE_PRECISION (inner_mode) <= HOST_BITS_PER_WIDE_INT
    7399      4796567 :           && (STORE_FLAG_VALUE & ~GET_MODE_MASK (inner_mode)) == 0)
    7400              :         return XEXP (XEXP (x, 0), 0);
    7401              : 
    7402              :       /* Likewise for (zero_extend:DI (subreg:SI foo:DI 0)).  */
    7403      4796567 :       if (GET_CODE (XEXP (x, 0)) == SUBREG
    7404       780400 :           && GET_MODE (SUBREG_REG (XEXP (x, 0))) == mode
    7405       708874 :           && subreg_lowpart_p (XEXP (x, 0))
    7406       342722 :           && COMPARISON_P (SUBREG_REG (XEXP (x, 0)))
    7407            0 :           && GET_MODE_PRECISION (inner_mode) <= HOST_BITS_PER_WIDE_INT
    7408      4796567 :           && (STORE_FLAG_VALUE & ~GET_MODE_MASK (inner_mode)) == 0)
    7409              :         return SUBREG_REG (XEXP (x, 0));
    7410              : 
    7411              :     }
    7412              : 
    7413              :   /* If we reach here, we want to return a pair of shifts.  The inner
    7414              :      shift is a left shift of BITSIZE - POS - LEN bits.  The outer
    7415              :      shift is a right shift of BITSIZE - LEN bits.  It is arithmetic or
    7416              :      logical depending on the value of UNSIGNEDP.
    7417              : 
    7418              :      If this was a ZERO_EXTEND or ZERO_EXTRACT, this pair of shifts will be
    7419              :      converted into an AND of a shift.
    7420              : 
    7421              :      We must check for the case where the left shift would have a negative
    7422              :      count.  This can happen in a case like (x >> 31) & 255 on machines
    7423              :      that can't shift by a constant.  On those machines, we would first
    7424              :      combine the shift with the AND to produce a variable-position
    7425              :      extraction.  Then the constant of 31 would be substituted in
    7426              :      to produce such a position.  */
    7427              : 
    7428      7003241 :   modewidth = GET_MODE_PRECISION (mode);
    7429      7003241 :   if (modewidth >= pos + len)
    7430              :     {
    7431      7003240 :       tem = gen_lowpart (mode, XEXP (x, 0));
    7432      7003240 :       if (!tem || GET_CODE (tem) == CLOBBER)
    7433              :         return x;
    7434      7591096 :       tem = simplify_shift_const (NULL_RTX, ASHIFT, mode,
    7435      3795548 :                                   tem, modewidth - pos - len);
    7436      3795548 :       tem = simplify_shift_const (NULL_RTX, unsignedp ? LSHIFTRT : ASHIFTRT,
    7437      3795548 :                                   mode, tem, modewidth - len);
    7438              :     }
    7439            1 :   else if (unsignedp && len < HOST_BITS_PER_WIDE_INT)
    7440              :     {
    7441            1 :       tem = simplify_shift_const (NULL_RTX, LSHIFTRT, inner_mode,
    7442              :                                   XEXP (x, 0), pos);
    7443            1 :       tem = gen_lowpart (mode, tem);
    7444            1 :       if (!tem || GET_CODE (tem) == CLOBBER)
    7445              :         return x;
    7446            1 :       tem = simplify_and_const_int (NULL_RTX, mode, tem,
    7447            1 :                                     (HOST_WIDE_INT_1U << len) - 1);
    7448              :     }
    7449              :   else
    7450              :     /* Any other cases we can't handle.  */
    7451              :     return x;
    7452              : 
    7453              :   /* If we couldn't do this for some reason, return the original
    7454              :      expression.  */
    7455      3795549 :   if (GET_CODE (tem) == CLOBBER)
    7456           25 :     return x;
    7457              : 
    7458              :   return tem;
    7459              : }
    7460              : 
    7461              : /* X is a SET which contains an assignment of one object into
    7462              :    a part of another (such as a bit-field assignment, STRICT_LOW_PART,
    7463              :    or certain SUBREGS). If possible, convert it into a series of
    7464              :    logical operations.
    7465              : 
    7466              :    We half-heartedly support variable positions, but do not at all
    7467              :    support variable lengths.  */
    7468              : 
    7469              : static const_rtx
    7470     86070600 : expand_field_assignment (const_rtx x)
    7471              : {
    7472     86070600 :   rtx inner;
    7473     86070600 :   rtx pos;                      /* Always counts from low bit.  */
    7474     86070600 :   int len, inner_len;
    7475     86070600 :   rtx mask, cleared, masked;
    7476     86070600 :   scalar_int_mode compute_mode;
    7477              : 
    7478              :   /* Loop until we find something we can't simplify.  */
    7479     86341567 :   while (1)
    7480              :     {
    7481     86341567 :       if (GET_CODE (SET_DEST (x)) == STRICT_LOW_PART
    7482        14285 :           && GET_CODE (XEXP (SET_DEST (x), 0)) == SUBREG)
    7483              :         {
    7484        14285 :           rtx x0 = XEXP (SET_DEST (x), 0);
    7485        14285 :           if (!GET_MODE_PRECISION (GET_MODE (x0)).is_constant (&len))
    7486              :             break;
    7487        14285 :           inner = SUBREG_REG (XEXP (SET_DEST (x), 0));
    7488        14285 :           pos = gen_int_mode (subreg_lsb (XEXP (SET_DEST (x), 0)),
    7489              :                               MAX_MODE_INT);
    7490        14285 :         }
    7491     86327282 :       else if (GET_CODE (SET_DEST (x)) == ZERO_EXTRACT
    7492         4533 :                && CONST_INT_P (XEXP (SET_DEST (x), 1)))
    7493              :         {
    7494         4533 :           inner = XEXP (SET_DEST (x), 0);
    7495         4533 :           if (!GET_MODE_PRECISION (GET_MODE (inner)).is_constant (&inner_len))
    7496              :             break;
    7497              : 
    7498         4533 :           len = INTVAL (XEXP (SET_DEST (x), 1));
    7499         4533 :           pos = XEXP (SET_DEST (x), 2);
    7500              : 
    7501              :           /* A constant position should stay within the width of INNER.  */
    7502         4533 :           if (CONST_INT_P (pos) && INTVAL (pos) + len > inner_len)
    7503              :             break;
    7504              : 
    7505              :           if (BITS_BIG_ENDIAN)
    7506              :             {
    7507              :               if (CONST_INT_P (pos))
    7508              :                 pos = GEN_INT (inner_len - len - INTVAL (pos));
    7509              :               else if (GET_CODE (pos) == MINUS
    7510              :                        && CONST_INT_P (XEXP (pos, 1))
    7511              :                        && INTVAL (XEXP (pos, 1)) == inner_len - len)
    7512              :                 /* If position is ADJUST - X, new position is X.  */
    7513              :                 pos = XEXP (pos, 0);
    7514              :               else
    7515              :                 pos = simplify_gen_binary (MINUS, GET_MODE (pos),
    7516              :                                            gen_int_mode (inner_len - len,
    7517              :                                                          GET_MODE (pos)),
    7518              :                                            pos);
    7519              :             }
    7520              :         }
    7521              : 
    7522              :       /* If the destination is a subreg that overwrites the whole of the inner
    7523              :          register, we can move the subreg to the source.  */
    7524     86581695 :       else if (GET_CODE (SET_DEST (x)) == SUBREG
    7525              :                /* We need SUBREGs to compute nonzero_bits properly.  */
    7526       917264 :                && nonzero_sign_valid
    7527     87151260 :                && !read_modify_subreg_p (SET_DEST (x)))
    7528              :         {
    7529       258946 :           x = gen_rtx_SET (SUBREG_REG (SET_DEST (x)),
    7530              :                            gen_lowpart
    7531              :                            (GET_MODE (SUBREG_REG (SET_DEST (x))),
    7532              :                             SET_SRC (x)));
    7533       258946 :           continue;
    7534              :         }
    7535              :       else
    7536              :         break;
    7537              : 
    7538        21090 :       while (GET_CODE (inner) == SUBREG && subreg_lowpart_p (inner))
    7539         2272 :         inner = SUBREG_REG (inner);
    7540              : 
    7541              :       /* Don't attempt bitwise arithmetic on non scalar integer modes.  */
    7542        18818 :       if (!is_a <scalar_int_mode> (GET_MODE (inner), &compute_mode))
    7543              :         {
    7544              :           /* Don't do anything for vector or complex integral types.  */
    7545              :           if (! FLOAT_MODE_P (GET_MODE (inner)))
    7546              :             break;
    7547              : 
    7548              :           /* Try to find an integral mode to pun with.  */
    7549           38 :           if (!int_mode_for_size (GET_MODE_BITSIZE (GET_MODE (inner)), 0)
    7550            0 :               .exists (&compute_mode))
    7551              :             break;
    7552              : 
    7553           19 :           inner = gen_lowpart (compute_mode, inner);
    7554              :         }
    7555              : 
    7556              :       /* Compute a mask of LEN bits, if we can do this on the host machine.  */
    7557        14110 :       if (len >= HOST_BITS_PER_WIDE_INT)
    7558              :         break;
    7559              : 
    7560              :       /* Don't try to compute in too wide unsupported modes.  */
    7561        14110 :       if (!targetm.scalar_mode_supported_p (compute_mode))
    7562              :         break;
    7563              : 
    7564              :       /* gen_lowpart_for_combine returns CLOBBER on failure.  */
    7565        14110 :       rtx lowpart = gen_lowpart (compute_mode, SET_SRC (x));
    7566        14110 :       if (GET_CODE (lowpart) == CLOBBER)
    7567              :         break;
    7568              : 
    7569              :       /* Now compute the equivalent expression.  Make a copy of INNER
    7570              :          for the SET_DEST in case it is a MEM into which we will substitute;
    7571              :          we don't want shared RTL in that case.  */
    7572        12021 :       mask = gen_int_mode ((HOST_WIDE_INT_1U << len) - 1,
    7573              :                            compute_mode);
    7574        12021 :       cleared = simplify_gen_binary (AND, compute_mode,
    7575              :                                      simplify_gen_unary (NOT, compute_mode,
    7576              :                                        simplify_gen_binary (ASHIFT,
    7577              :                                                             compute_mode,
    7578              :                                                             mask, pos),
    7579              :                                        compute_mode),
    7580              :                                      inner);
    7581        12021 :       masked = simplify_gen_binary (ASHIFT, compute_mode,
    7582              :                                     simplify_gen_binary (
    7583              :                                       AND, compute_mode, lowpart, mask),
    7584              :                                     pos);
    7585              : 
    7586        12021 :       x = gen_rtx_SET (copy_rtx (inner),
    7587              :                        simplify_gen_binary (IOR, compute_mode,
    7588              :                                             cleared, masked));
    7589              :     }
    7590              : 
    7591     86070600 :   return x;
    7592              : }
    7593              : 
    7594              : /* Return an RTX for a reference to LEN bits of INNER.  If POS_RTX is nonzero,
    7595              :    it is an RTX that represents the (variable) starting position; otherwise,
    7596              :    POS is the (constant) starting bit position.  Both are counted from the LSB.
    7597              : 
    7598              :    UNSIGNEDP is true for an unsigned reference and zero for a signed one.
    7599              : 
    7600              :    IN_DEST is true if this is a reference in the destination of a SET.
    7601              :    This is used when a ZERO_ or SIGN_EXTRACT isn't needed.  If nonzero,
    7602              :    a STRICT_LOW_PART will be used, if zero, ZERO_EXTEND or SIGN_EXTEND will
    7603              :    be used.
    7604              : 
    7605              :    IN_COMPARE is true if we are in a COMPARE.  This means that a
    7606              :    ZERO_EXTRACT should be built even for bits starting at bit 0.
    7607              : 
    7608              :    MODE is the desired mode of the result (if IN_DEST == 0).
    7609              : 
    7610              :    The result is an RTX for the extraction or NULL_RTX if the target
    7611              :    can't handle it.  */
    7612              : 
    7613              : static rtx
    7614      5268167 : make_extraction (machine_mode mode, rtx inner, HOST_WIDE_INT pos,
    7615              :                  rtx pos_rtx, unsigned HOST_WIDE_INT len, bool unsignedp,
    7616              :                  bool in_dest, bool in_compare)
    7617              : {
    7618              :   /* This mode describes the size of the storage area
    7619              :      to fetch the overall value from.  Within that, we
    7620              :      ignore the POS lowest bits, etc.  */
    7621      5268167 :   machine_mode is_mode = GET_MODE (inner);
    7622      5268167 :   machine_mode inner_mode;
    7623      5268167 :   scalar_int_mode wanted_inner_mode;
    7624      5268167 :   scalar_int_mode wanted_inner_reg_mode = word_mode;
    7625      5268167 :   scalar_int_mode pos_mode = word_mode;
    7626      5268167 :   machine_mode extraction_mode = word_mode;
    7627      5268167 :   rtx new_rtx = 0;
    7628      5268167 :   rtx orig_pos_rtx = pos_rtx;
    7629      5268167 :   HOST_WIDE_INT orig_pos;
    7630              : 
    7631      5268167 :   if (pos_rtx && CONST_INT_P (pos_rtx))
    7632       949030 :     pos = INTVAL (pos_rtx), pos_rtx = 0;
    7633              : 
    7634      5268167 :   if (GET_CODE (inner) == SUBREG
    7635      2696262 :       && subreg_lowpart_p (inner)
    7636      7960742 :       && (paradoxical_subreg_p (inner)
    7637              :           /* If trying or potentially trying to extract
    7638              :              bits outside of is_mode, don't look through
    7639              :              non-paradoxical SUBREGs.  See PR82192.  */
    7640       189134 :           || (pos_rtx == NULL_RTX
    7641       189110 :               && known_le (pos + len, GET_MODE_PRECISION (is_mode)))))
    7642              :     {
    7643              :       /* If going from (subreg:SI (mem:QI ...)) to (mem:QI ...),
    7644              :          consider just the QI as the memory to extract from.
    7645              :          The subreg adds or removes high bits; its mode is
    7646              :          irrelevant to the meaning of this extraction,
    7647              :          since POS and LEN count from the lsb.  */
    7648      2692551 :       if (MEM_P (SUBREG_REG (inner)))
    7649       507818 :         is_mode = GET_MODE (SUBREG_REG (inner));
    7650              :       inner = SUBREG_REG (inner);
    7651              :     }
    7652      2575616 :   else if (GET_CODE (inner) == ASHIFT
    7653       143293 :            && CONST_INT_P (XEXP (inner, 1))
    7654       142103 :            && pos_rtx == 0 && pos == 0
    7655       142079 :            && len > UINTVAL (XEXP (inner, 1)))
    7656              :     {
    7657              :       /* We're extracting the least significant bits of an rtx
    7658              :          (ashift X (const_int C)), where LEN > C.  Extract the
    7659              :          least significant (LEN - C) bits of X, giving an rtx
    7660              :          whose mode is MODE, then shift it left C times.  */
    7661       142079 :       new_rtx = make_extraction (mode, XEXP (inner, 0),
    7662              :                              0, 0, len - INTVAL (XEXP (inner, 1)),
    7663              :                              unsignedp, in_dest, in_compare);
    7664       142079 :       if (new_rtx != 0)
    7665       140433 :         return gen_rtx_ASHIFT (mode, new_rtx, XEXP (inner, 1));
    7666              :     }
    7667      2433537 :   else if (GET_CODE (inner) == MULT
    7668       175650 :            && CONST_INT_P (XEXP (inner, 1))
    7669       133439 :            && pos_rtx == 0 && pos == 0)
    7670              :     {
    7671              :       /* We're extracting the least significant bits of an rtx
    7672              :          (mult X (const_int 2^C)), where LEN > C.  Extract the
    7673              :          least significant (LEN - C) bits of X, giving an rtx
    7674              :          whose mode is MODE, then multiply it by 2^C.  */
    7675       113234 :       const HOST_WIDE_INT shift_amt = exact_log2 (INTVAL (XEXP (inner, 1)));
    7676       113234 :       if (len > 1 && IN_RANGE (shift_amt, 1, len - 1))
    7677              :         {
    7678       108578 :           new_rtx = make_extraction (mode, XEXP (inner, 0),
    7679              :                                      0, 0, len - shift_amt,
    7680              :                                      unsignedp, in_dest, in_compare);
    7681       108578 :           if (new_rtx)
    7682       108578 :             return gen_rtx_MULT (mode, new_rtx, XEXP (inner, 1));
    7683              :         }
    7684              :     }
    7685      2320303 :   else if (GET_CODE (inner) == TRUNCATE
    7686              :            /* If trying or potentially trying to extract
    7687              :               bits outside of is_mode, don't look through
    7688              :               TRUNCATE.  See PR82192.  */
    7689            0 :            && pos_rtx == NULL_RTX
    7690      2320303 :            && known_le (pos + len, GET_MODE_PRECISION (is_mode)))
    7691            0 :     inner = XEXP (inner, 0);
    7692              : 
    7693      5019156 :   inner_mode = GET_MODE (inner);
    7694              : 
    7695              :   /* See if this can be done without an extraction.  We never can if the
    7696              :      width of the field is not the same as that of some integer mode. For
    7697              :      registers, we can only avoid the extraction if the position is at the
    7698              :      low-order bit and this is either not in the destination or we have the
    7699              :      appropriate STRICT_LOW_PART operation available.
    7700              : 
    7701              :      For MEM, we can avoid an extract if the field starts on an appropriate
    7702              :      boundary and we can change the mode of the memory reference.  */
    7703              : 
    7704      5019156 :   scalar_int_mode tmode;
    7705      5019156 :   if (int_mode_for_size (len, 1).exists (&tmode)
    7706      2438824 :       && ((pos_rtx == 0 && (pos % BITS_PER_WORD) == 0
    7707      2119051 :            && !MEM_P (inner)
    7708      1721805 :            && (pos == 0 || REG_P (inner))
    7709      1721805 :            && (inner_mode == tmode
    7710       266905 :                || !REG_P (inner)
    7711      2347358 :                || TRULY_NOOP_TRUNCATION_MODES_P (tmode, inner_mode)
    7712            0 :                || reg_truncated_to_mode (tmode, inner))
    7713      1721805 :            && (! in_dest
    7714           29 :                || (REG_P (inner)
    7715           29 :                    && have_insn_for (STRICT_LOW_PART, tmode))))
    7716       575884 :           || (MEM_P (inner) && pos_rtx == 0
    7717       398626 :               && (pos
    7718              :                   % (STRICT_ALIGNMENT ? GET_MODE_ALIGNMENT (tmode)
    7719              :                      : BITS_PER_UNIT)) == 0
    7720              :               /* We can't do this if we are widening INNER_MODE (it
    7721              :                  may not be aligned, for one thing).  */
    7722       397604 :               && !paradoxical_subreg_p (tmode, inner_mode)
    7723       397604 :               && known_le (pos + len, GET_MODE_PRECISION (is_mode))
    7724       397604 :               && (inner_mode == tmode
    7725          579 :                   || (! mode_dependent_address_p (XEXP (inner, 0),
    7726          579 :                                                   MEM_ADDR_SPACE (inner))
    7727          579 :                       && ! MEM_VOLATILE_P (inner))))))
    7728              :     {
    7729              :       /* If INNER is a MEM, make a new MEM that encompasses just the desired
    7730              :          field.  If the original and current mode are the same, we need not
    7731              :          adjust the offset.  Otherwise, we do if bytes big endian.
    7732              : 
    7733              :          If INNER is not a MEM, get a piece consisting of just the field
    7734              :          of interest (in this case POS % BITS_PER_WORD must be 0).  */
    7735              : 
    7736      2119380 :       if (MEM_P (inner))
    7737              :         {
    7738       397591 :           poly_int64 offset;
    7739              : 
    7740              :           /* POS counts from lsb, but make OFFSET count in memory order.  */
    7741       397591 :           if (BYTES_BIG_ENDIAN)
    7742              :             offset = bits_to_bytes_round_down (GET_MODE_PRECISION (is_mode)
    7743              :                                                - len - pos);
    7744              :           else
    7745       397591 :             offset = pos / BITS_PER_UNIT;
    7746              : 
    7747       397591 :           new_rtx = adjust_address_nv (inner, tmode, offset);
    7748              :         }
    7749      1721789 :       else if (REG_P (inner))
    7750              :         {
    7751      1078702 :           if (tmode != inner_mode)
    7752              :             {
    7753              :               /* We can't call gen_lowpart in a DEST since we
    7754              :                  always want a SUBREG (see below) and it would sometimes
    7755              :                  return a new hard register.  */
    7756       227962 :               if (pos || in_dest)
    7757              :                 {
    7758           16 :                   poly_uint64 offset
    7759           16 :                     = subreg_offset_from_lsb (tmode, inner_mode, pos);
    7760              : 
    7761              :                   /* Avoid creating invalid subregs, for example when
    7762              :                      simplifying (x>>32)&255.  */
    7763           16 :                   if (!validate_subreg (tmode, inner_mode, inner, offset))
    7764            0 :                     return NULL_RTX;
    7765              : 
    7766           16 :                   new_rtx = gen_rtx_SUBREG (tmode, inner, offset);
    7767           16 :                 }
    7768              :               else
    7769       227946 :                 new_rtx = gen_lowpart (tmode, inner);
    7770              :             }
    7771              :           else
    7772              :             new_rtx = inner;
    7773              :         }
    7774              :       else
    7775      1286174 :         new_rtx = force_to_mode (inner, tmode,
    7776              :                                  len >= HOST_BITS_PER_WIDE_INT
    7777              :                                  ? HOST_WIDE_INT_M1U
    7778       643087 :                                  : (HOST_WIDE_INT_1U << len) - 1, false);
    7779              : 
    7780              :       /* If this extraction is going into the destination of a SET,
    7781              :          make a STRICT_LOW_PART unless we made a MEM.  */
    7782              : 
    7783      2119380 :       if (in_dest)
    7784           55 :         return (MEM_P (new_rtx) ? new_rtx
    7785              :                 : (GET_CODE (new_rtx) != SUBREG
    7786           13 :                    ? gen_rtx_CLOBBER (tmode, const0_rtx)
    7787           13 :                    : gen_rtx_STRICT_LOW_PART (VOIDmode, new_rtx)));
    7788              : 
    7789      2119325 :       if (mode == tmode)
    7790              :         return new_rtx;
    7791              : 
    7792      2119296 :       if (CONST_SCALAR_INT_P (new_rtx))
    7793            5 :         return simplify_unary_operation (unsignedp ? ZERO_EXTEND : SIGN_EXTEND,
    7794            5 :                                          mode, new_rtx, tmode);
    7795              : 
    7796              :       /* If we know that no extraneous bits are set, and that the high
    7797              :          bit is not set, convert the extraction to the cheaper of
    7798              :          sign and zero extension, that are equivalent in these cases.  */
    7799      2119291 :       if (flag_expensive_optimizations
    7800      2119291 :           && (HWI_COMPUTABLE_MODE_P (tmode)
    7801      1965230 :               && ((nonzero_bits (new_rtx, tmode)
    7802      1965230 :                    & ~(((unsigned HOST_WIDE_INT)GET_MODE_MASK (tmode)) >> 1))
    7803              :                   == 0)))
    7804              :         {
    7805         7074 :           rtx temp = gen_rtx_ZERO_EXTEND (mode, new_rtx);
    7806         7074 :           rtx temp1 = gen_rtx_SIGN_EXTEND (mode, new_rtx);
    7807              : 
    7808              :           /* Prefer ZERO_EXTENSION, since it gives more information to
    7809              :              backends.  */
    7810         7074 :           if (set_src_cost (temp, mode, optimize_this_for_speed_p)
    7811         7074 :               <= set_src_cost (temp1, mode, optimize_this_for_speed_p))
    7812              :             return temp;
    7813            0 :           return temp1;
    7814              :         }
    7815              : 
    7816              :       /* Otherwise, sign- or zero-extend unless we already are in the
    7817              :          proper mode.  */
    7818              : 
    7819      2112217 :       return (gen_rtx_fmt_e (unsignedp ? ZERO_EXTEND : SIGN_EXTEND,
    7820      2112217 :                              mode, new_rtx));
    7821              :     }
    7822              : 
    7823              :   /* Unless this is a COMPARE or we have a funny memory reference,
    7824              :      don't do anything with zero-extending field extracts starting at
    7825              :      the low-order bit since they are simple AND operations.  */
    7826      2899776 :   if (pos_rtx == 0 && pos == 0 && ! in_dest
    7827      1818401 :       && ! in_compare && unsignedp)
    7828              :     return 0;
    7829              : 
    7830              :   /* Unless INNER is not MEM, reject this if we would be spanning bytes or
    7831              :      if the position is not a constant and the length is not 1.  In all
    7832              :      other cases, we would only be going outside our object in cases when
    7833              :      an original shift would have been undefined.  */
    7834      1495374 :   if (MEM_P (inner)
    7835      1495374 :       && ((pos_rtx == 0 && maybe_gt (pos + len, GET_MODE_PRECISION (is_mode)))
    7836         2971 :           || (pos_rtx != 0 && len != 1)))
    7837              :     return 0;
    7838              : 
    7839      1616141 :   enum extraction_pattern pattern = (in_dest ? EP_insv
    7840      1488694 :                                      : unsignedp ? EP_extzv : EP_extv);
    7841              : 
    7842              :   /* If INNER is not from memory, we want it to have the mode of a register
    7843              :      extraction pattern's structure operand, or word_mode if there is no
    7844              :      such pattern.  The same applies to extraction_mode and pos_mode
    7845              :      and their respective operands.
    7846              : 
    7847              :      For memory, assume that the desired extraction_mode and pos_mode
    7848              :      are the same as for a register operation, since at present we don't
    7849              :      have named patterns for aligned memory structures.  */
    7850      1495334 :   class extraction_insn insn;
    7851      1495334 :   unsigned int inner_size;
    7852      2990668 :   if (GET_MODE_BITSIZE (inner_mode).is_constant (&inner_size)
    7853      1495334 :       && get_best_reg_extraction_insn (&insn, pattern, inner_size, mode))
    7854              :     {
    7855      1391920 :       wanted_inner_reg_mode = insn.struct_mode.require ();
    7856      1391920 :       pos_mode = insn.pos_mode;
    7857      1391920 :       extraction_mode = insn.field_mode;
    7858              :     }
    7859              : 
    7860              :   /* Never narrow an object, since that might not be safe.  */
    7861              : 
    7862      1495334 :   if (mode != VOIDmode
    7863      1495334 :       && partial_subreg_p (extraction_mode, mode))
    7864              :     extraction_mode = mode;
    7865              : 
    7866              :   /* Punt if len is too large for extraction_mode.  */
    7867      1495334 :   if (maybe_gt (len, GET_MODE_PRECISION (extraction_mode)))
    7868              :     return NULL_RTX;
    7869              : 
    7870      1495322 :   if (!MEM_P (inner))
    7871      1315075 :     wanted_inner_mode = wanted_inner_reg_mode;
    7872              :   else
    7873              :     {
    7874              :       /* Be careful not to go beyond the extracted object and maintain the
    7875              :          natural alignment of the memory.  */
    7876       180247 :       wanted_inner_mode = smallest_int_mode_for_size (len).require ();
    7877       363720 :       while (pos % GET_MODE_BITSIZE (wanted_inner_mode) + len
    7878       366946 :              > GET_MODE_BITSIZE (wanted_inner_mode))
    7879         3226 :         wanted_inner_mode = GET_MODE_WIDER_MODE (wanted_inner_mode).require ();
    7880              :     }
    7881              : 
    7882      1495322 :   orig_pos = pos;
    7883              : 
    7884      1495322 :   if (BITS_BIG_ENDIAN)
    7885              :     {
    7886              :       /* POS is passed as if BITS_BIG_ENDIAN == 0, so we need to convert it to
    7887              :          BITS_BIG_ENDIAN style.  If position is constant, compute new
    7888              :          position.  Otherwise, build subtraction.
    7889              :          Note that POS is relative to the mode of the original argument.
    7890              :          If it's a MEM we need to recompute POS relative to that.
    7891              :          However, if we're extracting from (or inserting into) a register,
    7892              :          we want to recompute POS relative to wanted_inner_mode.  */
    7893              :       int width;
    7894              :       if (!MEM_P (inner))
    7895              :         width = GET_MODE_BITSIZE (wanted_inner_mode);
    7896              :       else if (!GET_MODE_BITSIZE (is_mode).is_constant (&width))
    7897              :         return NULL_RTX;
    7898              : 
    7899              :       if (pos_rtx == 0)
    7900              :         pos = width - len - pos;
    7901              :       else
    7902              :         pos_rtx
    7903              :           = gen_rtx_MINUS (GET_MODE (pos_rtx),
    7904              :                            gen_int_mode (width - len, GET_MODE (pos_rtx)),
    7905              :                            pos_rtx);
    7906              :       /* POS may be less than 0 now, but we check for that below.
    7907              :          Note that it can only be less than 0 if !MEM_P (inner).  */
    7908              :     }
    7909              : 
    7910              :   /* If INNER has a wider mode, and this is a constant extraction, try to
    7911              :      make it smaller and adjust the byte to point to the byte containing
    7912              :      the value.  */
    7913      1495322 :   if (wanted_inner_mode != VOIDmode
    7914      1495322 :       && inner_mode != wanted_inner_mode
    7915       228300 :       && ! pos_rtx
    7916       219824 :       && partial_subreg_p (wanted_inner_mode, is_mode)
    7917       112660 :       && MEM_P (inner)
    7918        26914 :       && ! mode_dependent_address_p (XEXP (inner, 0), MEM_ADDR_SPACE (inner))
    7919      1522236 :       && ! MEM_VOLATILE_P (inner))
    7920              :     {
    7921        25171 :       poly_int64 offset = 0;
    7922              : 
    7923              :       /* The computations below will be correct if the machine is big
    7924              :          endian in both bits and bytes or little endian in bits and bytes.
    7925              :          If it is mixed, we must adjust.  */
    7926              : 
    7927              :       /* If bytes are big endian and we had a paradoxical SUBREG, we must
    7928              :          adjust OFFSET to compensate.  */
    7929        25171 :       if (BYTES_BIG_ENDIAN
    7930              :           && paradoxical_subreg_p (is_mode, inner_mode))
    7931              :         offset -= GET_MODE_SIZE (is_mode) - GET_MODE_SIZE (inner_mode);
    7932              : 
    7933              :       /* We can now move to the desired byte.  */
    7934        50342 :       offset += (pos / GET_MODE_BITSIZE (wanted_inner_mode))
    7935        25171 :                 * GET_MODE_SIZE (wanted_inner_mode);
    7936        25171 :       pos %= GET_MODE_BITSIZE (wanted_inner_mode);
    7937              : 
    7938        25171 :       if (BYTES_BIG_ENDIAN != BITS_BIG_ENDIAN
    7939              :           && is_mode != wanted_inner_mode)
    7940              :         offset = (GET_MODE_SIZE (is_mode)
    7941              :                   - GET_MODE_SIZE (wanted_inner_mode) - offset);
    7942              : 
    7943        25171 :       inner = adjust_address_nv (inner, wanted_inner_mode, offset);
    7944              :     }
    7945              : 
    7946              :   /* If INNER is not memory, get it into the proper mode.  If we are changing
    7947              :      its mode, POS must be a constant and smaller than the size of the new
    7948              :      mode.  */
    7949      1470151 :   else if (!MEM_P (inner))
    7950              :     {
    7951              :       /* On the LHS, don't create paradoxical subregs implicitly truncating
    7952              :          the register unless TARGET_TRULY_NOOP_TRUNCATION.  */
    7953      1315075 :       if (in_dest
    7954      1315075 :           && !TRULY_NOOP_TRUNCATION_MODES_P (GET_MODE (inner),
    7955              :                                              wanted_inner_mode))
    7956            0 :         return NULL_RTX;
    7957              : 
    7958      1315075 :       if (GET_MODE (inner) != wanted_inner_mode
    7959      1315075 :           && (pos_rtx != 0
    7960       385820 :               || orig_pos + len > GET_MODE_BITSIZE (wanted_inner_mode)))
    7961              :         return NULL_RTX;
    7962              : 
    7963      1251674 :       if (orig_pos < 0)
    7964              :         return NULL_RTX;
    7965              : 
    7966      2484201 :       inner = force_to_mode (inner, wanted_inner_mode,
    7967              :                              pos_rtx
    7968      1232527 :                              || len + orig_pos >= HOST_BITS_PER_WIDE_INT
    7969              :                              ? HOST_WIDE_INT_M1U
    7970      1069152 :                              : (((HOST_WIDE_INT_1U << len) - 1)
    7971      1069152 :                                 << orig_pos), false);
    7972              :     }
    7973              : 
    7974              :   /* Adjust mode of POS_RTX, if needed.  If we want a wider mode, we
    7975              :      have to zero extend.  Otherwise, we can just use a SUBREG.
    7976              : 
    7977              :      We dealt with constant rtxes earlier, so pos_rtx cannot
    7978              :      have VOIDmode at this point.  */
    7979      1431921 :   if (pos_rtx != 0
    7980      1431921 :       && (GET_MODE_SIZE (pos_mode)
    7981      1453999 :           > GET_MODE_SIZE (as_a <scalar_int_mode> (GET_MODE (pos_rtx)))))
    7982              :     {
    7983           78 :       rtx temp = simplify_gen_unary (ZERO_EXTEND, pos_mode, pos_rtx,
    7984              :                                      GET_MODE (pos_rtx));
    7985              : 
    7986              :       /* If we know that no extraneous bits are set, and that the high
    7987              :          bit is not set, convert extraction to cheaper one - either
    7988              :          SIGN_EXTENSION or ZERO_EXTENSION, that are equivalent in these
    7989              :          cases.  */
    7990           78 :       if (flag_expensive_optimizations
    7991           78 :           && (HWI_COMPUTABLE_MODE_P (GET_MODE (pos_rtx))
    7992           78 :               && ((nonzero_bits (pos_rtx, GET_MODE (pos_rtx))
    7993           78 :                    & ~(((unsigned HOST_WIDE_INT)
    7994           78 :                         GET_MODE_MASK (GET_MODE (pos_rtx)))
    7995           78 :                        >> 1))
    7996              :                   == 0)))
    7997              :         {
    7998           56 :           rtx temp1 = simplify_gen_unary (SIGN_EXTEND, pos_mode, pos_rtx,
    7999              :                                           GET_MODE (pos_rtx));
    8000              : 
    8001              :           /* Prefer ZERO_EXTENSION, since it gives more information to
    8002              :              backends.  */
    8003           56 :           if (set_src_cost (temp1, pos_mode, optimize_this_for_speed_p)
    8004           56 :               < set_src_cost (temp, pos_mode, optimize_this_for_speed_p))
    8005      1431921 :             temp = temp1;
    8006              :         }
    8007              :       pos_rtx = temp;
    8008              :     }
    8009              : 
    8010              :   /* Make POS_RTX unless we already have it and it is correct.  If we don't
    8011              :      have a POS_RTX but we do have an ORIG_POS_RTX, the latter must
    8012              :      be a CONST_INT.  */
    8013      1431921 :   if (pos_rtx == 0 && orig_pos_rtx != 0 && INTVAL (orig_pos_rtx) == pos)
    8014              :     pos_rtx = orig_pos_rtx;
    8015              : 
    8016       507227 :   else if (pos_rtx == 0)
    8017       485149 :     pos_rtx = GEN_INT (pos);
    8018              : 
    8019              :   /* Make the required operation.  See if we can use existing rtx.  */
    8020      1431921 :   new_rtx = gen_rtx_fmt_eee (unsignedp ? ZERO_EXTRACT : SIGN_EXTRACT,
    8021              :                          extraction_mode, inner, GEN_INT (len), pos_rtx);
    8022      1431921 :   if (! in_dest)
    8023      1425325 :     new_rtx = gen_lowpart (mode, new_rtx);
    8024              : 
    8025              :   return new_rtx;
    8026              : }
    8027              : 
    8028              : /* See if X (of mode MODE) contains an ASHIFT of COUNT or more bits that
    8029              :    can be commuted with any other operations in X.  Return X without
    8030              :    that shift if so.  */
    8031              : 
    8032              : static rtx
    8033      1634031 : extract_left_shift (scalar_int_mode mode, rtx x, int count)
    8034              : {
    8035      1634031 :   enum rtx_code code = GET_CODE (x);
    8036      1634031 :   rtx tem;
    8037              : 
    8038      1634031 :   switch (code)
    8039              :     {
    8040       266634 :     case ASHIFT:
    8041              :       /* This is the shift itself.  If it is wide enough, we will return
    8042              :          either the value being shifted if the shift count is equal to
    8043              :          COUNT or a shift for the difference.  */
    8044       266634 :       if (CONST_INT_P (XEXP (x, 1))
    8045       261286 :           && INTVAL (XEXP (x, 1)) >= count)
    8046       260078 :         return simplify_shift_const (NULL_RTX, ASHIFT, mode, XEXP (x, 0),
    8047       260078 :                                      INTVAL (XEXP (x, 1)) - count);
    8048              :       break;
    8049              : 
    8050         5374 :     case NEG:  case NOT:
    8051         5374 :       if ((tem = extract_left_shift (mode, XEXP (x, 0), count)) != 0)
    8052         2569 :         return simplify_gen_unary (code, mode, tem, mode);
    8053              : 
    8054              :       break;
    8055              : 
    8056       582405 :     case PLUS:  case IOR:  case XOR:  case AND:
    8057              :       /* If we can safely shift this constant and we find the inner shift,
    8058              :          make a new operation.  */
    8059       582405 :       if (CONST_INT_P (XEXP (x, 1))
    8060       309994 :           && (UINTVAL (XEXP (x, 1))
    8061       309994 :               & (((HOST_WIDE_INT_1U << count)) - 1)) == 0
    8062       733091 :           && (tem = extract_left_shift (mode, XEXP (x, 0), count)) != 0)
    8063              :         {
    8064         6889 :           HOST_WIDE_INT val = INTVAL (XEXP (x, 1)) >> count;
    8065         6889 :           return simplify_gen_binary (code, mode, tem,
    8066         6889 :                                       gen_int_mode (val, mode));
    8067              :         }
    8068              :       break;
    8069              : 
    8070              :     default:
    8071              :       break;
    8072              :     }
    8073              : 
    8074              :   return 0;
    8075              : }
    8076              : 
    8077              : /* Subroutine of make_compound_operation.  *X_PTR is the rtx at the current
    8078              :    level of the expression and MODE is its mode.  IN_CODE is as for
    8079              :    make_compound_operation.  *NEXT_CODE_PTR is the value of IN_CODE
    8080              :    that should be used when recursing on operands of *X_PTR.
    8081              : 
    8082              :    There are two possible actions:
    8083              : 
    8084              :    - Return null.  This tells the caller to recurse on *X_PTR with IN_CODE
    8085              :      equal to *NEXT_CODE_PTR, after which *X_PTR holds the final value.
    8086              : 
    8087              :    - Return a new rtx, which the caller returns directly.  */
    8088              : 
    8089              : static rtx
    8090    285472410 : make_compound_operation_int (scalar_int_mode mode, rtx *x_ptr,
    8091              :                              enum rtx_code in_code,
    8092              :                              enum rtx_code *next_code_ptr)
    8093              : {
    8094    285472410 :   rtx x = *x_ptr;
    8095    285472410 :   enum rtx_code next_code = *next_code_ptr;
    8096    285472410 :   enum rtx_code code = GET_CODE (x);
    8097    285472410 :   int mode_width = GET_MODE_PRECISION (mode);
    8098    285472410 :   rtx rhs, lhs;
    8099    285472410 :   rtx new_rtx = 0;
    8100    285472410 :   int i;
    8101    285472410 :   rtx tem;
    8102    285472410 :   scalar_int_mode inner_mode;
    8103    285472410 :   bool equality_comparison = false;
    8104              : 
    8105    285472410 :   if (in_code == EQ)
    8106              :     {
    8107      9201963 :       equality_comparison = true;
    8108      9201963 :       in_code = COMPARE;
    8109              :     }
    8110              : 
    8111              :   /* Process depending on the code of this operation.  If NEW is set
    8112              :      nonzero, it will be returned.  */
    8113              : 
    8114    285472410 :   switch (code)
    8115              :     {
    8116      6581561 :     case ASHIFT:
    8117              :       /* Convert shifts by constants into multiplications if inside
    8118              :          an address.  */
    8119      6581561 :       if (in_code == MEM && CONST_INT_P (XEXP (x, 1))
    8120      1977922 :           && INTVAL (XEXP (x, 1)) < HOST_BITS_PER_WIDE_INT
    8121      1977922 :           && INTVAL (XEXP (x, 1)) >= 0)
    8122              :         {
    8123      1977922 :           HOST_WIDE_INT count = INTVAL (XEXP (x, 1));
    8124      1977922 :           HOST_WIDE_INT multval = HOST_WIDE_INT_1 << count;
    8125              : 
    8126      1977922 :           new_rtx = make_compound_operation (XEXP (x, 0), next_code);
    8127      1977922 :           if (GET_CODE (new_rtx) == NEG)
    8128              :             {
    8129            9 :               new_rtx = XEXP (new_rtx, 0);
    8130            9 :               multval = -multval;
    8131              :             }
    8132      1977922 :           multval = trunc_int_for_mode (multval, mode);
    8133      1977922 :           new_rtx = gen_rtx_MULT (mode, new_rtx, gen_int_mode (multval, mode));
    8134              :         }
    8135              :       break;
    8136              : 
    8137     56273170 :     case PLUS:
    8138     56273170 :     case MINUS:
    8139     56273170 :       lhs = make_compound_operation (XEXP (x, 0), next_code);
    8140     56273170 :       rhs = make_compound_operation (XEXP (x, 1), next_code);
    8141     56273170 :       if (lhs != XEXP (x, 0) || rhs != XEXP (x, 1))
    8142      3254939 :         return simplify_gen_binary (code, mode, lhs, rhs);
    8143              :       return x;
    8144              : 
    8145      7654948 :     case AND:
    8146              :       /* If the second operand is not a constant, we can't do anything
    8147              :          with it.  */
    8148      7654948 :       if (!CONST_INT_P (XEXP (x, 1)))
    8149              :         break;
    8150              : 
    8151              :       /* If the constant is a power of two minus one and the first operand
    8152              :          is a logical right shift, make an extraction.  */
    8153      6045534 :       if (GET_CODE (XEXP (x, 0)) == LSHIFTRT
    8154      6045534 :           && (i = exact_log2 (UINTVAL (XEXP (x, 1)) + 1)) >= 0)
    8155              :         {
    8156       656148 :           new_rtx = make_compound_operation (XEXP (XEXP (x, 0), 0), next_code);
    8157       656148 :           new_rtx = make_extraction (mode, new_rtx, 0, XEXP (XEXP (x, 0), 1),
    8158              :                                      i, true, false, in_code == COMPARE);
    8159              :         }
    8160              : 
    8161              :       /* Same as previous, but for (subreg (lshiftrt ...)) in first op.  */
    8162      5389386 :       else if (GET_CODE (XEXP (x, 0)) == SUBREG
    8163      1412848 :                && subreg_lowpart_p (XEXP (x, 0))
    8164      6757080 :                && is_a <scalar_int_mode> (GET_MODE (SUBREG_REG (XEXP (x, 0))),
    8165              :                                           &inner_mode)
    8166      1397945 :                && GET_CODE (SUBREG_REG (XEXP (x, 0))) == LSHIFTRT
    8167      5421735 :                && (i = exact_log2 (UINTVAL (XEXP (x, 1)) + 1)) >= 0)
    8168              :         {
    8169        30251 :           rtx inner_x0 = SUBREG_REG (XEXP (x, 0));
    8170        30251 :           new_rtx = make_compound_operation (XEXP (inner_x0, 0), next_code);
    8171        30251 :           new_rtx = make_extraction (inner_mode, new_rtx, 0,
    8172              :                                      XEXP (inner_x0, 1),
    8173              :                                      i, true, false, in_code == COMPARE);
    8174              : 
    8175              :           /* If we narrowed the mode when dropping the subreg, then we lose.  */
    8176        90753 :           if (GET_MODE_SIZE (inner_mode) < GET_MODE_SIZE (mode))
    8177        30251 :             new_rtx = NULL;
    8178              : 
    8179              :           /* If that didn't give anything, see if the AND simplifies on
    8180              :              its own.  */
    8181        30251 :           if (!new_rtx && i >= 0)
    8182              :             {
    8183         3320 :               new_rtx = make_compound_operation (XEXP (x, 0), next_code);
    8184         3320 :               new_rtx = make_extraction (mode, new_rtx, 0, NULL_RTX, i,
    8185              :                                          true, false, in_code == COMPARE);
    8186              :             }
    8187              :         }
    8188              :       /* Same as previous, but for (xor/ior (lshiftrt...) (lshiftrt...)).  */
    8189      5359135 :       else if ((GET_CODE (XEXP (x, 0)) == XOR
    8190      5359135 :                 || GET_CODE (XEXP (x, 0)) == IOR)
    8191        31307 :                && GET_CODE (XEXP (XEXP (x, 0), 0)) == LSHIFTRT
    8192         2738 :                && GET_CODE (XEXP (XEXP (x, 0), 1)) == LSHIFTRT
    8193      5359145 :                && (i = exact_log2 (UINTVAL (XEXP (x, 1)) + 1)) >= 0)
    8194              :         {
    8195              :           /* Apply the distributive law, and then try to make extractions.  */
    8196           10 :           new_rtx = gen_rtx_fmt_ee (GET_CODE (XEXP (x, 0)), mode,
    8197              :                                     gen_rtx_AND (mode, XEXP (XEXP (x, 0), 0),
    8198              :                                                  XEXP (x, 1)),
    8199              :                                     gen_rtx_AND (mode, XEXP (XEXP (x, 0), 1),
    8200              :                                                  XEXP (x, 1)));
    8201           10 :           new_rtx = make_compound_operation (new_rtx, in_code);
    8202              :         }
    8203              : 
    8204              :       /* If we are have (and (rotate X C) M) and C is larger than the number
    8205              :          of bits in M, this is an extraction.  */
    8206              : 
    8207      5359125 :       else if (GET_CODE (XEXP (x, 0)) == ROTATE
    8208          935 :                && CONST_INT_P (XEXP (XEXP (x, 0), 1))
    8209          925 :                && (i = exact_log2 (UINTVAL (XEXP (x, 1)) + 1)) >= 0
    8210      5359162 :                && i <= INTVAL (XEXP (XEXP (x, 0), 1)))
    8211              :         {
    8212            0 :           new_rtx = make_compound_operation (XEXP (XEXP (x, 0), 0), next_code);
    8213            0 :           new_rtx = make_extraction (mode, new_rtx,
    8214            0 :                                      (GET_MODE_PRECISION (mode)
    8215            0 :                                       - INTVAL (XEXP (XEXP (x, 0), 1))),
    8216              :                                      NULL_RTX, i, true, false,
    8217              :                                      in_code == COMPARE);
    8218              :         }
    8219              : 
    8220              :       /* On machines without logical shifts, if the operand of the AND is
    8221              :          a logical shift and our mask turns off all the propagated sign
    8222              :          bits, we can replace the logical shift with an arithmetic shift.  */
    8223      5359125 :       else if (GET_CODE (XEXP (x, 0)) == LSHIFTRT
    8224        90306 :                && !have_insn_for (LSHIFTRT, mode)
    8225            0 :                && have_insn_for (ASHIFTRT, mode)
    8226            0 :                && CONST_INT_P (XEXP (XEXP (x, 0), 1))
    8227            0 :                && INTVAL (XEXP (XEXP (x, 0), 1)) >= 0
    8228            0 :                && INTVAL (XEXP (XEXP (x, 0), 1)) < HOST_BITS_PER_WIDE_INT
    8229      5359125 :                && mode_width <= HOST_BITS_PER_WIDE_INT)
    8230              :         {
    8231            0 :           unsigned HOST_WIDE_INT mask = GET_MODE_MASK (mode);
    8232              : 
    8233            0 :           mask >>= INTVAL (XEXP (XEXP (x, 0), 1));
    8234            0 :           if ((INTVAL (XEXP (x, 1)) & ~mask) == 0)
    8235            0 :             SUBST (XEXP (x, 0),
    8236              :                    gen_rtx_ASHIFTRT (mode,
    8237              :                                      make_compound_operation (XEXP (XEXP (x,
    8238              :                                                                           0),
    8239              :                                                                     0),
    8240              :                                                               next_code),
    8241              :                                      XEXP (XEXP (x, 0), 1)));
    8242              :         }
    8243              : 
    8244              :       /* If the constant is one less than a power of two, this might be
    8245              :          representable by an extraction even if no shift is present.
    8246              :          If it doesn't end up being a ZERO_EXTEND, we will ignore it unless
    8247              :          we are in a COMPARE.  */
    8248      5359125 :       else if ((i = exact_log2 (UINTVAL (XEXP (x, 1)) + 1)) >= 0)
    8249      2775168 :         new_rtx = make_extraction (mode,
    8250              :                                    make_compound_operation (XEXP (x, 0),
    8251              :                                                             next_code),
    8252              :                                    0, NULL_RTX, i,
    8253              :                                    true, false, in_code == COMPARE);
    8254              : 
    8255              :       /* If we are in a comparison and this is an AND with a power of two,
    8256              :          convert this into the appropriate bit extract.  */
    8257      2583957 :       else if (in_code == COMPARE
    8258       478230 :                && (i = exact_log2 (UINTVAL (XEXP (x, 1)))) >= 0
    8259      2665365 :                && (equality_comparison || i < GET_MODE_PRECISION (mode) - 1))
    8260        81408 :         new_rtx = make_extraction (mode,
    8261              :                                    make_compound_operation (XEXP (x, 0),
    8262              :                                                             next_code),
    8263              :                                    i, NULL_RTX, 1, true, false, true);
    8264              : 
    8265              :       /* If the one operand is a paradoxical subreg of a register or memory and
    8266              :          the constant (limited to the smaller mode) has only zero bits where
    8267              :          the sub expression has known zero bits, this can be expressed as
    8268              :          a zero_extend.  */
    8269      2502549 :       else if (GET_CODE (XEXP (x, 0)) == SUBREG)
    8270              :         {
    8271        76789 :           rtx sub;
    8272              : 
    8273        76789 :           sub = XEXP (XEXP (x, 0), 0);
    8274        76789 :           machine_mode sub_mode = GET_MODE (sub);
    8275        76789 :           int sub_width;
    8276        30031 :           if ((REG_P (sub) || MEM_P (sub))
    8277        47301 :               && GET_MODE_PRECISION (sub_mode).is_constant (&sub_width)
    8278        47301 :               && sub_width < mode_width
    8279        76789 :               && (!WORD_REGISTER_OPERATIONS
    8280              :                   || sub_width >= BITS_PER_WORD
    8281              :                   /* On WORD_REGISTER_OPERATIONS targets the bits
    8282              :                      beyond sub_mode aren't considered undefined,
    8283              :                      so optimize only if it is a MEM load when MEM loads
    8284              :                      zero extend, because then the upper bits are all zero.  */
    8285              :                   || (MEM_P (sub)
    8286              :                       && load_extend_op (sub_mode) == ZERO_EXTEND)))
    8287              :             {
    8288        23297 :               unsigned HOST_WIDE_INT mode_mask = GET_MODE_MASK (sub_mode);
    8289        23297 :               unsigned HOST_WIDE_INT mask;
    8290              : 
    8291              :               /* Original AND constant with all the known zero bits set.  */
    8292        23297 :               mask = UINTVAL (XEXP (x, 1)) | (~nonzero_bits (sub, sub_mode));
    8293        23297 :               if ((mask & mode_mask) == mode_mask)
    8294              :                 {
    8295        17357 :                   new_rtx = make_compound_operation (sub, next_code);
    8296        17357 :                   new_rtx = make_extraction (mode, new_rtx, 0, 0, sub_width,
    8297              :                                              true, false, in_code == COMPARE);
    8298              :                 }
    8299              :             }
    8300              :         }
    8301              : 
    8302              :       break;
    8303              : 
    8304      2009207 :     case LSHIFTRT:
    8305              :       /* If the sign bit is known to be zero, replace this with an
    8306              :          arithmetic shift.  */
    8307      2009207 :       if (have_insn_for (ASHIFTRT, mode)
    8308      2009207 :           && ! have_insn_for (LSHIFTRT, mode)
    8309            0 :           && mode_width <= HOST_BITS_PER_WIDE_INT
    8310      2009207 :           && (nonzero_bits (XEXP (x, 0), mode) & (1 << (mode_width - 1))) == 0)
    8311              :         {
    8312            0 :           new_rtx = gen_rtx_ASHIFTRT (mode,
    8313              :                                       make_compound_operation (XEXP (x, 0),
    8314              :                                                                next_code),
    8315              :                                       XEXP (x, 1));
    8316            0 :           break;
    8317              :         }
    8318              : 
    8319              :       /* fall through */
    8320              : 
    8321      4910849 :     case ASHIFTRT:
    8322      4910849 :       lhs = XEXP (x, 0);
    8323      4910849 :       rhs = XEXP (x, 1);
    8324              : 
    8325              :       /* If we have (ashiftrt (ashift foo C1) C2) with C2 >= C1,
    8326              :          this is a SIGN_EXTRACT.  */
    8327      4910849 :       if (CONST_INT_P (rhs)
    8328      4729500 :           && GET_CODE (lhs) == ASHIFT
    8329      1152854 :           && CONST_INT_P (XEXP (lhs, 1))
    8330      1147510 :           && INTVAL (rhs) >= INTVAL (XEXP (lhs, 1))
    8331       896825 :           && INTVAL (XEXP (lhs, 1)) >= 0
    8332       896821 :           && INTVAL (rhs) < mode_width)
    8333              :         {
    8334       896820 :           new_rtx = make_compound_operation (XEXP (lhs, 0), next_code);
    8335       896820 :           new_rtx = make_extraction (mode, new_rtx,
    8336       896820 :                                      INTVAL (rhs) - INTVAL (XEXP (lhs, 1)),
    8337       896820 :                                      NULL_RTX, mode_width - INTVAL (rhs),
    8338              :                                      code == LSHIFTRT, false,
    8339              :                                      in_code == COMPARE);
    8340       896820 :           break;
    8341              :         }
    8342              : 
    8343              :       /* See if we have operations between an ASHIFTRT and an ASHIFT.
    8344              :          If so, try to merge the shifts into a SIGN_EXTEND.  We could
    8345              :          also do this for some cases of SIGN_EXTRACT, but it doesn't
    8346              :          seem worth the effort; the case checked for occurs on Alpha.  */
    8347              : 
    8348      4014029 :       if (!OBJECT_P (lhs)
    8349      1560935 :           && ! (GET_CODE (lhs) == SUBREG
    8350        66333 :                 && (OBJECT_P (SUBREG_REG (lhs))))
    8351      1509290 :           && CONST_INT_P (rhs)
    8352      1483994 :           && INTVAL (rhs) >= 0
    8353      1483994 :           && INTVAL (rhs) < HOST_BITS_PER_WIDE_INT
    8354      1477972 :           && INTVAL (rhs) < mode_width
    8355      5492000 :           && (new_rtx = extract_left_shift (mode, lhs, INTVAL (rhs))) != 0)
    8356       260078 :         new_rtx = make_extraction (mode, make_compound_operation (new_rtx,
    8357              :                                                                   next_code),
    8358       260078 :                                    0, NULL_RTX, mode_width - INTVAL (rhs),
    8359              :                                    code == LSHIFTRT, false, in_code == COMPARE);
    8360              : 
    8361              :       break;
    8362              : 
    8363      9791927 :     case SUBREG:
    8364              :       /* Call ourselves recursively on the inner expression.  If we are
    8365              :          narrowing the object and it has a different RTL code from
    8366              :          what it originally did, do this SUBREG as a force_to_mode.  */
    8367      9791927 :       {
    8368      9791927 :         rtx inner = SUBREG_REG (x), simplified;
    8369      9791927 :         enum rtx_code subreg_code = in_code;
    8370              : 
    8371              :         /* If the SUBREG is masking of a logical right shift,
    8372              :            make an extraction.  */
    8373      9791927 :         if (GET_CODE (inner) == LSHIFTRT
    8374      9802695 :             && is_a <scalar_int_mode> (GET_MODE (inner), &inner_mode)
    8375       602066 :             && GET_MODE_SIZE (mode) < GET_MODE_SIZE (inner_mode)
    8376       297050 :             && CONST_INT_P (XEXP (inner, 1))
    8377       291657 :             && UINTVAL (XEXP (inner, 1)) < GET_MODE_PRECISION (inner_mode)
    8378     10083584 :             && subreg_lowpart_p (x))
    8379              :           {
    8380       290265 :             new_rtx = make_compound_operation (XEXP (inner, 0), next_code);
    8381       290265 :             int width = GET_MODE_PRECISION (inner_mode)
    8382       290265 :                         - INTVAL (XEXP (inner, 1));
    8383       290265 :             if (width > mode_width)
    8384              :               width = mode_width;
    8385       290265 :             new_rtx = make_extraction (mode, new_rtx, 0, XEXP (inner, 1),
    8386              :                                        width, true, false, in_code == COMPARE);
    8387       290265 :             break;
    8388              :           }
    8389              : 
    8390              :         /* If in_code is COMPARE, it isn't always safe to pass it through
    8391              :            to the recursive make_compound_operation call.  */
    8392      9501662 :         if (subreg_code == COMPARE
    8393      9501662 :             && (!subreg_lowpart_p (x)
    8394       243651 :                 || GET_CODE (inner) == SUBREG
    8395              :                 /* (subreg:SI (and:DI (reg:DI) (const_int 0x800000000)) 0)
    8396              :                    is (const_int 0), rather than
    8397              :                    (subreg:SI (lshiftrt:DI (reg:DI) (const_int 35)) 0).
    8398              :                    Similarly (subreg:QI (and:SI (reg:SI) (const_int 0x80)) 0)
    8399              :                    for non-equality comparisons against 0 is not equivalent
    8400              :                    to (subreg:QI (lshiftrt:SI (reg:SI) (const_int 7)) 0).  */
    8401       243651 :                 || (GET_CODE (inner) == AND
    8402         1234 :                     && CONST_INT_P (XEXP (inner, 1))
    8403          175 :                     && partial_subreg_p (x)
    8404          350 :                     && exact_log2 (UINTVAL (XEXP (inner, 1)))
    8405          175 :                        >= GET_MODE_BITSIZE (mode) - 1)))
    8406              :           subreg_code = SET;
    8407              : 
    8408      9501662 :         tem = make_compound_operation (inner, subreg_code);
    8409              : 
    8410              :         /* TEM's code might be CLOBBER if combine_simplify_rtx
    8411              :            could not transform a subexpression, e.g. a volatile MEM.
    8412              :            simplify_subreg cannot be called with clobber, so bail out.  */
    8413      9501662 :         if (GET_CODE (tem) == CLOBBER)
    8414              :           return NULL_RTX;
    8415              : 
    8416      9501643 :         simplified
    8417      9501643 :           = simplify_subreg (mode, tem, GET_MODE (inner), SUBREG_BYTE (x));
    8418      9501643 :         if (simplified)
    8419        16632 :           tem = simplified;
    8420              : 
    8421      9501643 :         if (GET_CODE (tem) != GET_CODE (inner)
    8422        21914 :             && partial_subreg_p (x)
    8423      9520246 :             && subreg_lowpart_p (x))
    8424              :           {
    8425        18587 :             rtx newer
    8426        18587 :               = force_to_mode (tem, mode, HOST_WIDE_INT_M1U, false);
    8427              : 
    8428              :             /* If we have something other than a SUBREG, we might have
    8429              :                done an expansion, so rerun ourselves.  */
    8430        18587 :             if (GET_CODE (newer) != SUBREG)
    8431        17161 :               newer = make_compound_operation (newer, in_code);
    8432              : 
    8433              :             /* force_to_mode can expand compounds.  If it just re-expanded
    8434              :                the compound, use gen_lowpart to convert to the desired
    8435              :                mode.  */
    8436        18587 :             if (rtx_equal_p (newer, x)
    8437              :                 /* Likewise if it re-expanded the compound only partially.
    8438              :                    This happens for SUBREG of ZERO_EXTRACT if they extract
    8439              :                    the same number of bits.  */
    8440        18587 :                 || (GET_CODE (newer) == SUBREG
    8441         2230 :                     && (GET_CODE (SUBREG_REG (newer)) == LSHIFTRT
    8442         2230 :                         || GET_CODE (SUBREG_REG (newer)) == ASHIFTRT)
    8443           78 :                     && GET_CODE (inner) == AND
    8444           60 :                     && rtx_equal_p (SUBREG_REG (newer), XEXP (inner, 0))))
    8445          779 :               return gen_lowpart (GET_MODE (x), tem);
    8446              : 
    8447              :             return newer;
    8448              :           }
    8449              : 
    8450      9483056 :         if (simplified)
    8451              :           return tem;
    8452              :       }
    8453              :       break;
    8454              : 
    8455              :     default:
    8456              :       break;
    8457              :     }
    8458              : 
    8459     10739378 :   if (new_rtx)
    8460      5522379 :     *x_ptr = gen_lowpart (mode, new_rtx);
    8461    229180261 :   *next_code_ptr = next_code;
    8462    229180261 :   return NULL_RTX;
    8463              : }
    8464              : 
    8465              : /* Look at the expression rooted at X.  Look for expressions
    8466              :    equivalent to ZERO_EXTRACT, SIGN_EXTRACT, ZERO_EXTEND, SIGN_EXTEND.
    8467              :    Form these expressions.
    8468              : 
    8469              :    Return the new rtx, usually just X.
    8470              : 
    8471              :    Also, for machines like the VAX that don't have logical shift insns,
    8472              :    try to convert logical to arithmetic shift operations in cases where
    8473              :    they are equivalent.  This undoes the canonicalizations to logical
    8474              :    shifts done elsewhere.
    8475              : 
    8476              :    We try, as much as possible, to re-use rtl expressions to save memory.
    8477              : 
    8478              :    IN_CODE says what kind of expression we are processing.  Normally, it is
    8479              :    SET.  In a memory address it is MEM.  When processing the arguments of
    8480              :    a comparison or a COMPARE against zero, it is COMPARE, or EQ if more
    8481              :    precisely it is an equality comparison against zero.  */
    8482              : 
    8483              : rtx
    8484    489090191 : make_compound_operation (rtx x, enum rtx_code in_code)
    8485              : {
    8486    489090191 :   enum rtx_code code = GET_CODE (x);
    8487    489090191 :   const char *fmt;
    8488    489090191 :   int i, j;
    8489    489090191 :   enum rtx_code next_code;
    8490    489090191 :   rtx new_rtx, tem;
    8491              : 
    8492              :   /* Select the code to be used in recursive calls.  Once we are inside an
    8493              :      address, we stay there.  If we have a comparison, set to COMPARE,
    8494              :      but once inside, go back to our default of SET.  */
    8495              : 
    8496    489090191 :   next_code = (code == MEM ? MEM
    8497    460307067 :                : ((code == COMPARE || COMPARISON_P (x))
    8498    481110372 :                   && XEXP (x, 1) == const0_rtx) ? COMPARE
    8499    452057981 :                : in_code == COMPARE || in_code == EQ ? SET : in_code);
    8500              : 
    8501    489090191 :   scalar_int_mode mode;
    8502    489090191 :   if (is_a <scalar_int_mode> (GET_MODE (x), &mode))
    8503              :     {
    8504    285472410 :       rtx new_rtx = make_compound_operation_int (mode, &x, in_code,
    8505              :                                                  &next_code);
    8506    285472410 :       if (new_rtx)
    8507              :         return new_rtx;
    8508    229180280 :       code = GET_CODE (x);
    8509              :     }
    8510              : 
    8511              :   /* Now recursively process each operand of this operation.  We need to
    8512              :      handle ZERO_EXTEND specially so that we don't lose track of the
    8513              :      inner mode.  */
    8514    432798061 :   if (code == ZERO_EXTEND)
    8515              :     {
    8516      3450753 :       new_rtx = make_compound_operation (XEXP (x, 0), next_code);
    8517      6901506 :       tem = simplify_unary_operation (ZERO_EXTEND, GET_MODE (x),
    8518      3450753 :                                       new_rtx, GET_MODE (XEXP (x, 0)));
    8519      3450753 :       if (tem)
    8520              :         return tem;
    8521      3440364 :       SUBST (XEXP (x, 0), new_rtx);
    8522      3440364 :       return x;
    8523              :     }
    8524              : 
    8525    429347308 :   fmt = GET_RTX_FORMAT (code);
    8526    998960577 :   for (i = 0; i < GET_RTX_LENGTH (code); i++)
    8527    569613269 :     if (fmt[i] == 'e')
    8528              :       {
    8529    217961791 :         new_rtx = make_compound_operation (XEXP (x, i), next_code);
    8530    217961791 :         SUBST (XEXP (x, i), new_rtx);
    8531              :       }
    8532    351651478 :     else if (fmt[i] == 'E')
    8533     27868106 :       for (j = 0; j < XVECLEN (x, i); j++)
    8534              :         {
    8535     20284540 :           new_rtx = make_compound_operation (XVECEXP (x, i, j), next_code);
    8536     20284540 :           SUBST (XVECEXP (x, i, j), new_rtx);
    8537              :         }
    8538              : 
    8539    429347308 :   maybe_swap_commutative_operands (x);
    8540    429347308 :   return x;
    8541              : }
    8542              : 
    8543              : /* Given M see if it is a value that would select a field of bits
    8544              :    within an item, but not the entire word.  Return -1 if not.
    8545              :    Otherwise, return the starting position of the field, where 0 is the
    8546              :    low-order bit.
    8547              : 
    8548              :    *PLEN is set to the length of the field.  */
    8549              : 
    8550              : static int
    8551        22292 : get_pos_from_mask (unsigned HOST_WIDE_INT m, unsigned HOST_WIDE_INT *plen)
    8552              : {
    8553              :   /* Get the bit number of the first 1 bit from the right, -1 if none.  */
    8554        22292 :   int pos = m ? ctz_hwi (m) : -1;
    8555        22292 :   int len = 0;
    8556              : 
    8557        22292 :   if (pos >= 0)
    8558              :     /* Now shift off the low-order zero bits and see if we have a
    8559              :        power of two minus 1.  */
    8560        22292 :     len = exact_log2 ((m >> pos) + 1);
    8561              : 
    8562        10852 :   if (len <= 0)
    8563              :     pos = -1;
    8564              : 
    8565        22292 :   *plen = len;
    8566        22292 :   return pos;
    8567              : }
    8568              : 
    8569              : /* If X refers to a register that equals REG in value, replace these
    8570              :    references with REG.  */
    8571              : static rtx
    8572         9418 : canon_reg_for_combine (rtx x, rtx reg)
    8573              : {
    8574         9418 :   rtx op0, op1, op2;
    8575         9418 :   const char *fmt;
    8576         9418 :   int i;
    8577         9418 :   bool copied;
    8578              : 
    8579         9418 :   enum rtx_code code = GET_CODE (x);
    8580         9418 :   switch (GET_RTX_CLASS (code))
    8581              :     {
    8582            0 :     case RTX_UNARY:
    8583            0 :       op0 = canon_reg_for_combine (XEXP (x, 0), reg);
    8584            0 :       if (op0 != XEXP (x, 0))
    8585            0 :         return simplify_gen_unary (GET_CODE (x), GET_MODE (x), op0,
    8586            0 :                                    GET_MODE (reg));
    8587              :       break;
    8588              : 
    8589         1672 :     case RTX_BIN_ARITH:
    8590         1672 :     case RTX_COMM_ARITH:
    8591         1672 :       op0 = canon_reg_for_combine (XEXP (x, 0), reg);
    8592         1672 :       op1 = canon_reg_for_combine (XEXP (x, 1), reg);
    8593         1672 :       if (op0 != XEXP (x, 0) || op1 != XEXP (x, 1))
    8594            0 :         return simplify_gen_binary (GET_CODE (x), GET_MODE (x), op0, op1);
    8595              :       break;
    8596              : 
    8597           17 :     case RTX_COMPARE:
    8598           17 :     case RTX_COMM_COMPARE:
    8599           17 :       op0 = canon_reg_for_combine (XEXP (x, 0), reg);
    8600           17 :       op1 = canon_reg_for_combine (XEXP (x, 1), reg);
    8601           17 :       if (op0 != XEXP (x, 0) || op1 != XEXP (x, 1))
    8602            0 :         return simplify_gen_relational (GET_CODE (x), GET_MODE (x),
    8603            0 :                                         GET_MODE (op0), op0, op1);
    8604              :       break;
    8605              : 
    8606            2 :     case RTX_TERNARY:
    8607            2 :     case RTX_BITFIELD_OPS:
    8608            2 :       op0 = canon_reg_for_combine (XEXP (x, 0), reg);
    8609            2 :       op1 = canon_reg_for_combine (XEXP (x, 1), reg);
    8610            2 :       op2 = canon_reg_for_combine (XEXP (x, 2), reg);
    8611            2 :       if (op0 != XEXP (x, 0) || op1 != XEXP (x, 1) || op2 != XEXP (x, 2))
    8612            0 :         return simplify_gen_ternary (GET_CODE (x), GET_MODE (x),
    8613            0 :                                      GET_MODE (op0), op0, op1, op2);
    8614              :       /* FALLTHRU */
    8615              : 
    8616         5434 :     case RTX_OBJ:
    8617         5434 :       if (REG_P (x))
    8618              :         {
    8619         5426 :           if (rtx_equal_p (get_last_value (reg), x)
    8620         5426 :               || rtx_equal_p (reg, get_last_value (x)))
    8621            0 :             return reg;
    8622              :           else
    8623              :             break;
    8624              :         }
    8625              : 
    8626              :       /* fall through */
    8627              : 
    8628         2303 :     default:
    8629         2303 :       fmt = GET_RTX_FORMAT (code);
    8630         2303 :       copied = false;
    8631         4672 :       for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
    8632         2369 :         if (fmt[i] == 'e')
    8633              :           {
    8634           68 :             rtx op = canon_reg_for_combine (XEXP (x, i), reg);
    8635           68 :             if (op != XEXP (x, i))
    8636              :               {
    8637            0 :                 if (!copied)
    8638              :                   {
    8639            0 :                     copied = true;
    8640            0 :                     x = copy_rtx (x);
    8641              :                   }
    8642            0 :                 XEXP (x, i) = op;
    8643              :               }
    8644              :           }
    8645         2301 :         else if (fmt[i] == 'E')
    8646              :           {
    8647              :             int j;
    8648            0 :             for (j = 0; j < XVECLEN (x, i); j++)
    8649              :               {
    8650            0 :                 rtx op = canon_reg_for_combine (XVECEXP (x, i, j), reg);
    8651            0 :                 if (op != XVECEXP (x, i, j))
    8652              :                   {
    8653            0 :                     if (!copied)
    8654              :                       {
    8655            0 :                         copied = true;
    8656            0 :                         x = copy_rtx (x);
    8657              :                       }
    8658            0 :                     XVECEXP (x, i, j) = op;
    8659              :                   }
    8660              :               }
    8661              :           }
    8662              : 
    8663              :       break;
    8664              :     }
    8665              : 
    8666              :   return x;
    8667              : }
    8668              : 
    8669              : /* Return X converted to MODE.  If the value is already truncated to
    8670              :    MODE we can just return a subreg even though in the general case we
    8671              :    would need an explicit truncation.
    8672              : 
    8673              :    If an explicit truncation is needed but either mode has no
    8674              :    equivalent integer mode (e.g. a wide vector mode), return
    8675              :    (clobber (const_int 0)) so that the combination fails.  */
    8676              : 
    8677              : static rtx
    8678    121107705 : gen_lowpart_or_truncate (machine_mode mode, rtx x)
    8679              : {
    8680    121107705 :   if (!CONST_INT_P (x)
    8681    115392198 :       && partial_subreg_p (mode, GET_MODE (x))
    8682    121107705 :       && !TRULY_NOOP_TRUNCATION_MODES_P (mode, GET_MODE (x))
    8683    121107705 :       && !(REG_P (x) && reg_truncated_to_mode (mode, x)))
    8684              :     {
    8685            0 :       scalar_int_mode imode, omode;
    8686            0 :       if (!int_mode_for_mode (GET_MODE (x)).exists (&imode)
    8687            0 :           || !int_mode_for_mode (mode).exists (&omode))
    8688            0 :         return gen_rtx_CLOBBER (mode, const0_rtx);
    8689              : 
    8690              :       /* Bit-cast X into an integer mode.  */
    8691            0 :       if (!SCALAR_INT_MODE_P (GET_MODE (x)))
    8692            0 :         x = gen_lowpart (imode, x);
    8693            0 :       x = simplify_gen_unary (TRUNCATE, omode, x, GET_MODE (x));
    8694              :     }
    8695              : 
    8696    121107705 :   return gen_lowpart (mode, x);
    8697              : }
    8698              : 
    8699              : /* See if X can be simplified knowing that we will only refer to it in
    8700              :    MODE and will only refer to those bits that are nonzero in MASK.
    8701              :    If other bits are being computed or if masking operations are done
    8702              :    that select a superset of the bits in MASK, they can sometimes be
    8703              :    ignored.
    8704              : 
    8705              :    Return a possibly simplified expression, but always convert X to
    8706              :    MODE.  If X is a CONST_INT, AND the CONST_INT with MASK.
    8707              : 
    8708              :    If JUST_SELECT is true, don't optimize by noticing that bits in MASK
    8709              :    are all off in X.  This is used when X will be complemented, by either
    8710              :    NOT, NEG, or XOR.  */
    8711              : 
    8712              : static rtx
    8713     83660147 : force_to_mode (rtx x, machine_mode mode, unsigned HOST_WIDE_INT mask,
    8714              :                bool just_select)
    8715              : {
    8716     90933729 :   enum rtx_code code = GET_CODE (x);
    8717     90933729 :   bool next_select = just_select || code == XOR || code == NOT || code == NEG;
    8718     90933729 :   machine_mode op_mode;
    8719     90933729 :   unsigned HOST_WIDE_INT nonzero;
    8720              : 
    8721              :   /* If this is a CALL or ASM_OPERANDS, don't do anything.  Some of the
    8722              :      code below will do the wrong thing since the mode of such an
    8723              :      expression is VOIDmode.
    8724              : 
    8725              :      Also do nothing if X is a CLOBBER; this can happen if X was
    8726              :      the return value from a call to gen_lowpart.  */
    8727     90933729 :   if (code == CALL || code == ASM_OPERANDS || code == CLOBBER)
    8728              :     return x;
    8729              : 
    8730              :   /* We want to perform the operation in its present mode unless we know
    8731              :      that the operation is valid in MODE, in which case we do the operation
    8732              :      in MODE.  */
    8733    148823620 :   op_mode = ((GET_MODE_CLASS (mode) == GET_MODE_CLASS (GET_MODE (x))
    8734     84642462 :               && have_insn_for (code, mode))
    8735    142611798 :              ? mode : GET_MODE (x));
    8736              : 
    8737              :   /* It is not valid to do a right-shift in a narrower mode
    8738              :      than the one it came in with.  */
    8739     90854284 :   if ((code == LSHIFTRT || code == ASHIFTRT)
    8740     90854284 :       && partial_subreg_p (mode, GET_MODE (x)))
    8741       399077 :     op_mode = GET_MODE (x);
    8742              : 
    8743              :   /* Truncate MASK to fit OP_MODE.  */
    8744     90854284 :   if (op_mode)
    8745     84721713 :     mask &= GET_MODE_MASK (op_mode);
    8746              : 
    8747              :   /* Determine what bits of X are guaranteed to be (non)zero.  */
    8748     90854284 :   nonzero = nonzero_bits (x, mode);
    8749              : 
    8750              :   /* If none of the bits in X are needed, return a zero.  */
    8751     90854284 :   if (!just_select && (nonzero & mask) == 0 && !side_effects_p (x))
    8752       677558 :     x = const0_rtx;
    8753              : 
    8754              :   /* If X is a CONST_INT, return a new one.  Do this here since the
    8755              :      test below will fail.  */
    8756     90854284 :   if (CONST_INT_P (x))
    8757              :     {
    8758      6345237 :       if (SCALAR_INT_MODE_P (mode))
    8759      6345237 :         return gen_int_mode (INTVAL (x) & mask, mode);
    8760              :       else
    8761              :         {
    8762            0 :           x = GEN_INT (INTVAL (x) & mask);
    8763            0 :           return gen_lowpart_common (mode, x);
    8764              :         }
    8765              :     }
    8766              : 
    8767              :   /* If X is narrower than MODE and we want all the bits in X's mode, just
    8768              :      get X in the proper mode.  */
    8769     84509047 :   if (paradoxical_subreg_p (mode, GET_MODE (x))
    8770     84509047 :       && (GET_MODE_MASK (GET_MODE (x)) & ~mask) == 0)
    8771      3654590 :     return gen_lowpart (mode, x);
    8772              : 
    8773              :   /* We can ignore the effect of a SUBREG if it narrows the mode or
    8774              :      if the constant masks to zero all the bits the mode doesn't have.  */
    8775     80854457 :   if (GET_CODE (x) == SUBREG
    8776      7473361 :       && subreg_lowpart_p (x)
    8777     88163296 :       && (partial_subreg_p (x)
    8778      5487344 :           || (mask
    8779      5487344 :               & GET_MODE_MASK (GET_MODE (x))
    8780      5487344 :               & ~GET_MODE_MASK (GET_MODE (SUBREG_REG (x)))) == 0))
    8781      7273582 :     return force_to_mode (SUBREG_REG (x), mode, mask, next_select);
    8782              : 
    8783     73580875 :   scalar_int_mode int_mode, xmode;
    8784     73580875 :   if (is_a <scalar_int_mode> (mode, &int_mode)
    8785     73580875 :       && is_a <scalar_int_mode> (GET_MODE (x), &xmode))
    8786              :     /* OP_MODE is either MODE or XMODE, so it must be a scalar
    8787              :        integer too.  */
    8788     73542575 :     return force_int_to_mode (x, int_mode, xmode,
    8789              :                               as_a <scalar_int_mode> (op_mode),
    8790     73542575 :                               mask, just_select);
    8791              : 
    8792        38300 :   return gen_lowpart_or_truncate (mode, x);
    8793              : }
    8794              : 
    8795              : /* Subroutine of force_to_mode that handles cases in which both X and
    8796              :    the result are scalar integers.  MODE is the mode of the result,
    8797              :    XMODE is the mode of X, and OP_MODE says which of MODE or XMODE
    8798              :    is preferred for simplified versions of X.  The other arguments
    8799              :    are as for force_to_mode.  */
    8800              : 
    8801              : static rtx
    8802     73542575 : force_int_to_mode (rtx x, scalar_int_mode mode, scalar_int_mode xmode,
    8803              :                    scalar_int_mode op_mode, unsigned HOST_WIDE_INT mask,
    8804              :                    bool just_select)
    8805              : {
    8806     73542575 :   enum rtx_code code = GET_CODE (x);
    8807     73542575 :   bool next_select = just_select || code == XOR || code == NOT || code == NEG;
    8808     73542575 :   unsigned HOST_WIDE_INT fuller_mask;
    8809     73542575 :   rtx op0, op1, temp;
    8810     73542575 :   poly_int64 const_op0;
    8811              : 
    8812              :   /* When we have an arithmetic operation, or a shift whose count we
    8813              :      do not know, we need to assume that all bits up to the highest-order
    8814              :      bit in MASK will be needed.  This is how we form such a mask.  */
    8815     73542575 :   if (mask & (HOST_WIDE_INT_1U << (HOST_BITS_PER_WIDE_INT - 1)))
    8816              :     fuller_mask = HOST_WIDE_INT_M1U;
    8817              :   else
    8818     79193785 :     fuller_mask = ((HOST_WIDE_INT_1U << (floor_log2 (mask) + 1)) - 1);
    8819              : 
    8820     73542575 :   switch (code)
    8821              :     {
    8822              :     case CLOBBER:
    8823              :       /* If X is a (clobber (const_int)), return it since we know we are
    8824              :          generating something that won't match.  */
    8825              :       return x;
    8826              : 
    8827       326474 :     case SIGN_EXTEND:
    8828       326474 :     case ZERO_EXTEND:
    8829       326474 :     case ZERO_EXTRACT:
    8830       326474 :     case SIGN_EXTRACT:
    8831       326474 :       x = expand_compound_operation (x);
    8832       326474 :       if (GET_CODE (x) != code)
    8833       195077 :         return force_to_mode (x, mode, mask, next_select);
    8834              :       break;
    8835              : 
    8836          180 :     case TRUNCATE:
    8837              :       /* Similarly for a truncate.  */
    8838          180 :       return force_to_mode (XEXP (x, 0), mode, mask, next_select);
    8839              : 
    8840      3672150 :     case AND:
    8841              :       /* If this is an AND with a constant, convert it into an AND
    8842              :          whose constant is the AND of that constant with MASK.  If it
    8843              :          remains an AND of MASK, delete it since it is redundant.  */
    8844              : 
    8845      3672150 :       if (CONST_INT_P (XEXP (x, 1)))
    8846              :         {
    8847      5858426 :           x = simplify_and_const_int (x, op_mode, XEXP (x, 0),
    8848      2929213 :                                       mask & INTVAL (XEXP (x, 1)));
    8849      2929213 :           xmode = op_mode;
    8850              : 
    8851              :           /* If X is still an AND, see if it is an AND with a mask that
    8852              :              is just some low-order bits.  If so, and it is MASK, we don't
    8853              :              need it.  */
    8854              : 
    8855      2903290 :           if (GET_CODE (x) == AND && CONST_INT_P (XEXP (x, 1))
    8856      5832503 :               && (INTVAL (XEXP (x, 1)) & GET_MODE_MASK (xmode)) == mask)
    8857        38211 :             x = XEXP (x, 0);
    8858              : 
    8859              :           /* If it remains an AND, try making another AND with the bits
    8860              :              in the mode mask that aren't in MASK turned on.  If the
    8861              :              constant in the AND is wide enough, this might make a
    8862              :              cheaper constant.  */
    8863              : 
    8864      2865169 :           if (GET_CODE (x) == AND && CONST_INT_P (XEXP (x, 1))
    8865      2865079 :               && GET_MODE_MASK (xmode) != mask
    8866      3036848 :               && HWI_COMPUTABLE_MODE_P (xmode))
    8867              :             {
    8868       107635 :               unsigned HOST_WIDE_INT cval
    8869       107635 :                 = UINTVAL (XEXP (x, 1)) | (GET_MODE_MASK (xmode) & ~mask);
    8870       107635 :               rtx y;
    8871              : 
    8872       107635 :               y = simplify_gen_binary (AND, xmode, XEXP (x, 0),
    8873       107635 :                                        gen_int_mode (cval, xmode));
    8874       107635 :               if (set_src_cost (y, xmode, optimize_this_for_speed_p)
    8875       107635 :                   < set_src_cost (x, xmode, optimize_this_for_speed_p))
    8876     73237637 :                 x = y;
    8877              :             }
    8878              : 
    8879              :           break;
    8880              :         }
    8881              : 
    8882       742937 :       goto binop;
    8883              : 
    8884     10629158 :     case PLUS:
    8885              :       /* In (and (plus FOO C1) M), if M is a mask that just turns off
    8886              :          low-order bits (as in an alignment operation) and FOO is already
    8887              :          aligned to that boundary, mask C1 to that boundary as well.
    8888              :          This may eliminate that PLUS and, later, the AND.  */
    8889              : 
    8890     10629158 :       {
    8891     10629158 :         unsigned int width = GET_MODE_PRECISION (mode);
    8892     10629158 :         unsigned HOST_WIDE_INT smask = mask;
    8893              : 
    8894              :         /* If MODE is narrower than HOST_WIDE_INT and mask is a negative
    8895              :            number, sign extend it.  */
    8896              : 
    8897     10629158 :         if (width < HOST_BITS_PER_WIDE_INT
    8898      3282476 :             && (smask & (HOST_WIDE_INT_1U << (width - 1))) != 0)
    8899      2940452 :           smask |= HOST_WIDE_INT_M1U << width;
    8900              : 
    8901     10629158 :         if (CONST_INT_P (XEXP (x, 1))
    8902      3779548 :             && pow2p_hwi (- smask)
    8903      3240205 :             && (nonzero_bits (XEXP (x, 0), mode) & ~smask) == 0
    8904     13475234 :             && (INTVAL (XEXP (x, 1)) & ~smask) != 0)
    8905        10939 :           return force_to_mode (plus_constant (xmode, XEXP (x, 0),
    8906        10939 :                                                (INTVAL (XEXP (x, 1)) & smask)),
    8907              :                                 mode, smask, next_select);
    8908              :       }
    8909              : 
    8910              :       /* fall through */
    8911              : 
    8912     12752784 :     case MULT:
    8913              :       /* Substituting into the operands of a widening MULT is not likely to
    8914              :          create RTL matching a machine insn.  */
    8915     12752784 :       if (code == MULT
    8916      2134565 :           && (GET_CODE (XEXP (x, 0)) == ZERO_EXTEND
    8917      2134565 :               || GET_CODE (XEXP (x, 0)) == SIGN_EXTEND)
    8918        87652 :           && (GET_CODE (XEXP (x, 1)) == ZERO_EXTEND
    8919        87652 :               || GET_CODE (XEXP (x, 1)) == SIGN_EXTEND)
    8920        49012 :           && REG_P (XEXP (XEXP (x, 0), 0))
    8921        42149 :           && REG_P (XEXP (XEXP (x, 1), 0)))
    8922        34765 :         return gen_lowpart_or_truncate (mode, x);
    8923              : 
    8924              :       /* For PLUS, MINUS and MULT, we need any bits less significant than the
    8925              :          most significant bit in MASK since carries from those bits will
    8926              :          affect the bits we are interested in.  */
    8927     12718019 :       mask = fuller_mask;
    8928     12718019 :       goto binop;
    8929              : 
    8930      2420654 :     case MINUS:
    8931              :       /* If X is (minus C Y) where C's least set bit is larger than any bit
    8932              :          in the mask, then we may replace with (neg Y).  */
    8933      2420654 :       if (poly_int_rtx_p (XEXP (x, 0), &const_op0)
    8934       151763 :           && known_alignment (poly_uint64 (const_op0)) > mask)
    8935              :         {
    8936           30 :           x = simplify_gen_unary (NEG, xmode, XEXP (x, 1), xmode);
    8937           30 :           return force_to_mode (x, mode, mask, next_select);
    8938              :         }
    8939              : 
    8940              :       /* Similarly, if C contains every bit in the fuller_mask, then we may
    8941              :          replace with (not Y).  */
    8942      2420624 :       if (CONST_INT_P (XEXP (x, 0))
    8943       151733 :           && ((UINTVAL (XEXP (x, 0)) | fuller_mask) == UINTVAL (XEXP (x, 0))))
    8944              :         {
    8945          328 :           x = simplify_gen_unary (NOT, xmode, XEXP (x, 1), xmode);
    8946          328 :           return force_to_mode (x, mode, mask, next_select);
    8947              :         }
    8948              : 
    8949      2420296 :       mask = fuller_mask;
    8950      2420296 :       goto binop;
    8951              : 
    8952      1926186 :     case IOR:
    8953      1926186 :     case XOR:
    8954              :       /* If X is (ior (lshiftrt FOO C1) C2), try to commute the IOR and
    8955              :          LSHIFTRT so we end up with an (and (lshiftrt (ior ...) ...) ...)
    8956              :          operation which may be a bitfield extraction.  Ensure that the
    8957              :          constant we form is not wider than the mode of X.  */
    8958              : 
    8959      1926186 :       if (GET_CODE (XEXP (x, 0)) == LSHIFTRT
    8960        63066 :           && CONST_INT_P (XEXP (XEXP (x, 0), 1))
    8961        52171 :           && INTVAL (XEXP (XEXP (x, 0), 1)) >= 0
    8962        52171 :           && INTVAL (XEXP (XEXP (x, 0), 1)) < HOST_BITS_PER_WIDE_INT
    8963        52171 :           && CONST_INT_P (XEXP (x, 1))
    8964         8542 :           && ((INTVAL (XEXP (XEXP (x, 0), 1))
    8965        17084 :                + floor_log2 (INTVAL (XEXP (x, 1))))
    8966         8542 :               < GET_MODE_PRECISION (xmode))
    8967      1926186 :           && (UINTVAL (XEXP (x, 1))
    8968         4339 :               & ~nonzero_bits (XEXP (x, 0), xmode)) == 0)
    8969              :         {
    8970         8208 :           temp = gen_int_mode ((INTVAL (XEXP (x, 1)) & mask)
    8971         4104 :                                << INTVAL (XEXP (XEXP (x, 0), 1)),
    8972              :                                xmode);
    8973         8208 :           temp = simplify_gen_binary (GET_CODE (x), xmode,
    8974         4104 :                                       XEXP (XEXP (x, 0), 0), temp);
    8975         8208 :           x = simplify_gen_binary (LSHIFTRT, xmode, temp,
    8976         4104 :                                    XEXP (XEXP (x, 0), 1));
    8977         4104 :           return force_to_mode (x, mode, mask, next_select);
    8978              :         }
    8979              : 
    8980     17803334 :     binop:
    8981              :       /* For most binary operations, just propagate into the operation and
    8982              :          change the mode if we have an operation of that mode.  */
    8983              : 
    8984     17803334 :       op0 = force_to_mode (XEXP (x, 0), mode, mask, next_select);
    8985     17803334 :       op1 = force_to_mode (XEXP (x, 1), mode, mask, next_select);
    8986              : 
    8987              :       /* If we ended up truncating both operands, truncate the result of the
    8988              :          operation instead.  */
    8989     17803334 :       if (GET_CODE (op0) == TRUNCATE
    8990            0 :           && GET_CODE (op1) == TRUNCATE)
    8991              :         {
    8992            0 :           op0 = XEXP (op0, 0);
    8993            0 :           op1 = XEXP (op1, 0);
    8994              :         }
    8995              : 
    8996     17803334 :       op0 = gen_lowpart_or_truncate (op_mode, op0);
    8997     17803334 :       op1 = gen_lowpart_or_truncate (op_mode, op1);
    8998              : 
    8999     17803334 :       if (op_mode != xmode || op0 != XEXP (x, 0) || op1 != XEXP (x, 1))
    9000              :         {
    9001      2219782 :           x = simplify_gen_binary (code, op_mode, op0, op1);
    9002      2219782 :           xmode = op_mode;
    9003              :         }
    9004              :       break;
    9005              : 
    9006      4274081 :     case ASHIFT:
    9007              :       /* For left shifts, do the same, but just for the first operand.
    9008              :          However, we cannot do anything with shifts where we cannot
    9009              :          guarantee that the counts are smaller than the size of the mode
    9010              :          because such a count will have a different meaning in a
    9011              :          wider mode.  */
    9012              : 
    9013      4067125 :       if (! (CONST_INT_P (XEXP (x, 1))
    9014      4067129 :              && INTVAL (XEXP (x, 1)) >= 0
    9015      4067125 :              && INTVAL (XEXP (x, 1)) < GET_MODE_PRECISION (mode))
    9016      4276640 :           && ! (GET_MODE (XEXP (x, 1)) != VOIDmode
    9017       206952 :                 && (nonzero_bits (XEXP (x, 1), GET_MODE (XEXP (x, 1)))
    9018       206952 :                     < (unsigned HOST_WIDE_INT) GET_MODE_PRECISION (mode))))
    9019              :         break;
    9020              : 
    9021              :       /* If the shift count is a constant and we can do arithmetic in
    9022              :          the mode of the shift, refine which bits we need.  Otherwise, use the
    9023              :          conservative form of the mask.  */
    9024      4131183 :       if (CONST_INT_P (XEXP (x, 1))
    9025      4064570 :           && INTVAL (XEXP (x, 1)) >= 0
    9026      4064570 :           && INTVAL (XEXP (x, 1)) < GET_MODE_PRECISION (op_mode)
    9027      8195753 :           && HWI_COMPUTABLE_MODE_P (op_mode))
    9028      4063820 :         mask >>= INTVAL (XEXP (x, 1));
    9029              :       else
    9030              :         mask = fuller_mask;
    9031              : 
    9032      4131183 :       op0 = gen_lowpart_or_truncate (op_mode,
    9033              :                                      force_to_mode (XEXP (x, 0), mode,
    9034              :                                                     mask, next_select));
    9035              : 
    9036      4131183 :       if (op_mode != xmode || op0 != XEXP (x, 0))
    9037              :         {
    9038       983070 :           x = simplify_gen_binary (code, op_mode, op0, XEXP (x, 1));
    9039       983070 :           xmode = op_mode;
    9040              :         }
    9041              :       break;
    9042              : 
    9043      3122722 :     case LSHIFTRT:
    9044              :       /* Here we can only do something if the shift count is a constant,
    9045              :          this shift constant is valid for the host, and we can do arithmetic
    9046              :          in OP_MODE.  */
    9047              : 
    9048      3122722 :       if (CONST_INT_P (XEXP (x, 1))
    9049      3010982 :           && INTVAL (XEXP (x, 1)) >= 0
    9050      3010981 :           && INTVAL (XEXP (x, 1)) < HOST_BITS_PER_WIDE_INT
    9051      6133605 :           && HWI_COMPUTABLE_MODE_P (op_mode))
    9052              :         {
    9053      3007492 :           rtx inner = XEXP (x, 0);
    9054      3007492 :           unsigned HOST_WIDE_INT inner_mask;
    9055              : 
    9056              :           /* Select the mask of the bits we need for the shift operand.  */
    9057      3007492 :           inner_mask = mask << INTVAL (XEXP (x, 1));
    9058              : 
    9059              :           /* We can only change the mode of the shift if we can do arithmetic
    9060              :              in the mode of the shift and INNER_MASK is no wider than the
    9061              :              width of X's mode.  */
    9062      3007492 :           if ((inner_mask & ~GET_MODE_MASK (xmode)) != 0)
    9063       262236 :             op_mode = xmode;
    9064              : 
    9065      3007492 :           inner = force_to_mode (inner, op_mode, inner_mask, next_select);
    9066              : 
    9067      3007492 :           if (xmode != op_mode || inner != XEXP (x, 0))
    9068              :             {
    9069       805680 :               x = simplify_gen_binary (LSHIFTRT, op_mode, inner, XEXP (x, 1));
    9070       805680 :               xmode = op_mode;
    9071              :             }
    9072              :         }
    9073              : 
    9074              :       /* If we have (and (lshiftrt FOO C1) C2) where the combination of the
    9075              :          shift and AND produces only copies of the sign bit (C2 is one less
    9076              :          than a power of two), we can do this with just a shift.  */
    9077              : 
    9078      3122722 :       if (GET_CODE (x) == LSHIFTRT
    9079      3122665 :           && CONST_INT_P (XEXP (x, 1))
    9080              :           /* The shift puts one of the sign bit copies in the least significant
    9081              :              bit.  */
    9082      6021850 :           && ((INTVAL (XEXP (x, 1))
    9083      3010925 :                + num_sign_bit_copies (XEXP (x, 0), GET_MODE (XEXP (x, 0))))
    9084      3010925 :               >= GET_MODE_PRECISION (xmode))
    9085       242074 :           && pow2p_hwi (mask + 1)
    9086              :           /* Number of bits left after the shift must be more than the mask
    9087              :              needs.  */
    9088        79409 :           && ((INTVAL (XEXP (x, 1)) + exact_log2 (mask + 1))
    9089        79409 :               <= GET_MODE_PRECISION (xmode))
    9090              :           /* Must be more sign bit copies than the mask needs.  */
    9091      3151951 :           && ((int) num_sign_bit_copies (XEXP (x, 0), GET_MODE (XEXP (x, 0)))
    9092        29229 :               >= exact_log2 (mask + 1)))
    9093              :         {
    9094        29229 :           int nbits = GET_MODE_PRECISION (xmode) - exact_log2 (mask + 1);
    9095        29229 :           x = simplify_gen_binary (LSHIFTRT, xmode, XEXP (x, 0),
    9096        29229 :                                    gen_int_shift_amount (xmode, nbits));
    9097              :         }
    9098      3122722 :       goto shiftrt;
    9099              : 
    9100      2063706 :     case ASHIFTRT:
    9101              :       /* If we are just looking for the sign bit, we don't need this shift at
    9102              :          all, even if it has a variable count.  */
    9103      2063706 :       if (val_signbit_p (xmode, mask))
    9104         1179 :         return force_to_mode (XEXP (x, 0), mode, mask, next_select);
    9105              : 
    9106              :       /* If this is a shift by a constant, get a mask that contains those bits
    9107              :          that are not copies of the sign bit.  We then have two cases:  If
    9108              :          MASK only includes those bits, this can be a logical shift, which may
    9109              :          allow simplifications.  If MASK is a single-bit field not within
    9110              :          those bits, we are requesting a copy of the sign bit and hence can
    9111              :          shift the sign bit to the appropriate location.  */
    9112              : 
    9113      2062527 :       if (CONST_INT_P (XEXP (x, 1)) && INTVAL (XEXP (x, 1)) >= 0
    9114      2016621 :           && INTVAL (XEXP (x, 1)) < HOST_BITS_PER_WIDE_INT)
    9115              :         {
    9116      2016510 :           unsigned HOST_WIDE_INT nonzero;
    9117      2016510 :           int i;
    9118              : 
    9119              :           /* If the considered data is wider than HOST_WIDE_INT, we can't
    9120              :              represent a mask for all its bits in a single scalar.
    9121              :              But we only care about the lower bits, so calculate these.  */
    9122              : 
    9123      2016510 :           if (GET_MODE_PRECISION (xmode) > HOST_BITS_PER_WIDE_INT)
    9124              :             {
    9125          412 :               nonzero = HOST_WIDE_INT_M1U;
    9126              : 
    9127              :               /* GET_MODE_PRECISION (GET_MODE (x)) - INTVAL (XEXP (x, 1))
    9128              :                  is the number of bits a full-width mask would have set.
    9129              :                  We need only shift if these are fewer than nonzero can
    9130              :                  hold.  If not, we must keep all bits set in nonzero.  */
    9131              : 
    9132          412 :               if (GET_MODE_PRECISION (xmode) - INTVAL (XEXP (x, 1))
    9133              :                   < HOST_BITS_PER_WIDE_INT)
    9134            0 :                 nonzero >>= INTVAL (XEXP (x, 1))
    9135            0 :                             + HOST_BITS_PER_WIDE_INT
    9136            0 :                             - GET_MODE_PRECISION (xmode);
    9137              :             }
    9138              :           else
    9139              :             {
    9140      2016098 :               nonzero = GET_MODE_MASK (xmode);
    9141      2016098 :               nonzero >>= INTVAL (XEXP (x, 1));
    9142              :             }
    9143              : 
    9144      2016510 :           if ((mask & ~nonzero) == 0)
    9145              :             {
    9146        50025 :               x = simplify_shift_const (NULL_RTX, LSHIFTRT, xmode,
    9147              :                                         XEXP (x, 0), INTVAL (XEXP (x, 1)));
    9148        50025 :               if (GET_CODE (x) != ASHIFTRT)
    9149        50025 :                 return force_to_mode (x, mode, mask, next_select);
    9150              :             }
    9151              : 
    9152      1966485 :           else if ((i = exact_log2 (mask)) >= 0)
    9153              :             {
    9154          190 :               x = simplify_shift_const
    9155          380 :                   (NULL_RTX, LSHIFTRT, xmode, XEXP (x, 0),
    9156          190 :                    GET_MODE_PRECISION (xmode) - 1 - i);
    9157              : 
    9158          190 :               if (GET_CODE (x) != ASHIFTRT)
    9159          190 :                 return force_to_mode (x, mode, mask, next_select);
    9160              :             }
    9161              :         }
    9162              : 
    9163              :       /* If MASK is 1, convert this to an LSHIFTRT.  This can be done
    9164              :          even if the shift count isn't a constant.  */
    9165      2012312 :       if (mask == 1)
    9166         3117 :         x = simplify_gen_binary (LSHIFTRT, xmode, XEXP (x, 0), XEXP (x, 1));
    9167              : 
    9168      2009195 :     shiftrt:
    9169              : 
    9170              :       /* If this is a zero- or sign-extension operation that just affects bits
    9171              :          we don't care about, remove it.  Be sure the call above returned
    9172              :          something that is still a shift.  */
    9173              : 
    9174      5135034 :       if ((GET_CODE (x) == LSHIFTRT || GET_CODE (x) == ASHIFTRT)
    9175      5134977 :           && CONST_INT_P (XEXP (x, 1))
    9176      4977331 :           && INTVAL (XEXP (x, 1)) >= 0
    9177      4977330 :           && (INTVAL (XEXP (x, 1))
    9178      9954660 :               <= GET_MODE_PRECISION (xmode) - (floor_log2 (mask) + 1))
    9179      1792874 :           && GET_CODE (XEXP (x, 0)) == ASHIFT
    9180      5136139 :           && XEXP (XEXP (x, 0), 1) == XEXP (x, 1))
    9181          883 :         return force_to_mode (XEXP (XEXP (x, 0), 0), mode, mask, next_select);
    9182              : 
    9183              :       break;
    9184              : 
    9185        36776 :     case ROTATE:
    9186        36776 :     case ROTATERT:
    9187              :       /* If the shift count is constant and we can do computations
    9188              :          in the mode of X, compute where the bits we care about are.
    9189              :          Otherwise, we can't do anything.  Don't change the mode of
    9190              :          the shift or propagate MODE into the shift, though.  */
    9191        36776 :       if (CONST_INT_P (XEXP (x, 1))
    9192        26500 :           && INTVAL (XEXP (x, 1)) >= 0)
    9193              :         {
    9194        26498 :           temp = simplify_binary_operation (code == ROTATE ? ROTATERT : ROTATE,
    9195        26498 :                                             xmode, gen_int_mode (mask, xmode),
    9196              :                                             XEXP (x, 1));
    9197        26498 :           if (temp && CONST_INT_P (temp))
    9198        26498 :             x = simplify_gen_binary (code, xmode,
    9199              :                                      force_to_mode (XEXP (x, 0), xmode,
    9200        26498 :                                                     INTVAL (temp), next_select),
    9201              :                                      XEXP (x, 1));
    9202              :         }
    9203              :       break;
    9204              : 
    9205       160204 :     case NEG:
    9206              :       /* If we just want the low-order bit, the NEG isn't needed since it
    9207              :          won't change the low-order bit.  */
    9208       160204 :       if (mask == 1)
    9209          370 :         return force_to_mode (XEXP (x, 0), mode, mask, just_select);
    9210              : 
    9211              :       /* We need any bits less significant than the most significant bit in
    9212              :          MASK since carries from those bits will affect the bits we are
    9213              :          interested in.  */
    9214       159834 :       mask = fuller_mask;
    9215       159834 :       goto unop;
    9216              : 
    9217       474694 :     case NOT:
    9218              :       /* (not FOO) is (xor FOO CONST), so if FOO is an LSHIFTRT, we can do the
    9219              :          same as the XOR case above.  Ensure that the constant we form is not
    9220              :          wider than the mode of X.  */
    9221              : 
    9222       474694 :       if (GET_CODE (XEXP (x, 0)) == LSHIFTRT
    9223        21169 :           && CONST_INT_P (XEXP (XEXP (x, 0), 1))
    9224        20596 :           && INTVAL (XEXP (XEXP (x, 0), 1)) >= 0
    9225        41192 :           && (INTVAL (XEXP (XEXP (x, 0), 1)) + floor_log2 (mask)
    9226        20596 :               < GET_MODE_PRECISION (xmode))
    9227       481562 :           && INTVAL (XEXP (XEXP (x, 0), 1)) < HOST_BITS_PER_WIDE_INT)
    9228              :         {
    9229         6868 :           temp = gen_int_mode (mask << INTVAL (XEXP (XEXP (x, 0), 1)), xmode);
    9230         6868 :           temp = simplify_gen_binary (XOR, xmode, XEXP (XEXP (x, 0), 0), temp);
    9231        13736 :           x = simplify_gen_binary (LSHIFTRT, xmode,
    9232         6868 :                                    temp, XEXP (XEXP (x, 0), 1));
    9233              : 
    9234         6868 :           return force_to_mode (x, mode, mask, next_select);
    9235              :         }
    9236              : 
    9237              :       /* (and (not FOO) CONST) is (not (or FOO (not CONST))), so we must
    9238              :          use the full mask inside the NOT.  */
    9239              :       mask = fuller_mask;
    9240              : 
    9241       627660 :     unop:
    9242       627660 :       op0 = gen_lowpart_or_truncate (op_mode,
    9243              :                                      force_to_mode (XEXP (x, 0), mode, mask,
    9244              :                                                     next_select));
    9245       627660 :       if (op_mode != xmode || op0 != XEXP (x, 0))
    9246              :         {
    9247        77267 :           x = simplify_gen_unary (code, op_mode, op0, op_mode);
    9248        77267 :           xmode = op_mode;
    9249              :         }
    9250              :       break;
    9251              : 
    9252       588675 :     case NE:
    9253              :       /* (and (ne FOO 0) CONST) can be (and FOO CONST) if CONST is included
    9254              :          in STORE_FLAG_VALUE and FOO has a single bit that might be nonzero,
    9255              :          which is equal to STORE_FLAG_VALUE.  */
    9256       588675 :       if ((mask & ~STORE_FLAG_VALUE) == 0
    9257         2462 :           && XEXP (x, 1) == const0_rtx
    9258         2441 :           && GET_MODE (XEXP (x, 0)) == mode
    9259            9 :           && pow2p_hwi (nonzero_bits (XEXP (x, 0), mode))
    9260       588675 :           && (nonzero_bits (XEXP (x, 0), mode)
    9261              :               == (unsigned HOST_WIDE_INT) STORE_FLAG_VALUE))
    9262            0 :         return force_to_mode (XEXP (x, 0), mode, mask, next_select);
    9263              : 
    9264              :       break;
    9265              : 
    9266      1643154 :     case IF_THEN_ELSE:
    9267              :       /* We have no way of knowing if the IF_THEN_ELSE can itself be
    9268              :          written in a narrower mode.  We play it safe and do not do so.  */
    9269              : 
    9270      1643154 :       op0 = gen_lowpart_or_truncate (xmode,
    9271              :                                      force_to_mode (XEXP (x, 1), mode,
    9272              :                                                     mask, next_select));
    9273      1643154 :       op1 = gen_lowpart_or_truncate (xmode,
    9274              :                                      force_to_mode (XEXP (x, 2), mode,
    9275              :                                                     mask, next_select));
    9276      1643154 :       if (op0 != XEXP (x, 1) || op1 != XEXP (x, 2))
    9277       266506 :         x = simplify_gen_ternary (IF_THEN_ELSE, xmode,
    9278       266506 :                                   GET_MODE (XEXP (x, 0)), XEXP (x, 0),
    9279              :                                   op0, op1);
    9280              :       break;
    9281              : 
    9282              :     default:
    9283              :       break;
    9284              :     }
    9285              : 
    9286              :   /* Ensure we return a value of the proper mode.  */
    9287     73237637 :   return gen_lowpart_or_truncate (mode, x);
    9288              : }
    9289              : 
    9290              : /* Return nonzero if X is an expression that has one of two values depending on
    9291              :    whether some other value is zero or nonzero.  In that case, we return the
    9292              :    value that is being tested, *PTRUE is set to the value if the rtx being
    9293              :    returned has a nonzero value, and *PFALSE is set to the other alternative.
    9294              : 
    9295              :    If we return zero, we set *PTRUE and *PFALSE to X.  */
    9296              : 
    9297              : static rtx
    9298    246990709 : if_then_else_cond (rtx x, rtx *ptrue, rtx *pfalse)
    9299              : {
    9300    246990709 :   machine_mode mode = GET_MODE (x);
    9301    246990709 :   enum rtx_code code = GET_CODE (x);
    9302    246990709 :   rtx cond0, cond1, true0, true1, false0, false1;
    9303    246990709 :   unsigned HOST_WIDE_INT nz;
    9304    246990709 :   scalar_int_mode int_mode;
    9305              : 
    9306              :   /* If we are comparing a value against zero, we are done.  */
    9307    246990709 :   if ((code == NE || code == EQ)
    9308      2697118 :       && XEXP (x, 1) == const0_rtx)
    9309              :     {
    9310      1651725 :       *ptrue = (code == NE) ? const_true_rtx : const0_rtx;
    9311      1651725 :       *pfalse = (code == NE) ? const0_rtx : const_true_rtx;
    9312      1651725 :       return XEXP (x, 0);
    9313              :     }
    9314              : 
    9315              :   /* If this is a unary operation whose operand has one of two values, apply
    9316              :      our opcode to compute those values.  */
    9317    245338984 :   else if (UNARY_P (x)
    9318    245338984 :            && (cond0 = if_then_else_cond (XEXP (x, 0), &true0, &false0)) != 0)
    9319              :     {
    9320       514333 :       *ptrue = simplify_gen_unary (code, mode, true0, GET_MODE (XEXP (x, 0)));
    9321      1028666 :       *pfalse = simplify_gen_unary (code, mode, false0,
    9322       514333 :                                     GET_MODE (XEXP (x, 0)));
    9323       514333 :       return cond0;
    9324              :     }
    9325              : 
    9326              :   /* If this is a COMPARE, do nothing, since the IF_THEN_ELSE we would
    9327              :      make can't possibly match and would suppress other optimizations.  */
    9328    244824651 :   else if (code == COMPARE)
    9329              :     ;
    9330              : 
    9331              :   /* If this is a binary operation, see if either side has only one of two
    9332              :      values.  If either one does or if both do and they are conditional on
    9333              :      the same value, compute the new true and false values.  */
    9334    240432726 :   else if (BINARY_P (x))
    9335              :     {
    9336     88895149 :       rtx op0 = XEXP (x, 0);
    9337     88895149 :       rtx op1 = XEXP (x, 1);
    9338     88895149 :       cond0 = if_then_else_cond (op0, &true0, &false0);
    9339     88895149 :       cond1 = if_then_else_cond (op1, &true1, &false1);
    9340              : 
    9341       484756 :       if ((cond0 != 0 && cond1 != 0 && !rtx_equal_p (cond0, cond1))
    9342     89324320 :           && (REG_P (op0) || REG_P (op1)))
    9343              :         {
    9344              :           /* Try to enable a simplification by undoing work done by
    9345              :              if_then_else_cond if it converted a REG into something more
    9346              :              complex.  */
    9347       359335 :           if (REG_P (op0))
    9348              :             {
    9349        73435 :               cond0 = 0;
    9350        73435 :               true0 = false0 = op0;
    9351              :             }
    9352              :           else
    9353              :             {
    9354       285900 :               cond1 = 0;
    9355       285900 :               true1 = false1 = op1;
    9356              :             }
    9357              :         }
    9358              : 
    9359     88895149 :       if ((cond0 != 0 || cond1 != 0)
    9360     88895149 :           && ! (cond0 != 0 && cond1 != 0 && !rtx_equal_p (cond0, cond1)))
    9361              :         {
    9362              :           /* If if_then_else_cond returned zero, then true/false are the
    9363              :              same rtl.  We must copy one of them to prevent invalid rtl
    9364              :              sharing.  */
    9365      5012652 :           if (cond0 == 0)
    9366      1394641 :             true0 = copy_rtx (true0);
    9367      3618011 :           else if (cond1 == 0)
    9368      3562426 :             true1 = copy_rtx (true1);
    9369              : 
    9370      5012652 :           if (COMPARISON_P (x))
    9371              :             {
    9372       271646 :               *ptrue = simplify_gen_relational (code, mode, VOIDmode,
    9373              :                                                 true0, true1);
    9374       271646 :               *pfalse = simplify_gen_relational (code, mode, VOIDmode,
    9375              :                                                  false0, false1);
    9376              :              }
    9377              :           else
    9378              :             {
    9379      4741006 :               *ptrue = simplify_gen_binary (code, mode, true0, true1);
    9380      4741006 :               *pfalse = simplify_gen_binary (code, mode, false0, false1);
    9381              :             }
    9382              : 
    9383      5012652 :           return cond0 ? cond0 : cond1;
    9384              :         }
    9385              : 
    9386              :       /* See if we have PLUS, IOR, XOR, MINUS or UMAX, where one of the
    9387              :          operands is zero when the other is nonzero, and vice-versa,
    9388              :          and STORE_FLAG_VALUE is 1 or -1.  */
    9389              : 
    9390     83882497 :       if ((STORE_FLAG_VALUE == 1 || STORE_FLAG_VALUE == -1)
    9391     83882497 :           && (code == PLUS || code == IOR || code == XOR || code == MINUS
    9392              :               || code == UMAX)
    9393     34981678 :           && GET_CODE (XEXP (x, 0)) == MULT && GET_CODE (XEXP (x, 1)) == MULT)
    9394              :         {
    9395        39646 :           rtx op0 = XEXP (XEXP (x, 0), 1);
    9396        39646 :           rtx op1 = XEXP (XEXP (x, 1), 1);
    9397              : 
    9398        39646 :           cond0 = XEXP (XEXP (x, 0), 0);
    9399        39646 :           cond1 = XEXP (XEXP (x, 1), 0);
    9400              : 
    9401        39646 :           if (COMPARISON_P (cond0)
    9402           11 :               && COMPARISON_P (cond1)
    9403            0 :               && SCALAR_INT_MODE_P (mode)
    9404            0 :               && ((GET_CODE (cond0) == reversed_comparison_code (cond1, NULL)
    9405            0 :                    && rtx_equal_p (XEXP (cond0, 0), XEXP (cond1, 0))
    9406            0 :                    && rtx_equal_p (XEXP (cond0, 1), XEXP (cond1, 1)))
    9407            0 :                   || ((swap_condition (GET_CODE (cond0))
    9408            0 :                        == reversed_comparison_code (cond1, NULL))
    9409            0 :                       && rtx_equal_p (XEXP (cond0, 0), XEXP (cond1, 1))
    9410            0 :                       && rtx_equal_p (XEXP (cond0, 1), XEXP (cond1, 0))))
    9411        39646 :               && ! side_effects_p (x))
    9412              :             {
    9413            0 :               *ptrue = simplify_gen_binary (MULT, mode, op0, const_true_rtx);
    9414            0 :               *pfalse = simplify_gen_binary (MULT, mode,
    9415              :                                              (code == MINUS
    9416            0 :                                               ? simplify_gen_unary (NEG, mode,
    9417              :                                                                     op1, mode)
    9418              :                                               : op1),
    9419              :                                               const_true_rtx);
    9420            0 :               return cond0;
    9421              :             }
    9422              :         }
    9423              : 
    9424              :       /* Similarly for MULT, AND and UMIN, except that for these the result
    9425              :          is always zero.  */
    9426     83882497 :       if ((STORE_FLAG_VALUE == 1 || STORE_FLAG_VALUE == -1)
    9427     83882497 :           && (code == MULT || code == AND || code == UMIN)
    9428     12303076 :           && GET_CODE (XEXP (x, 0)) == MULT && GET_CODE (XEXP (x, 1)) == MULT)
    9429              :         {
    9430          965 :           cond0 = XEXP (XEXP (x, 0), 0);
    9431          965 :           cond1 = XEXP (XEXP (x, 1), 0);
    9432              : 
    9433          965 :           if (COMPARISON_P (cond0)
    9434            0 :               && COMPARISON_P (cond1)
    9435            0 :               && ((GET_CODE (cond0) == reversed_comparison_code (cond1, NULL)
    9436            0 :                    && rtx_equal_p (XEXP (cond0, 0), XEXP (cond1, 0))
    9437            0 :                    && rtx_equal_p (XEXP (cond0, 1), XEXP (cond1, 1)))
    9438            0 :                   || ((swap_condition (GET_CODE (cond0))
    9439            0 :                        == reversed_comparison_code (cond1, NULL))
    9440            0 :                       && rtx_equal_p (XEXP (cond0, 0), XEXP (cond1, 1))
    9441            0 :                       && rtx_equal_p (XEXP (cond0, 1), XEXP (cond1, 0))))
    9442          965 :               && ! side_effects_p (x))
    9443              :             {
    9444            0 :               *ptrue = *pfalse = const0_rtx;
    9445            0 :               return cond0;
    9446              :             }
    9447              :         }
    9448              :     }
    9449              : 
    9450    151537577 :   else if (code == IF_THEN_ELSE)
    9451              :     {
    9452              :       /* If we have IF_THEN_ELSE already, extract the condition and
    9453              :          canonicalize it if it is NE or EQ.  */
    9454       753801 :       cond0 = XEXP (x, 0);
    9455       753801 :       *ptrue = XEXP (x, 1), *pfalse = XEXP (x, 2);
    9456       753801 :       if (GET_CODE (cond0) == NE && XEXP (cond0, 1) == const0_rtx)
    9457       263447 :         return XEXP (cond0, 0);
    9458       490354 :       else if (GET_CODE (cond0) == EQ && XEXP (cond0, 1) == const0_rtx)
    9459              :         {
    9460        34365 :           *ptrue = XEXP (x, 2), *pfalse = XEXP (x, 1);
    9461        34365 :           return XEXP (cond0, 0);
    9462              :         }
    9463              :       else
    9464              :         return cond0;
    9465              :     }
    9466              : 
    9467              :   /* If X is a SUBREG, we can narrow both the true and false values
    9468              :      if the inner expression, if there is a condition.  */
    9469    150783776 :   else if (code == SUBREG
    9470    150783776 :            && (cond0 = if_then_else_cond (SUBREG_REG (x), &true0,
    9471              :                                           &false0)) != 0)
    9472              :     {
    9473      1277208 :       true0 = simplify_gen_subreg (mode, true0,
    9474       638604 :                                    GET_MODE (SUBREG_REG (x)), SUBREG_BYTE (x));
    9475      1277208 :       false0 = simplify_gen_subreg (mode, false0,
    9476       638604 :                                     GET_MODE (SUBREG_REG (x)), SUBREG_BYTE (x));
    9477       638604 :       if (true0 && false0)
    9478              :         {
    9479       638604 :           *ptrue = true0;
    9480       638604 :           *pfalse = false0;
    9481       638604 :           return cond0;
    9482              :         }
    9483              :     }
    9484              : 
    9485              :   /* If X is a constant, this isn't special and will cause confusions
    9486              :      if we treat it as such.  Likewise if it is equivalent to a constant.  */
    9487    150145172 :   else if (CONSTANT_P (x)
    9488    150145172 :            || ((cond0 = get_last_value (x)) != 0 && CONSTANT_P (cond0)))
    9489              :     ;
    9490              : 
    9491              :   /* If we're in BImode, canonicalize on 0 and STORE_FLAG_VALUE, as that
    9492              :      will be least confusing to the rest of the compiler.  */
    9493    101657658 :   else if (mode == BImode)
    9494              :     {
    9495            0 :       *ptrue = GEN_INT (STORE_FLAG_VALUE), *pfalse = const0_rtx;
    9496            0 :       return x;
    9497              :     }
    9498              : 
    9499              :   /* If X is known to be either 0 or -1, those are the true and
    9500              :      false values when testing X.  */
    9501    101657658 :   else if (x == constm1_rtx || x == const0_rtx
    9502    101657658 :            || (is_a <scalar_int_mode> (mode, &int_mode)
    9503     72714836 :                && (num_sign_bit_copies (x, int_mode)
    9504     72714836 :                    == GET_MODE_PRECISION (int_mode))))
    9505              :     {
    9506      1028831 :       *ptrue = constm1_rtx, *pfalse = const0_rtx;
    9507      1028831 :       return x;
    9508              :     }
    9509              : 
    9510              :   /* Likewise for 0 or a single bit.  */
    9511    100628827 :   else if (HWI_COMPUTABLE_MODE_P (mode)
    9512     68194231 :            && pow2p_hwi (nz = nonzero_bits (x, mode)))
    9513              :     {
    9514      1778685 :       *ptrue = gen_int_mode (nz, mode), *pfalse = const0_rtx;
    9515      1778685 :       return x;
    9516              :     }
    9517              : 
    9518              :   /* Otherwise fail; show no condition with true and false values the same.  */
    9519    235612078 :   *ptrue = *pfalse = x;
    9520    235612078 :   return 0;
    9521              : }
    9522              : 
    9523              : /* Return the value of expression X given the fact that condition COND
    9524              :    is known to be true when applied to REG as its first operand and VAL
    9525              :    as its second.  X is known to not be shared and so can be modified in
    9526              :    place.
    9527              : 
    9528              :    We only handle the simplest cases, and specifically those cases that
    9529              :    arise with IF_THEN_ELSE expressions.  */
    9530              : 
    9531              : static rtx
    9532       678603 : known_cond (rtx x, enum rtx_code cond, rtx reg, rtx val)
    9533              : {
    9534       678603 :   enum rtx_code code = GET_CODE (x);
    9535       678603 :   const char *fmt;
    9536       678603 :   int i, j;
    9537              : 
    9538       678603 :   if (side_effects_p (x))
    9539              :     return x;
    9540              : 
    9541              :   /* If either operand of the condition is a floating point value,
    9542              :      then we have to avoid collapsing an EQ comparison.  */
    9543       678603 :   if (cond == EQ
    9544       125771 :       && rtx_equal_p (x, reg)
    9545        83306 :       && ! FLOAT_MODE_P (GET_MODE (x))
    9546       761909 :       && ! FLOAT_MODE_P (GET_MODE (val)))
    9547              :     return val;
    9548              : 
    9549       595297 :   if (cond == UNEQ && rtx_equal_p (x, reg))
    9550              :     return val;
    9551              : 
    9552              :   /* If X is (abs REG) and we know something about REG's relationship
    9553              :      with zero, we may be able to simplify this.  */
    9554              : 
    9555       595297 :   if (code == ABS && rtx_equal_p (XEXP (x, 0), reg) && val == const0_rtx)
    9556            3 :     switch (cond)
    9557              :       {
    9558            1 :       case GE:  case GT:  case EQ:
    9559            1 :         return XEXP (x, 0);
    9560            2 :       case LT:  case LE:
    9561            4 :         return simplify_gen_unary (NEG, GET_MODE (XEXP (x, 0)),
    9562              :                                    XEXP (x, 0),
    9563            2 :                                    GET_MODE (XEXP (x, 0)));
    9564              :       default:
    9565              :         break;
    9566              :       }
    9567              : 
    9568              :   /* The only other cases we handle are MIN, MAX, and comparisons if the
    9569              :      operands are the same as REG and VAL.  */
    9570              : 
    9571       595294 :   else if (COMPARISON_P (x) || COMMUTATIVE_ARITH_P (x))
    9572              :     {
    9573       243864 :       if (rtx_equal_p (XEXP (x, 0), val))
    9574              :         {
    9575            2 :           std::swap (val, reg);
    9576            2 :           cond = swap_condition (cond);
    9577              :         }
    9578              : 
    9579       243864 :       if (rtx_equal_p (XEXP (x, 0), reg) && rtx_equal_p (XEXP (x, 1), val))
    9580              :         {
    9581       221832 :           if (COMPARISON_P (x))
    9582              :             {
    9583       221578 :               if (comparison_dominates_p (cond, code))
    9584         1308 :                 return VECTOR_MODE_P (GET_MODE (x)) ? x : const_true_rtx;
    9585              : 
    9586       220270 :               code = reversed_comparison_code (x, NULL);
    9587       220270 :               if (code != UNKNOWN
    9588       220270 :                   && comparison_dominates_p (cond, code))
    9589           71 :                 return CONST0_RTX (GET_MODE (x));
    9590              :               else
    9591              :                 return x;
    9592              :             }
    9593          254 :           else if (code == SMAX || code == SMIN
    9594          254 :                    || code == UMIN || code == UMAX)
    9595              :             {
    9596           46 :               int unsignedp = (code == UMIN || code == UMAX);
    9597              : 
    9598              :               /* Do not reverse the condition when it is NE or EQ.
    9599              :                  This is because we cannot conclude anything about
    9600              :                  the value of 'SMAX (x, y)' when x is not equal to y,
    9601              :                  but we can when x equals y.  */
    9602           46 :               if ((code == SMAX || code == UMAX)
    9603           43 :                   && ! (cond == EQ || cond == NE))
    9604           10 :                 cond = reverse_condition (cond);
    9605              : 
    9606           13 :               switch (cond)
    9607              :                 {
    9608            3 :                 case GE:   case GT:
    9609            3 :                   return unsignedp ? x : XEXP (x, 1);
    9610           10 :                 case LE:   case LT:
    9611           10 :                   return unsignedp ? x : XEXP (x, 0);
    9612            0 :                 case GEU:  case GTU:
    9613            0 :                   return unsignedp ? XEXP (x, 1) : x;
    9614            0 :                 case LEU:  case LTU:
    9615            0 :                   return unsignedp ? XEXP (x, 0) : x;
    9616              :                 default:
    9617              :                   break;
    9618              :                 }
    9619              :             }
    9620              :         }
    9621              :     }
    9622       351430 :   else if (code == SUBREG)
    9623              :     {
    9624         9282 :       machine_mode inner_mode = GET_MODE (SUBREG_REG (x));
    9625         9282 :       rtx new_rtx, r = known_cond (SUBREG_REG (x), cond, reg, val);
    9626              : 
    9627         9282 :       if (SUBREG_REG (x) != r)
    9628              :         {
    9629              :           /* We must simplify subreg here, before we lose track of the
    9630              :              original inner_mode.  */
    9631          334 :           new_rtx = simplify_subreg (GET_MODE (x), r,
    9632          167 :                                      inner_mode, SUBREG_BYTE (x));
    9633          167 :           if (new_rtx)
    9634              :             return new_rtx;
    9635              :           else
    9636          167 :             SUBST (SUBREG_REG (x), r);
    9637              :         }
    9638              : 
    9639              :       return x;
    9640              :     }
    9641              :   /* We don't have to handle SIGN_EXTEND here, because even in the
    9642              :      case of replacing something with a modeless CONST_INT, a
    9643              :      CONST_INT is already (supposed to be) a valid sign extension for
    9644              :      its narrower mode, which implies it's already properly
    9645              :      sign-extended for the wider mode.  Now, for ZERO_EXTEND, the
    9646              :      story is different.  */
    9647       342148 :   else if (code == ZERO_EXTEND)
    9648              :     {
    9649         1324 :       machine_mode inner_mode = GET_MODE (XEXP (x, 0));
    9650         1324 :       rtx new_rtx, r = known_cond (XEXP (x, 0), cond, reg, val);
    9651              : 
    9652         1324 :       if (XEXP (x, 0) != r)
    9653              :         {
    9654              :           /* We must simplify the zero_extend here, before we lose
    9655              :              track of the original inner_mode.  */
    9656            0 :           new_rtx = simplify_unary_operation (ZERO_EXTEND, GET_MODE (x),
    9657              :                                               r, inner_mode);
    9658            0 :           if (new_rtx)
    9659              :             return new_rtx;
    9660              :           else
    9661            0 :             SUBST (XEXP (x, 0), r);
    9662              :         }
    9663              : 
    9664              :       return x;
    9665              :     }
    9666              : 
    9667       363097 :   fmt = GET_RTX_FORMAT (code);
    9668       823168 :   for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
    9669              :     {
    9670       460071 :       if (fmt[i] == 'e')
    9671       199381 :         SUBST (XEXP (x, i), known_cond (XEXP (x, i), cond, reg, val));
    9672       260690 :       else if (fmt[i] == 'E')
    9673        15144 :         for (j = XVECLEN (x, i) - 1; j >= 0; j--)
    9674        12210 :           SUBST (XVECEXP (x, i, j), known_cond (XVECEXP (x, i, j),
    9675              :                                                 cond, reg, val));
    9676              :     }
    9677              : 
    9678              :   return x;
    9679              : }
    9680              : 
    9681              : /* See if X and Y are equal for the purposes of seeing if we can rewrite an
    9682              :    assignment as a field assignment.  */
    9683              : 
    9684              : static bool
    9685       983427 : rtx_equal_for_field_assignment_p (rtx x, rtx y, bool widen_x)
    9686              : {
    9687       983427 :   if (widen_x && GET_MODE (x) != GET_MODE (y))
    9688              :     {
    9689        74450 :       if (paradoxical_subreg_p (GET_MODE (x), GET_MODE (y)))
    9690              :         return false;
    9691        74450 :       if (BYTES_BIG_ENDIAN != WORDS_BIG_ENDIAN)
    9692              :         return false;
    9693        74450 :       x = adjust_address_nv (x, GET_MODE (y),
    9694              :                              byte_lowpart_offset (GET_MODE (y),
    9695              :                                                   GET_MODE (x)));
    9696              :     }
    9697              : 
    9698       983427 :   if (x == y || rtx_equal_p (x, y))
    9699              :     return true;
    9700              : 
    9701       960476 :   if (x == 0 || y == 0 || GET_MODE (x) != GET_MODE (y))
    9702              :     return false;
    9703              : 
    9704              :   /* Check for a paradoxical SUBREG of a MEM compared with the MEM.
    9705              :      Note that all SUBREGs of MEM are paradoxical; otherwise they
    9706              :      would have been rewritten.  */
    9707       115135 :   if (MEM_P (x) && GET_CODE (y) == SUBREG
    9708         6346 :       && MEM_P (SUBREG_REG (y))
    9709       960222 :       && rtx_equal_p (SUBREG_REG (y),
    9710            0 :                       gen_lowpart (GET_MODE (SUBREG_REG (y)), x)))
    9711              :     return true;
    9712              : 
    9713        71616 :   if (MEM_P (y) && GET_CODE (x) == SUBREG
    9714        10363 :       && MEM_P (SUBREG_REG (x))
    9715       961025 :       && rtx_equal_p (SUBREG_REG (x),
    9716          803 :                       gen_lowpart (GET_MODE (SUBREG_REG (x)), y)))
    9717              :     return true;
    9718              : 
    9719              :   /* We used to see if get_last_value of X and Y were the same but that's
    9720              :      not correct.  In one direction, we'll cause the assignment to have
    9721              :      the wrong destination and in the case, we'll import a register into this
    9722              :      insn that might have already have been dead.   So fail if none of the
    9723              :      above cases are true.  */
    9724              :   return false;
    9725              : }
    9726              : 
    9727              : /* See if X, a SET operation, can be rewritten as a bit-field assignment.
    9728              :    Return that assignment if so.
    9729              : 
    9730              :    We only handle the most common cases.  */
    9731              : 
    9732              : static rtx
    9733     48105967 : make_field_assignment (rtx x)
    9734              : {
    9735     48105967 :   rtx dest = SET_DEST (x);
    9736     48105967 :   rtx src = SET_SRC (x);
    9737     48105967 :   rtx assign;
    9738     48105967 :   rtx rhs, lhs;
    9739     48105967 :   HOST_WIDE_INT c1;
    9740     48105967 :   HOST_WIDE_INT pos;
    9741     48105967 :   unsigned HOST_WIDE_INT len;
    9742     48105967 :   rtx other;
    9743              : 
    9744              :   /* All the rules in this function are specific to scalar integers.  */
    9745     48105967 :   scalar_int_mode mode;
    9746     48105967 :   if (!is_a <scalar_int_mode> (GET_MODE (dest), &mode))
    9747              :     return x;
    9748              : 
    9749              :   /* If SRC was (and (not (ashift (const_int 1) POS)) DEST), this is
    9750              :      a clear of a one-bit field.  We will have changed it to
    9751              :      (and (rotate (const_int -2) POS) DEST), so check for that.  Also check
    9752              :      for a SUBREG.  */
    9753              : 
    9754      1293518 :   if (GET_CODE (src) == AND && GET_CODE (XEXP (src, 0)) == ROTATE
    9755         1609 :       && CONST_INT_P (XEXP (XEXP (src, 0), 0))
    9756          547 :       && INTVAL (XEXP (XEXP (src, 0), 0)) == -2
    9757     22193431 :       && rtx_equal_for_field_assignment_p (dest, XEXP (src, 1)))
    9758              :     {
    9759          156 :       assign = make_extraction (VOIDmode, dest, 0, XEXP (XEXP (src, 0), 1),
    9760              :                                 1, true, true, false);
    9761          156 :       if (assign != 0)
    9762          153 :         return gen_rtx_SET (assign, const0_rtx);
    9763              :       return x;
    9764              :     }
    9765              : 
    9766      1293362 :   if (GET_CODE (src) == AND && GET_CODE (XEXP (src, 0)) == SUBREG
    9767        88514 :       && subreg_lowpart_p (XEXP (src, 0))
    9768        88481 :       && partial_subreg_p (XEXP (src, 0))
    9769        20326 :       && GET_CODE (SUBREG_REG (XEXP (src, 0))) == ROTATE
    9770          125 :       && CONST_INT_P (XEXP (SUBREG_REG (XEXP (src, 0)), 0))
    9771           57 :       && INTVAL (XEXP (SUBREG_REG (XEXP (src, 0)), 0)) == -2
    9772     22192785 :       && rtx_equal_for_field_assignment_p (dest, XEXP (src, 1)))
    9773              :     {
    9774           14 :       assign = make_extraction (VOIDmode, dest, 0,
    9775            7 :                                 XEXP (SUBREG_REG (XEXP (src, 0)), 1),
    9776              :                                 1, true, true, false);
    9777            7 :       if (assign != 0)
    9778            7 :         return gen_rtx_SET (assign, const0_rtx);
    9779              :       return x;
    9780              :     }
    9781              : 
    9782              :   /* If SRC is (ior (ashift (const_int 1) POS) DEST), this is a set of a
    9783              :      one-bit field.  */
    9784      1378397 :   if (GET_CODE (src) == IOR && GET_CODE (XEXP (src, 0)) == ASHIFT
    9785       346541 :       && XEXP (XEXP (src, 0), 0) == const1_rtx
    9786     22195024 :       && rtx_equal_for_field_assignment_p (dest, XEXP (src, 1)))
    9787              :     {
    9788          554 :       assign = make_extraction (VOIDmode, dest, 0, XEXP (XEXP (src, 0), 1),
    9789              :                                 1, true, true, false);
    9790          554 :       if (assign != 0)
    9791          525 :         return gen_rtx_SET (assign, const1_rtx);
    9792              :       return x;
    9793              :     }
    9794              : 
    9795              :   /* If DEST is already a field assignment, i.e. ZERO_EXTRACT, and the
    9796              :      SRC is an AND with all bits of that field set, then we can discard
    9797              :      the AND.  */
    9798     22192167 :   if (GET_CODE (dest) == ZERO_EXTRACT
    9799         2794 :       && CONST_INT_P (XEXP (dest, 1))
    9800         2794 :       && GET_CODE (src) == AND
    9801          858 :       && CONST_INT_P (XEXP (src, 1)))
    9802              :     {
    9803          858 :       HOST_WIDE_INT width = INTVAL (XEXP (dest, 1));
    9804          858 :       unsigned HOST_WIDE_INT and_mask = INTVAL (XEXP (src, 1));
    9805          858 :       unsigned HOST_WIDE_INT ze_mask;
    9806              : 
    9807          858 :       if (width >= HOST_BITS_PER_WIDE_INT)
    9808              :         ze_mask = -1;
    9809              :       else
    9810          858 :         ze_mask = (HOST_WIDE_INT_1U << width) - 1;
    9811              : 
    9812              :       /* Complete overlap.  We can remove the source AND.  */
    9813          858 :       if ((and_mask & ze_mask) == ze_mask)
    9814          792 :         return gen_rtx_SET (dest, XEXP (src, 0));
    9815              : 
    9816              :       /* Partial overlap.  We can reduce the source AND.  */
    9817           66 :       if ((and_mask & ze_mask) != and_mask)
    9818              :         {
    9819            6 :           src = gen_rtx_AND (mode, XEXP (src, 0),
    9820              :                              gen_int_mode (and_mask & ze_mask, mode));
    9821            6 :           return gen_rtx_SET (dest, src);
    9822              :         }
    9823              :     }
    9824              : 
    9825              :   /* The other case we handle is assignments into a constant-position
    9826              :      field.  They look like (ior/xor (and DEST C1) OTHER).  If C1 represents
    9827              :      a mask that has all one bits except for a group of zero bits and
    9828              :      OTHER is known to have zeros where C1 has ones, this is such an
    9829              :      assignment.  Compute the position and length from C1.  Shift OTHER
    9830              :      to the appropriate position, force it to the required mode, and
    9831              :      make the extraction.  Check for the AND in both operands.  */
    9832              : 
    9833              :   /* One or more SUBREGs might obscure the constant-position field
    9834              :      assignment.  The first one we are likely to encounter is an outer
    9835              :      narrowing SUBREG, which we can just strip for the purposes of
    9836              :      identifying the constant-field assignment.  */
    9837     22191369 :   scalar_int_mode src_mode = mode;
    9838     22191369 :   if (GET_CODE (src) == SUBREG
    9839       212499 :       && subreg_lowpart_p (src)
    9840     22387835 :       && is_a <scalar_int_mode> (GET_MODE (SUBREG_REG (src)), &src_mode))
    9841     22191369 :     src = SUBREG_REG (src);
    9842              : 
    9843     22191369 :   if (GET_CODE (src) != IOR
    9844     22191369 :       && GET_CODE (src) != XOR
    9845     20603679 :       && GET_CODE (src) != PLUS)
    9846              :     return x;
    9847              : 
    9848      8061304 :   rhs = expand_compound_operation (XEXP (src, 0));
    9849      8061304 :   lhs = expand_compound_operation (XEXP (src, 1));
    9850              : 
    9851      8061304 :   if (GET_CODE (rhs) == AND
    9852      1216715 :       && CONST_INT_P (XEXP (rhs, 1))
    9853      8866882 :       && rtx_equal_for_field_assignment_p (XEXP (rhs, 0), dest))
    9854        22262 :     c1 = INTVAL (XEXP (rhs, 1)), other = lhs;
    9855              :   /* The second SUBREG that might get in the way is a paradoxical
    9856              :      SUBREG around the first operand of the AND.  We want to
    9857              :      pretend the operand is as wide as the destination here.   We
    9858              :      do this by adjusting the MEM to wider mode for the sole
    9859              :      purpose of the call to rtx_equal_for_field_assignment_p.   Also
    9860              :      note this trick only works for MEMs.  */
    9861      8039042 :   else if (GET_CODE (rhs) == AND
    9862      1194453 :            && paradoxical_subreg_p (XEXP (rhs, 0))
    9863       176271 :            && MEM_P (SUBREG_REG (XEXP (rhs, 0)))
    9864        46404 :            && CONST_INT_P (XEXP (rhs, 1))
    9865      8085446 :            && rtx_equal_for_field_assignment_p (SUBREG_REG (XEXP (rhs, 0)),
    9866              :                                                 dest, true))
    9867            0 :     c1 = INTVAL (XEXP (rhs, 1)), other = lhs;
    9868      8039042 :   else if (GET_CODE (lhs) == AND
    9869       111113 :            && CONST_INT_P (XEXP (lhs, 1))
    9870      8139534 :            && rtx_equal_for_field_assignment_p (XEXP (lhs, 0), dest))
    9871           30 :     c1 = INTVAL (XEXP (lhs, 1)), other = rhs;
    9872              :   /* The second SUBREG that might get in the way is a paradoxical
    9873              :      SUBREG around the first operand of the AND.  We want to
    9874              :      pretend the operand is as wide as the destination here.   We
    9875              :      do this by adjusting the MEM to wider mode for the sole
    9876              :      purpose of the call to rtx_equal_for_field_assignment_p.   Also
    9877              :      note this trick only works for MEMs.  */
    9878      8039012 :   else if (GET_CODE (lhs) == AND
    9879       111083 :            && paradoxical_subreg_p (XEXP (lhs, 0))
    9880        58322 :            && MEM_P (SUBREG_REG (XEXP (lhs, 0)))
    9881        28046 :            && CONST_INT_P (XEXP (lhs, 1))
    9882      8067058 :            && rtx_equal_for_field_assignment_p (SUBREG_REG (XEXP (lhs, 0)),
    9883              :                                                 dest, true))
    9884            0 :     c1 = INTVAL (XEXP (lhs, 1)), other = rhs;
    9885              :   else
    9886              :     return x;
    9887              : 
    9888        22292 :   pos = get_pos_from_mask ((~c1) & GET_MODE_MASK (mode), &len);
    9889        22292 :   if (pos < 0
    9890        10852 :       || pos + len > GET_MODE_PRECISION (mode)
    9891        10852 :       || GET_MODE_PRECISION (mode) > HOST_BITS_PER_WIDE_INT
    9892        33132 :       || (c1 & nonzero_bits (other, mode)) != 0)
    9893              :     return x;
    9894              : 
    9895         5978 :   assign = make_extraction (VOIDmode, dest, pos, NULL_RTX, len,
    9896              :                             true, true, false);
    9897         5978 :   if (assign == 0)
    9898              :     return x;
    9899              : 
    9900              :   /* The mode to use for the source is the mode of the assignment, or of
    9901              :      what is inside a possible STRICT_LOW_PART.  */
    9902        11932 :   machine_mode new_mode = (GET_CODE (assign) == STRICT_LOW_PART
    9903         5966 :                            ? GET_MODE (XEXP (assign, 0)) : GET_MODE (assign));
    9904              : 
    9905              :   /* Shift OTHER right POS places and make it the source, restricting it
    9906              :      to the proper length and mode.  */
    9907              : 
    9908         5966 :   src = canon_reg_for_combine (simplify_shift_const (NULL_RTX, LSHIFTRT,
    9909              :                                                      src_mode, other, pos),
    9910              :                                dest);
    9911        11932 :   src = force_to_mode (src, new_mode,
    9912              :                        len >= HOST_BITS_PER_WIDE_INT
    9913              :                        ? HOST_WIDE_INT_M1U
    9914         5966 :                        : (HOST_WIDE_INT_1U << len) - 1, false);
    9915              : 
    9916              :   /* If SRC is masked by an AND that does not make a difference in
    9917              :      the value being stored, strip it.  */
    9918         5966 :   if (GET_CODE (assign) == ZERO_EXTRACT
    9919         5911 :       && CONST_INT_P (XEXP (assign, 1))
    9920         5911 :       && INTVAL (XEXP (assign, 1)) < HOST_BITS_PER_WIDE_INT
    9921         5911 :       && GET_CODE (src) == AND
    9922            0 :       && CONST_INT_P (XEXP (src, 1))
    9923            0 :       && UINTVAL (XEXP (src, 1))
    9924            0 :          == (HOST_WIDE_INT_1U << INTVAL (XEXP (assign, 1))) - 1)
    9925            0 :     src = XEXP (src, 0);
    9926              : 
    9927         5966 :   return gen_rtx_SET (assign, src);
    9928              : }
    9929              : 
    9930              : /* See if X is of the form (+ (* a c) (* b c)) and convert to (* (+ a b) c)
    9931              :    if so.  */
    9932              : 
    9933              : static rtx
    9934     53474539 : apply_distributive_law (rtx x)
    9935              : {
    9936     53474539 :   enum rtx_code code = GET_CODE (x);
    9937     53474539 :   enum rtx_code inner_code;
    9938     53474539 :   rtx lhs, rhs, other;
    9939     53474539 :   rtx tem;
    9940              : 
    9941              :   /* Distributivity is not true for floating point as it can change the
    9942              :      value.  So we don't do it unless -funsafe-math-optimizations.  */
    9943     53474539 :   if (FLOAT_MODE_P (GET_MODE (x))
    9944      3659380 :       && ! flag_unsafe_math_optimizations)
    9945              :     return x;
    9946              : 
    9947              :   /* The outer operation can only be one of the following:  */
    9948     50250664 :   if (code != IOR && code != AND && code != XOR
    9949     50250664 :       && code != PLUS && code != MINUS)
    9950              :     return x;
    9951              : 
    9952     50236483 :   lhs = XEXP (x, 0);
    9953     50236483 :   rhs = XEXP (x, 1);
    9954              : 
    9955              :   /* If either operand is a primitive we can't do anything, so get out
    9956              :      fast.  */
    9957     50236483 :   if (OBJECT_P (lhs) || OBJECT_P (rhs))
    9958              :     return x;
    9959              : 
    9960      3656442 :   lhs = expand_compound_operation (lhs);
    9961      3656442 :   rhs = expand_compound_operation (rhs);
    9962      3656442 :   inner_code = GET_CODE (lhs);
    9963      3656442 :   if (inner_code != GET_CODE (rhs))
    9964              :     return x;
    9965              : 
    9966              :   /* See if the inner and outer operations distribute.  */
    9967      1049850 :   switch (inner_code)
    9968              :     {
    9969       273096 :     case LSHIFTRT:
    9970       273096 :     case ASHIFTRT:
    9971       273096 :     case AND:
    9972       273096 :     case IOR:
    9973              :       /* These all distribute except over PLUS.  */
    9974       273096 :       if (code == PLUS || code == MINUS)
    9975              :         return x;
    9976              :       break;
    9977              : 
    9978       103065 :     case MULT:
    9979       103065 :       if (code != PLUS && code != MINUS)
    9980              :         return x;
    9981              :       break;
    9982              : 
    9983              :     case ASHIFT:
    9984              :       /* This is also a multiply, so it distributes over everything.  */
    9985              :       break;
    9986              : 
    9987              :     /* This used to handle SUBREG, but this turned out to be counter-
    9988              :        productive, since (subreg (op ...)) usually is not handled by
    9989              :        insn patterns, and this "optimization" therefore transformed
    9990              :        recognizable patterns into unrecognizable ones.  Therefore the
    9991              :        SUBREG case was removed from here.
    9992              : 
    9993              :        It is possible that distributing SUBREG over arithmetic operations
    9994              :        leads to an intermediate result than can then be optimized further,
    9995              :        e.g. by moving the outer SUBREG to the other side of a SET as done
    9996              :        in simplify_set.  This seems to have been the original intent of
    9997              :        handling SUBREGs here.
    9998              : 
    9999              :        However, with current GCC this does not appear to actually happen,
   10000              :        at least on major platforms.  If some case is found where removing
   10001              :        the SUBREG case here prevents follow-on optimizations, distributing
   10002              :        SUBREGs ought to be re-added at that place, e.g. in simplify_set.  */
   10003              : 
   10004              :     default:
   10005              :       return x;
   10006              :     }
   10007              : 
   10008              :   /* Set LHS and RHS to the inner operands (A and B in the example
   10009              :      above) and set OTHER to the common operand (C in the example).
   10010              :      There is only one way to do this unless the inner operation is
   10011              :      commutative.  */
   10012       287183 :   if (COMMUTATIVE_ARITH_P (lhs)
   10013       287183 :       && rtx_equal_p (XEXP (lhs, 0), XEXP (rhs, 0)))
   10014         2205 :     other = XEXP (lhs, 0), lhs = XEXP (lhs, 1), rhs = XEXP (rhs, 1);
   10015       284978 :   else if (COMMUTATIVE_ARITH_P (lhs)
   10016       284978 :            && rtx_equal_p (XEXP (lhs, 0), XEXP (rhs, 1)))
   10017           21 :     other = XEXP (lhs, 0), lhs = XEXP (lhs, 1), rhs = XEXP (rhs, 0);
   10018       284957 :   else if (COMMUTATIVE_ARITH_P (lhs)
   10019       284957 :            && rtx_equal_p (XEXP (lhs, 1), XEXP (rhs, 0)))
   10020        11374 :     other = XEXP (lhs, 1), lhs = XEXP (lhs, 0), rhs = XEXP (rhs, 1);
   10021       273583 :   else if (rtx_equal_p (XEXP (lhs, 1), XEXP (rhs, 1)))
   10022        70447 :     other = XEXP (lhs, 1), lhs = XEXP (lhs, 0), rhs = XEXP (rhs, 0);
   10023              :   else
   10024              :     return x;
   10025              : 
   10026              :   /* Form the new inner operation, seeing if it simplifies first.  */
   10027        84047 :   tem = simplify_gen_binary (code, GET_MODE (x), lhs, rhs);
   10028              : 
   10029              :   /* There is one exception to the general way of distributing:
   10030              :      (a | c) ^ (b | c) -> (a ^ b) & ~c  */
   10031        84047 :   if (code == XOR && inner_code == IOR)
   10032              :     {
   10033         1269 :       inner_code = AND;
   10034         1269 :       other = simplify_gen_unary (NOT, GET_MODE (x), other, GET_MODE (x));
   10035              :     }
   10036              : 
   10037              :   /* We may be able to continuing distributing the result, so call
   10038              :      ourselves recursively on the inner operation before forming the
   10039              :      outer operation, which we return.  */
   10040        84047 :   return simplify_gen_binary (inner_code, GET_MODE (x),
   10041        84047 :                               apply_distributive_law (tem), other);
   10042              : }
   10043              : 
   10044              : /* See if X is of the form (* (+ A B) C), and if so convert to
   10045              :    (+ (* A C) (* B C)) and try to simplify.
   10046              : 
   10047              :    Most of the time, this results in no change.  However, if some of
   10048              :    the operands are the same or inverses of each other, simplifications
   10049              :    will result.
   10050              : 
   10051              :    For example, (and (ior A B) (not B)) can occur as the result of
   10052              :    expanding a bit field assignment.  When we apply the distributive
   10053              :    law to this, we get (ior (and (A (not B))) (and (B (not B)))),
   10054              :    which then simplifies to (and (A (not B))).
   10055              : 
   10056              :    Note that no checks happen on the validity of applying the inverse
   10057              :    distributive law.  This is pointless since we can do it in the
   10058              :    few places where this routine is called.
   10059              : 
   10060              :    N is the index of the term that is decomposed (the arithmetic operation,
   10061              :    i.e. (+ A B) in the first example above).  !N is the index of the term that
   10062              :    is distributed, i.e. of C in the first example above.  */
   10063              : static rtx
   10064      1735405 : distribute_and_simplify_rtx (rtx x, int n)
   10065              : {
   10066      1735405 :   machine_mode mode;
   10067      1735405 :   enum rtx_code outer_code, inner_code;
   10068      1735405 :   rtx decomposed, distributed, inner_op0, inner_op1, new_op0, new_op1, tmp;
   10069              : 
   10070              :   /* Distributivity is not true for floating point as it can change the
   10071              :      value.  So we don't do it unless -funsafe-math-optimizations.  */
   10072      1735405 :   if (FLOAT_MODE_P (GET_MODE (x))
   10073       165362 :       && ! flag_unsafe_math_optimizations)
   10074              :     return NULL_RTX;
   10075              : 
   10076      1573582 :   decomposed = XEXP (x, n);
   10077      1573582 :   if (!ARITHMETIC_P (decomposed))
   10078              :     return NULL_RTX;
   10079              : 
   10080      1573582 :   mode = GET_MODE (x);
   10081      1573582 :   outer_code = GET_CODE (x);
   10082      1573582 :   distributed = XEXP (x, !n);
   10083              : 
   10084      1573582 :   inner_code = GET_CODE (decomposed);
   10085      1573582 :   inner_op0 = XEXP (decomposed, 0);
   10086      1573582 :   inner_op1 = XEXP (decomposed, 1);
   10087              : 
   10088              :   /* Special case (and (xor B C) (not A)), which is equivalent to
   10089              :      (xor (ior A B) (ior A C))  */
   10090        47694 :   if (outer_code == AND && inner_code == XOR && GET_CODE (distributed) == NOT
   10091              :       /* But not when A is a paradoxical subreg since the masking effect
   10092              :          of the outer AND is lost and the duplicated use can result in
   10093              :          undefined bits to be exposed when spilling.  */
   10094      1574846 :       && !contains_paradoxical_subreg_p (XEXP (distributed, 0)))
   10095              :     {
   10096         1264 :       distributed = XEXP (distributed, 0);
   10097         1264 :       outer_code = IOR;
   10098              :     }
   10099              : 
   10100      1573582 :   if (n == 0)
   10101              :     {
   10102              :       /* Distribute the second term.  */
   10103      1521946 :       new_op0 = simplify_gen_binary (outer_code, mode, inner_op0, distributed);
   10104      1521946 :       new_op1 = simplify_gen_binary (outer_code, mode, inner_op1, distributed);
   10105              :     }
   10106              :   else
   10107              :     {
   10108              :       /* Distribute the first term.  */
   10109        51636 :       new_op0 = simplify_gen_binary (outer_code, mode, distributed, inner_op0);
   10110        51636 :       new_op1 = simplify_gen_binary (outer_code, mode, distributed, inner_op1);
   10111              :     }
   10112              : 
   10113      1573582 :   tmp = apply_distributive_law (simplify_gen_binary (inner_code, mode,
   10114              :                                                      new_op0, new_op1));
   10115      1573582 :   if (GET_CODE (tmp) != outer_code
   10116      1573582 :       && (set_src_cost (tmp, mode, optimize_this_for_speed_p)
   10117       268175 :           < set_src_cost (x, mode, optimize_this_for_speed_p)))
   10118        17486 :     return tmp;
   10119              : 
   10120              :   return NULL_RTX;
   10121              : }
   10122              : 
   10123              : /* Simplify a logical `and' of VAROP with the constant CONSTOP, to be done
   10124              :    in MODE.  Return an equivalent form, if different from (and VAROP
   10125              :    (const_int CONSTOP)).  Otherwise, return NULL_RTX.  */
   10126              : 
   10127              : static rtx
   10128     12790865 : simplify_and_const_int_1 (scalar_int_mode mode, rtx varop,
   10129              :                           unsigned HOST_WIDE_INT constop)
   10130              : {
   10131     12790865 :   unsigned HOST_WIDE_INT nonzero;
   10132     12790865 :   unsigned HOST_WIDE_INT orig_constop;
   10133     12790865 :   rtx orig_varop;
   10134     12790865 :   int i;
   10135              : 
   10136     12790865 :   orig_varop = varop;
   10137     12790865 :   orig_constop = constop;
   10138     12790865 :   if (GET_CODE (varop) == CLOBBER)
   10139              :     return NULL_RTX;
   10140              : 
   10141              :   /* Simplify VAROP knowing that we will be only looking at some of the
   10142              :      bits in it.
   10143              : 
   10144              :      Note by passing in CONSTOP, we guarantee that the bits not set in
   10145              :      CONSTOP are not significant and will never be examined.  We must
   10146              :      ensure that is the case by explicitly masking out those bits
   10147              :      before returning.  */
   10148     12790850 :   varop = force_to_mode (varop, mode, constop, false);
   10149              : 
   10150              :   /* If VAROP is a CLOBBER, we will fail so return it.  */
   10151     12790850 :   if (GET_CODE (varop) == CLOBBER)
   10152              :     return varop;
   10153              : 
   10154              :   /* If VAROP is a CONST_INT, then we need to apply the mask in CONSTOP
   10155              :      to VAROP and return the new constant.  */
   10156     12790804 :   if (CONST_INT_P (varop))
   10157       301598 :     return gen_int_mode (INTVAL (varop) & constop, mode);
   10158              : 
   10159              :   /* See what bits may be nonzero in VAROP.  Unlike the general case of
   10160              :      a call to nonzero_bits, here we don't care about bits outside
   10161              :      MODE unless WORD_REGISTER_OPERATIONS is true.  */
   10162              : 
   10163     12489206 :   scalar_int_mode tmode = mode;
   10164     12489206 :   if (WORD_REGISTER_OPERATIONS && GET_MODE_BITSIZE (mode) < BITS_PER_WORD)
   10165              :     tmode = word_mode;
   10166     12489206 :   nonzero = nonzero_bits (varop, tmode) & GET_MODE_MASK (tmode);
   10167              : 
   10168              :   /* Turn off all bits in the constant that are known to already be zero.
   10169              :      Thus, if the AND isn't needed at all, we will have CONSTOP == NONZERO_BITS
   10170              :      which is tested below.  */
   10171              : 
   10172     12489206 :   constop &= nonzero;
   10173              : 
   10174              :   /* If we don't have any bits left, return zero.  */
   10175     12489206 :   if (constop == 0 && !side_effects_p (varop))
   10176            0 :     return const0_rtx;
   10177              : 
   10178              :   /* If VAROP is a NEG of something known to be zero or 1 and CONSTOP is
   10179              :      a power of two, we can replace this with an ASHIFT.  */
   10180        34623 :   if (GET_CODE (varop) == NEG && nonzero_bits (XEXP (varop, 0), tmode) == 1
   10181     12495102 :       && (i = exact_log2 (constop)) >= 0)
   10182          163 :     return simplify_shift_const (NULL_RTX, ASHIFT, mode, XEXP (varop, 0), i);
   10183              : 
   10184              :   /* If VAROP is an IOR or XOR, apply the AND to both branches of the IOR
   10185              :      or XOR, then try to apply the distributive law.  This may eliminate
   10186              :      operations if either branch can be simplified because of the AND.
   10187              :      It may also make some cases more complex, but those cases probably
   10188              :      won't match a pattern either with or without this.  */
   10189              : 
   10190     12489043 :   if (GET_CODE (varop) == IOR || GET_CODE (varop) == XOR)
   10191              :     {
   10192        88479 :       scalar_int_mode varop_mode = as_a <scalar_int_mode> (GET_MODE (varop));
   10193        88479 :       return
   10194        88479 :         gen_lowpart
   10195        88479 :           (mode,
   10196              :            apply_distributive_law
   10197        88479 :            (simplify_gen_binary (GET_CODE (varop), varop_mode,
   10198              :                                  simplify_and_const_int (NULL_RTX, varop_mode,
   10199              :                                                          XEXP (varop, 0),
   10200              :                                                          constop),
   10201              :                                  simplify_and_const_int (NULL_RTX, varop_mode,
   10202              :                                                          XEXP (varop, 1),
   10203              :                                                          constop))));
   10204              :     }
   10205              : 
   10206              :   /* If VAROP is PLUS, and the constant is a mask of low bits, distribute
   10207              :      the AND and see if one of the operands simplifies to zero.  If so, we
   10208              :      may eliminate it.  */
   10209              : 
   10210     12400564 :   if (GET_CODE (varop) == PLUS
   10211     12400564 :       && pow2p_hwi (constop + 1))
   10212              :     {
   10213       443724 :       rtx o0, o1;
   10214              : 
   10215       443724 :       o0 = simplify_and_const_int (NULL_RTX, mode, XEXP (varop, 0), constop);
   10216       443724 :       o1 = simplify_and_const_int (NULL_RTX, mode, XEXP (varop, 1), constop);
   10217       443724 :       if (o0 == const0_rtx)
   10218              :         return o1;
   10219       443724 :       if (o1 == const0_rtx)
   10220              :         return o0;
   10221              :     }
   10222              : 
   10223              :   /* Make a SUBREG if necessary.  If we can't make it, fail.  */
   10224     12400488 :   varop = gen_lowpart (mode, varop);
   10225     12400488 :   if (varop == NULL_RTX || GET_CODE (varop) == CLOBBER)
   10226              :     return NULL_RTX;
   10227              : 
   10228              :   /* If we are only masking insignificant bits, return VAROP.  */
   10229     12400488 :   if (constop == nonzero)
   10230              :     return varop;
   10231              : 
   10232     11933829 :   if (varop == orig_varop && constop == orig_constop)
   10233              :     return NULL_RTX;
   10234              : 
   10235              :   /* Otherwise, return an AND.  */
   10236      6440971 :   return simplify_gen_binary (AND, mode, varop, gen_int_mode (constop, mode));
   10237              : }
   10238              : 
   10239              : 
   10240              : /* We have X, a logical `and' of VAROP with the constant CONSTOP, to be done
   10241              :    in MODE.
   10242              : 
   10243              :    Return an equivalent form, if different from X.  Otherwise, return X.  If
   10244              :    X is zero, we are to always construct the equivalent form.  */
   10245              : 
   10246              : static rtx
   10247     12790865 : simplify_and_const_int (rtx x, scalar_int_mode mode, rtx varop,
   10248              :                         unsigned HOST_WIDE_INT constop)
   10249              : {
   10250     12790865 :   rtx tem = simplify_and_const_int_1 (mode, varop, constop);
   10251     12790865 :   if (tem)
   10252              :     return tem;
   10253              : 
   10254      5492873 :   if (!x)
   10255      1407442 :     x = simplify_gen_binary (AND, GET_MODE (varop), varop,
   10256      1407442 :                              gen_int_mode (constop, mode));
   10257      5492873 :   if (GET_MODE (x) != mode)
   10258            2 :     x = gen_lowpart (mode, x);
   10259              :   return x;
   10260              : }
   10261              : 
   10262              : /* Given a REG X of mode XMODE, compute which bits in X can be nonzero.
   10263              :    We don't care about bits outside of those defined in MODE.
   10264              :    We DO care about all the bits in MODE, even if XMODE is smaller than MODE.
   10265              : 
   10266              :    For most X this is simply GET_MODE_MASK (GET_MODE (MODE)), but if X is
   10267              :    a shift, AND, or zero_extract, we can do better.  */
   10268              : 
   10269              : static rtx
   10270    467176840 : reg_nonzero_bits_for_combine (const_rtx x, scalar_int_mode xmode,
   10271              :                               scalar_int_mode mode,
   10272              :                               unsigned HOST_WIDE_INT *nonzero)
   10273              : {
   10274    467176840 :   rtx tem;
   10275    467176840 :   reg_stat_type *rsp;
   10276              : 
   10277              :   /* If X is a register whose nonzero bits value is current, use it.
   10278              :      Otherwise, if X is a register whose value we can find, use that
   10279              :      value.  Otherwise, use the previously-computed global nonzero bits
   10280              :      for this register.  */
   10281              : 
   10282    467176840 :   rsp = &reg_stat[REGNO (x)];
   10283    467176840 :   if (rsp->last_set_value != 0
   10284    431639805 :       && (rsp->last_set_mode == mode
   10285         1278 :           || (REGNO (x) >= FIRST_PSEUDO_REGISTER
   10286            0 :               && GET_MODE_CLASS (rsp->last_set_mode) == MODE_INT
   10287            0 :               && GET_MODE_CLASS (mode) == MODE_INT))
   10288    898815367 :       && ((rsp->last_set_label >= label_tick_ebb_start
   10289    328501160 :            && rsp->last_set_label < label_tick)
   10290    408406319 :           || (rsp->last_set_label == label_tick
   10291    305268952 :               && DF_INSN_LUID (rsp->last_set) < subst_low_luid)
   10292    132987625 :           || (REGNO (x) >= FIRST_PSEUDO_REGISTER
   10293    132934387 :               && REGNO (x) < reg_n_sets_max
   10294    132934257 :               && REG_N_SETS (REGNO (x)) == 1
   10295    154123224 :               && !REGNO_REG_SET_P
   10296              :                   (DF_LR_IN (ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb),
   10297              :                    REGNO (x)))))
   10298              :     {
   10299              :       /* Note that, even if the precision of last_set_mode is lower than that
   10300              :          of mode, record_value_for_reg invoked nonzero_bits on the register
   10301              :          with nonzero_bits_mode (because last_set_mode is necessarily integral
   10302              :          and HWI_COMPUTABLE_MODE_P in this case) so bits in nonzero_bits_mode
   10303              :          are all valid, hence in mode too since nonzero_bits_mode is defined
   10304              :          to the largest HWI_COMPUTABLE_MODE_P mode.  */
   10305    375615192 :       *nonzero &= rsp->last_set_nonzero_bits;
   10306    375615192 :       return NULL;
   10307              :     }
   10308              : 
   10309     91561648 :   tem = get_last_value (x);
   10310     91561648 :   if (tem)
   10311              :     {
   10312              :       if (SHORT_IMMEDIATES_SIGN_EXTEND)
   10313              :         tem = sign_extend_short_imm (tem, xmode, GET_MODE_PRECISION (mode));
   10314              : 
   10315              :       return tem;
   10316              :     }
   10317              : 
   10318     91561642 :   if (nonzero_sign_valid && rsp->nonzero_bits)
   10319              :     {
   10320     57935299 :       unsigned HOST_WIDE_INT mask = rsp->nonzero_bits;
   10321              : 
   10322     57935299 :       if (GET_MODE_PRECISION (xmode) < GET_MODE_PRECISION (mode))
   10323              :         /* We don't know anything about the upper bits.  */
   10324            0 :         mask |= GET_MODE_MASK (mode) ^ GET_MODE_MASK (xmode);
   10325              : 
   10326     57935299 :       *nonzero &= mask;
   10327              :     }
   10328              : 
   10329              :   return NULL;
   10330              : }
   10331              : 
   10332              : /* Given a reg X of mode XMODE, return the number of bits at the high-order
   10333              :    end of X that are known to be equal to the sign bit.  X will be used
   10334              :    in mode MODE; the returned value will always be between 1 and the
   10335              :    number of bits in MODE.  */
   10336              : 
   10337              : static rtx
   10338    135047238 : reg_num_sign_bit_copies_for_combine (const_rtx x, scalar_int_mode xmode,
   10339              :                                      scalar_int_mode mode,
   10340              :                                      unsigned int *result)
   10341              : {
   10342    135047238 :   rtx tem;
   10343    135047238 :   reg_stat_type *rsp;
   10344              : 
   10345    135047238 :   rsp = &reg_stat[REGNO (x)];
   10346    135047238 :   if (rsp->last_set_value != 0
   10347    122729763 :       && rsp->last_set_mode == mode
   10348    257776832 :       && ((rsp->last_set_label >= label_tick_ebb_start
   10349     93407050 :            && rsp->last_set_label < label_tick)
   10350    116625380 :           || (rsp->last_set_label == label_tick
   10351     87302836 :               && DF_INSN_LUID (rsp->last_set) < subst_low_luid)
   10352     37538173 :           || (REGNO (x) >= FIRST_PSEUDO_REGISTER
   10353     37528248 :               && REGNO (x) < reg_n_sets_max
   10354     37528162 :               && REG_N_SETS (REGNO (x)) == 1
   10355     43498438 :               && !REGNO_REG_SET_P
   10356              :                   (DF_LR_IN (ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb),
   10357              :                    REGNO (x)))))
   10358              :     {
   10359    106916021 :       *result = rsp->last_set_sign_bit_copies;
   10360    106916021 :       return NULL;
   10361              :     }
   10362              : 
   10363     28131217 :   tem = get_last_value (x);
   10364     28131217 :   if (tem != 0)
   10365              :     return tem;
   10366              : 
   10367     18684142 :   if (nonzero_sign_valid && rsp->sign_bit_copies != 0
   10368     42690381 :       && GET_MODE_PRECISION (xmode) == GET_MODE_PRECISION (mode))
   10369     14559169 :     *result = rsp->sign_bit_copies;
   10370              : 
   10371              :   return NULL;
   10372              : }
   10373              : 
   10374              : /* Return the number of "extended" bits there are in X, when interpreted
   10375              :    as a quantity in MODE whose signedness is indicated by UNSIGNEDP.  For
   10376              :    unsigned quantities, this is the number of high-order zero bits.
   10377              :    For signed quantities, this is the number of copies of the sign bit
   10378              :    minus 1.  In both case, this function returns the number of "spare"
   10379              :    bits.  For example, if two quantities for which this function returns
   10380              :    at least 1 are added, the addition is known not to overflow.
   10381              : 
   10382              :    This function will always return 0 unless called during combine, which
   10383              :    implies that it must be called from a define_split.  */
   10384              : 
   10385              : unsigned int
   10386            0 : extended_count (const_rtx x, machine_mode mode, bool unsignedp)
   10387              : {
   10388            0 :   if (nonzero_sign_valid == 0)
   10389              :     return 0;
   10390              : 
   10391            0 :   scalar_int_mode int_mode;
   10392            0 :   return (unsignedp
   10393            0 :           ? (is_a <scalar_int_mode> (mode, &int_mode)
   10394            0 :              && HWI_COMPUTABLE_MODE_P (int_mode)
   10395            0 :              ? (unsigned int) (GET_MODE_PRECISION (int_mode) - 1
   10396            0 :                                - floor_log2 (nonzero_bits (x, int_mode)))
   10397              :              : 0)
   10398            0 :           : num_sign_bit_copies (x, mode) - 1);
   10399              : }
   10400              : 
   10401              : /* This function is called from `simplify_shift_const' to merge two
   10402              :    outer operations.  Specifically, we have already found that we need
   10403              :    to perform operation *POP0 with constant *PCONST0 at the outermost
   10404              :    position.  We would now like to also perform OP1 with constant CONST1
   10405              :    (with *POP0 being done last).
   10406              : 
   10407              :    Return true if we can do the operation and update *POP0 and *PCONST0 with
   10408              :    the resulting operation.  *PCOMP_P is set to true if we would need to
   10409              :    complement the innermost operand, otherwise it is unchanged.
   10410              : 
   10411              :    MODE is the mode in which the operation will be done.  No bits outside
   10412              :    the width of this mode matter.  It is assumed that the width of this mode
   10413              :    is smaller than or equal to HOST_BITS_PER_WIDE_INT.
   10414              : 
   10415              :    If *POP0 or OP1 are UNKNOWN, it means no operation is required.  Only NEG, PLUS,
   10416              :    IOR, XOR, and AND are supported.  We may set *POP0 to SET if the proper
   10417              :    result is simply *PCONST0.
   10418              : 
   10419              :    If the resulting operation cannot be expressed as one operation, we
   10420              :    return false and do not change *POP0, *PCONST0, and *PCOMP_P.  */
   10421              : 
   10422              : static bool
   10423      4002308 : merge_outer_ops (enum rtx_code *pop0, HOST_WIDE_INT *pconst0,
   10424              :                  enum rtx_code op1, HOST_WIDE_INT const1,
   10425              :                  machine_mode mode, bool *pcomp_p)
   10426              : {
   10427      4002308 :   enum rtx_code op0 = *pop0;
   10428      4002308 :   HOST_WIDE_INT const0 = *pconst0;
   10429              : 
   10430      4002308 :   const0 &= GET_MODE_MASK (mode);
   10431      4002308 :   const1 &= GET_MODE_MASK (mode);
   10432              : 
   10433              :   /* If OP0 is an AND, clear unimportant bits in CONST1.  */
   10434      4002308 :   if (op0 == AND)
   10435        25609 :     const1 &= const0;
   10436              : 
   10437              :   /* If OP0 or OP1 is UNKNOWN, this is easy.  Similarly if they are the same or
   10438              :      if OP0 is SET.  */
   10439              : 
   10440      4002308 :   if (op1 == UNKNOWN || op0 == SET)
   10441              :     return true;
   10442              : 
   10443      4002308 :   else if (op0 == UNKNOWN)
   10444              :     op0 = op1, const0 = const1;
   10445              : 
   10446        77636 :   else if (op0 == op1)
   10447              :     {
   10448        25322 :       switch (op0)
   10449              :         {
   10450        25322 :         case AND:
   10451        25322 :           const0 &= const1;
   10452        25322 :           break;
   10453            0 :         case IOR:
   10454            0 :           const0 |= const1;
   10455            0 :           break;
   10456            0 :         case XOR:
   10457            0 :           const0 ^= const1;
   10458            0 :           break;
   10459            0 :         case PLUS:
   10460            0 :           const0 += const1;
   10461            0 :           break;
   10462              :         case NEG:
   10463      3974501 :           op0 = UNKNOWN;
   10464              :           break;
   10465              :         default:
   10466              :           break;
   10467              :         }
   10468              :     }
   10469              : 
   10470              :   /* Otherwise, if either is a PLUS or NEG, we can't do anything.  */
   10471        52314 :   else if (op0 == PLUS || op1 == PLUS || op0 == NEG || op1 == NEG)
   10472              :     return false;
   10473              : 
   10474              :   /* If the two constants aren't the same, we can't do anything.  The
   10475              :      remaining six cases can all be done.  */
   10476        25654 :   else if (const0 != const1)
   10477              :     return false;
   10478              : 
   10479              :   else
   10480        24507 :     switch (op0)
   10481              :       {
   10482            8 :       case IOR:
   10483            8 :         if (op1 == AND)
   10484              :           /* (a & b) | b == b */
   10485            0 :           op0 = SET;
   10486              :         else /* op1 == XOR */
   10487              :           /* (a ^ b) | b == a | b */
   10488              :           {;}
   10489              :         break;
   10490              : 
   10491        24216 :       case XOR:
   10492        24216 :         if (op1 == AND)
   10493              :           /* (a & b) ^ b == (~a) & b */
   10494        24216 :           op0 = AND, *pcomp_p = true;
   10495              :         else /* op1 == IOR */
   10496              :           /* (a | b) ^ b == a & ~b */
   10497            0 :           op0 = AND, const0 = ~const0;
   10498              :         break;
   10499              : 
   10500          283 :       case AND:
   10501          283 :         if (op1 == IOR)
   10502              :           /* (a | b) & b == b */
   10503              :         op0 = SET;
   10504              :         else /* op1 == XOR */
   10505              :           /* (a ^ b) & b) == (~a) & b */
   10506          283 :           *pcomp_p = true;
   10507              :         break;
   10508              :       default:
   10509              :         break;
   10510              :       }
   10511              : 
   10512              :   /* Check for NO-OP cases.  */
   10513      3974501 :   const0 &= GET_MODE_MASK (mode);
   10514      3974501 :   if (const0 == 0
   10515        19624 :       && (op0 == IOR || op0 == XOR || op0 == PLUS))
   10516              :     op0 = UNKNOWN;
   10517      3973063 :   else if (const0 == 0 && op0 == AND)
   10518              :     op0 = SET;
   10519      3973063 :   else if ((unsigned HOST_WIDE_INT) const0 == GET_MODE_MASK (mode)
   10520        25130 :            && op0 == AND)
   10521      3974501 :     op0 = UNKNOWN;
   10522              : 
   10523      3974501 :   *pop0 = op0;
   10524              : 
   10525              :   /* ??? Slightly redundant with the above mask, but not entirely.
   10526              :      Moving this above means we'd have to sign-extend the mode mask
   10527              :      for the final test.  */
   10528      3974501 :   if (op0 != UNKNOWN && op0 != NEG)
   10529      3938284 :     *pconst0 = trunc_int_for_mode (const0, mode);
   10530              : 
   10531              :   return true;
   10532              : }
   10533              : 
   10534              : /* A helper to simplify_shift_const_1 to determine the mode we can perform
   10535              :    the shift in.  The original shift operation CODE is performed on OP in
   10536              :    ORIG_MODE.  Return the wider mode MODE if we can perform the operation
   10537              :    in that mode.  Return ORIG_MODE otherwise.  We can also assume that the
   10538              :    result of the shift is subject to operation OUTER_CODE with operand
   10539              :    OUTER_CONST.  */
   10540              : 
   10541              : static scalar_int_mode
   10542       408114 : try_widen_shift_mode (enum rtx_code code, rtx op, int count,
   10543              :                       scalar_int_mode orig_mode, scalar_int_mode mode,
   10544              :                       enum rtx_code outer_code, HOST_WIDE_INT outer_const)
   10545              : {
   10546       408114 :   gcc_assert (GET_MODE_PRECISION (mode) > GET_MODE_PRECISION (orig_mode));
   10547              : 
   10548              :   /* In general we can't perform in wider mode for right shift and rotate.  */
   10549       408114 :   switch (code)
   10550              :     {
   10551        29587 :     case ASHIFTRT:
   10552              :       /* We can still widen if the bits brought in from the left are identical
   10553              :          to the sign bit of ORIG_MODE.  */
   10554        29587 :       if (num_sign_bit_copies (op, mode)
   10555        29587 :           > (unsigned) (GET_MODE_PRECISION (mode)
   10556        29587 :                         - GET_MODE_PRECISION (orig_mode)))
   10557          336 :         return mode;
   10558        29251 :       return orig_mode;
   10559              : 
   10560        74819 :     case LSHIFTRT:
   10561              :       /* Similarly here but with zero bits.  */
   10562        74819 :       if (HWI_COMPUTABLE_MODE_P (mode)
   10563        74819 :           && (nonzero_bits (op, mode) & ~GET_MODE_MASK (orig_mode)) == 0)
   10564         6462 :         return mode;
   10565              : 
   10566              :       /* We can also widen if the bits brought in will be masked off.  This
   10567              :          operation is performed in ORIG_MODE.  */
   10568        68357 :       if (outer_code == AND)
   10569              :         {
   10570        27384 :           int care_bits = low_bitmask_len (orig_mode, outer_const);
   10571              : 
   10572        27384 :           if (care_bits >= 0
   10573        27384 :               && GET_MODE_PRECISION (orig_mode) - care_bits >= count)
   10574        27364 :             return mode;
   10575              :         }
   10576              :       /* fall through */
   10577              : 
   10578        41479 :     case ROTATE:
   10579        41479 :       return orig_mode;
   10580              : 
   10581            0 :     case ROTATERT:
   10582            0 :       gcc_unreachable ();
   10583              : 
   10584       303222 :     default:
   10585       303222 :       return mode;
   10586              :     }
   10587              : }
   10588              : 
   10589              : /* Simplify a shift of VAROP by ORIG_COUNT bits.  CODE says what kind
   10590              :    of shift.  The result of the shift is RESULT_MODE.  Return NULL_RTX
   10591              :    if we cannot simplify it.  Otherwise, return a simplified value.
   10592              : 
   10593              :    The shift is normally computed in the widest mode we find in VAROP, as
   10594              :    long as it isn't a different number of words than RESULT_MODE.  Exceptions
   10595              :    are ASHIFTRT and ROTATE, which are always done in their original mode.  */
   10596              : 
   10597              : static rtx
   10598     25157211 : simplify_shift_const_1 (enum rtx_code code, machine_mode result_mode,
   10599              :                         rtx varop, int orig_count)
   10600              : {
   10601     25157211 :   enum rtx_code orig_code = code;
   10602     25157211 :   rtx orig_varop = varop;
   10603     25157211 :   int count, log2;
   10604     25157211 :   machine_mode mode = result_mode;
   10605     25157211 :   machine_mode shift_mode;
   10606     25157211 :   scalar_int_mode tmode, inner_mode, int_mode, int_varop_mode, int_result_mode;
   10607              :   /* We form (outer_op (code varop count) (outer_const)).  */
   10608     25157211 :   enum rtx_code outer_op = UNKNOWN;
   10609     25157211 :   HOST_WIDE_INT outer_const = 0;
   10610     25157211 :   bool complement_p = false;
   10611     25157211 :   rtx new_rtx, x;
   10612              : 
   10613              :   /* Make sure and truncate the "natural" shift on the way in.  We don't
   10614              :      want to do this inside the loop as it makes it more difficult to
   10615              :      combine shifts.  */
   10616     25157211 :   if (SHIFT_COUNT_TRUNCATED)
   10617              :     orig_count &= GET_MODE_UNIT_BITSIZE (mode) - 1;
   10618              : 
   10619              :   /* If we were given an invalid count, don't do anything except exactly
   10620              :      what was requested.  */
   10621              : 
   10622     50314362 :   if (orig_count < 0 || orig_count >= (int) GET_MODE_UNIT_PRECISION (mode))
   10623              :     return NULL_RTX;
   10624              : 
   10625              :   count = orig_count;
   10626              : 
   10627              :   /* Unless one of the branches of the `if' in this loop does a `continue',
   10628              :      we will `break' the loop after the `if'.  */
   10629              : 
   10630     29502647 :   while (count != 0)
   10631              :     {
   10632              :       /* If we have an operand of (clobber (const_int 0)), fail.  */
   10633     25232036 :       if (GET_CODE (varop) == CLOBBER)
   10634     25157211 :         return NULL_RTX;
   10635              : 
   10636              :       /* Convert ROTATERT to ROTATE.  */
   10637     25232036 :       if (code == ROTATERT)
   10638              :         {
   10639        11602 :           unsigned int bitsize = GET_MODE_UNIT_PRECISION (result_mode);
   10640        11602 :           code = ROTATE;
   10641        11602 :           count = bitsize - count;
   10642              :         }
   10643              : 
   10644     25232036 :       shift_mode = result_mode;
   10645     25232036 :       if (shift_mode != mode)
   10646              :         {
   10647              :           /* We only change the modes of scalar shifts.  */
   10648       210035 :           int_mode = as_a <scalar_int_mode> (mode);
   10649       210035 :           int_result_mode = as_a <scalar_int_mode> (result_mode);
   10650       210035 :           shift_mode = try_widen_shift_mode (code, varop, count,
   10651              :                                              int_result_mode, int_mode,
   10652              :                                              outer_op, outer_const);
   10653              :         }
   10654              : 
   10655     25232036 :       scalar_int_mode shift_unit_mode;
   10656     50464072 :       if (!is_a <scalar_int_mode> (GET_MODE_INNER (shift_mode),
   10657              :                                    &shift_unit_mode))
   10658              :         return NULL_RTX;
   10659              : 
   10660              :       /* Handle cases where the count is greater than the size of the mode
   10661              :          minus 1.  For ASHIFT, use the size minus one as the count (this can
   10662              :          occur when simplifying (lshiftrt (ashiftrt ..))).  For rotates,
   10663              :          take the count modulo the size.  For other shifts, the result is
   10664              :          zero.
   10665              : 
   10666              :          Since these shifts are being produced by the compiler by combining
   10667              :          multiple operations, each of which are defined, we know what the
   10668              :          result is supposed to be.  */
   10669              : 
   10670     25232036 :       if (count > (GET_MODE_PRECISION (shift_unit_mode) - 1))
   10671              :         {
   10672        12740 :           if (code == ASHIFTRT)
   10673        12734 :             count = GET_MODE_PRECISION (shift_unit_mode) - 1;
   10674            6 :           else if (code == ROTATE || code == ROTATERT)
   10675            6 :             count %= GET_MODE_PRECISION (shift_unit_mode);
   10676              :           else
   10677              :             {
   10678              :               /* We can't simply return zero because there may be an
   10679              :                  outer op.  */
   10680            0 :               varop = const0_rtx;
   10681            0 :               count = 0;
   10682            0 :               break;
   10683              :             }
   10684              :         }
   10685              : 
   10686              :       /* If we discovered we had to complement VAROP, leave.  Making a NOT
   10687              :          here would cause an infinite loop.  */
   10688     25232036 :       if (complement_p)
   10689              :         break;
   10690              : 
   10691     25218572 :       if (shift_mode == shift_unit_mode)
   10692              :         {
   10693              :           /* An arithmetic right shift of a quantity known to be -1 or 0
   10694              :              is a no-op.  */
   10695     24540715 :           if (code == ASHIFTRT
   10696     24540715 :               && (num_sign_bit_copies (varop, shift_unit_mode)
   10697      4575152 :                   == GET_MODE_PRECISION (shift_unit_mode)))
   10698              :             {
   10699              :               count = 0;
   10700              :               break;
   10701              :             }
   10702              : 
   10703              :           /* If we are doing an arithmetic right shift and discarding all but
   10704              :              the sign bit copies, this is equivalent to doing a shift by the
   10705              :              bitsize minus one.  Convert it into that shift because it will
   10706              :              often allow other simplifications.  */
   10707              : 
   10708     24540643 :           if (code == ASHIFTRT
   10709     24540643 :               && (count + num_sign_bit_copies (varop, shift_unit_mode)
   10710      4575080 :                   >= GET_MODE_PRECISION (shift_unit_mode)))
   10711       345187 :             count = GET_MODE_PRECISION (shift_unit_mode) - 1;
   10712              : 
   10713              :           /* We simplify the tests below and elsewhere by converting
   10714              :              ASHIFTRT to LSHIFTRT if we know the sign bit is clear.
   10715              :              `make_compound_operation' will convert it to an ASHIFTRT for
   10716              :              those machines (such as VAX) that don't have an LSHIFTRT.  */
   10717     24540643 :           if (code == ASHIFTRT
   10718      4575080 :               && HWI_COMPUTABLE_MODE_P (shift_unit_mode)
   10719     29091803 :               && val_signbit_known_clear_p (shift_unit_mode,
   10720              :                                             nonzero_bits (varop,
   10721              :                                                           shift_unit_mode)))
   10722              :             code = LSHIFTRT;
   10723              : 
   10724     24511427 :           if (((code == LSHIFTRT
   10725      6224243 :                 && HWI_COMPUTABLE_MODE_P (shift_unit_mode)
   10726      6202143 :                 && !(nonzero_bits (varop, shift_unit_mode) >> count))
   10727     24538768 :                || (code == ASHIFT
   10728     13741720 :                    && HWI_COMPUTABLE_MODE_P (shift_unit_mode)
   10729     13259539 :                    && !((nonzero_bits (varop, shift_unit_mode) << count)
   10730     13259539 :                         & GET_MODE_MASK (shift_unit_mode))))
   10731     24515436 :               && !side_effects_p (varop))
   10732         4009 :             varop = const0_rtx;
   10733              :         }
   10734              : 
   10735     25218500 :       switch (GET_CODE (varop))
   10736              :         {
   10737       521123 :         case SIGN_EXTEND:
   10738       521123 :         case ZERO_EXTEND:
   10739       521123 :         case SIGN_EXTRACT:
   10740       521123 :         case ZERO_EXTRACT:
   10741       521123 :           new_rtx = expand_compound_operation (varop);
   10742       521123 :           if (new_rtx != varop)
   10743              :             {
   10744        68727 :               varop = new_rtx;
   10745      4345708 :               continue;
   10746              :             }
   10747              :           break;
   10748              : 
   10749       342849 :         case MEM:
   10750              :           /* The following rules apply only to scalars.  */
   10751       342849 :           if (shift_mode != shift_unit_mode)
   10752              :             break;
   10753       329591 :           int_mode = as_a <scalar_int_mode> (mode);
   10754              : 
   10755              :           /* If we have (xshiftrt (mem ...) C) and C is MODE_WIDTH
   10756              :              minus the width of a smaller mode, we can do this with a
   10757              :              SIGN_EXTEND or ZERO_EXTEND from the narrower memory location.  */
   10758       333208 :           if ((code == ASHIFTRT || code == LSHIFTRT)
   10759       125541 :               && ! mode_dependent_address_p (XEXP (varop, 0),
   10760       125541 :                                              MEM_ADDR_SPACE (varop))
   10761       125541 :               && ! MEM_VOLATILE_P (varop)
   10762       453229 :               && (int_mode_for_size (GET_MODE_BITSIZE (int_mode) - count, 1)
   10763       325974 :                   .exists (&tmode)))
   10764              :             {
   10765         3617 :               new_rtx = adjust_address_nv (varop, tmode,
   10766              :                                            BYTES_BIG_ENDIAN ? 0
   10767              :                                            : count / BITS_PER_UNIT);
   10768              : 
   10769         3617 :               varop = gen_rtx_fmt_e (code == ASHIFTRT ? SIGN_EXTEND
   10770              :                                      : ZERO_EXTEND, int_mode, new_rtx);
   10771         3617 :               count = 0;
   10772         3617 :               continue;
   10773              :             }
   10774              :           break;
   10775              : 
   10776      5300511 :         case SUBREG:
   10777              :           /* The following rules apply only to scalars.  */
   10778      5300511 :           if (shift_mode != shift_unit_mode)
   10779              :             break;
   10780      4850714 :           int_mode = as_a <scalar_int_mode> (mode);
   10781      4850714 :           int_varop_mode = as_a <scalar_int_mode> (GET_MODE (varop));
   10782              : 
   10783              :           /* If VAROP is a SUBREG, strip it as long as the inner operand has
   10784              :              the same number of words as what we've seen so far.  Then store
   10785              :              the widest mode in MODE.  */
   10786      4850714 :           if (subreg_lowpart_p (varop)
   10787     29960748 :               && is_int_mode (GET_MODE (SUBREG_REG (varop)), &inner_mode)
   10788      9607618 :               && GET_MODE_SIZE (inner_mode) > GET_MODE_SIZE (int_varop_mode)
   10789       230738 :               && (CEIL (GET_MODE_SIZE (inner_mode), UNITS_PER_WORD)
   10790       212177 :                   == CEIL (GET_MODE_SIZE (int_mode), UNITS_PER_WORD))
   10791      5049363 :               && GET_MODE_CLASS (int_varop_mode) == MODE_INT)
   10792              :             {
   10793       198649 :               varop = SUBREG_REG (varop);
   10794       595947 :               if (GET_MODE_SIZE (inner_mode) > GET_MODE_SIZE (int_mode))
   10795       198649 :                 mode = inner_mode;
   10796       198649 :               continue;
   10797              :             }
   10798              :           break;
   10799              : 
   10800       419104 :         case MULT:
   10801              :           /* Some machines use MULT instead of ASHIFT because MULT
   10802              :              is cheaper.  But it is still better on those machines to
   10803              :              merge two shifts into one.  */
   10804       419104 :           if (CONST_INT_P (XEXP (varop, 1))
   10805       419104 :               && (log2 = exact_log2 (UINTVAL (XEXP (varop, 1)))) >= 0)
   10806              :             {
   10807            0 :               rtx log2_rtx = gen_int_shift_amount (GET_MODE (varop), log2);
   10808            0 :               varop = simplify_gen_binary (ASHIFT, GET_MODE (varop),
   10809              :                                            XEXP (varop, 0), log2_rtx);
   10810            0 :               continue;
   10811            0 :             }
   10812              :           break;
   10813              : 
   10814         8813 :         case UDIV:
   10815              :           /* Similar, for when divides are cheaper.  */
   10816         8813 :           if (CONST_INT_P (XEXP (varop, 1))
   10817         8813 :               && (log2 = exact_log2 (UINTVAL (XEXP (varop, 1)))) >= 0)
   10818              :             {
   10819            9 :               rtx log2_rtx = gen_int_shift_amount (GET_MODE (varop), log2);
   10820            9 :               varop = simplify_gen_binary (LSHIFTRT, GET_MODE (varop),
   10821              :                                            XEXP (varop, 0), log2_rtx);
   10822            9 :               continue;
   10823            9 :             }
   10824              :           break;
   10825              : 
   10826       384239 :         case ASHIFTRT:
   10827              :           /* If we are extracting just the sign bit of an arithmetic
   10828              :              right shift, that shift is not needed.  However, the sign
   10829              :              bit of a wider mode may be different from what would be
   10830              :              interpreted as the sign bit in a narrower mode, so, if
   10831              :              the result is narrower, don't discard the shift.  */
   10832       386171 :           if (code == LSHIFTRT
   10833        16021 :               && count == (GET_MODE_UNIT_BITSIZE (result_mode) - 1)
   10834       384239 :               && (GET_MODE_UNIT_BITSIZE (result_mode)
   10835         3890 :                   >= GET_MODE_UNIT_BITSIZE (GET_MODE (varop))))
   10836              :             {
   10837         1932 :               varop = XEXP (varop, 0);
   10838         1932 :               continue;
   10839              :             }
   10840              : 
   10841              :           /* fall through */
   10842              : 
   10843      6319561 :         case LSHIFTRT:
   10844      6319561 :         case ASHIFT:
   10845      6319561 :         case ROTATE:
   10846              :           /* The following rules apply only to scalars.  */
   10847      6319561 :           if (shift_mode != shift_unit_mode)
   10848              :             break;
   10849      6311528 :           int_mode = as_a <scalar_int_mode> (mode);
   10850      6311528 :           int_varop_mode = as_a <scalar_int_mode> (GET_MODE (varop));
   10851      6311528 :           int_result_mode = as_a <scalar_int_mode> (result_mode);
   10852              : 
   10853              :           /* Here we have two nested shifts.  The result is usually the
   10854              :              AND of a new shift with a mask.  We compute the result below.  */
   10855      6311528 :           if (CONST_INT_P (XEXP (varop, 1))
   10856      6291318 :               && INTVAL (XEXP (varop, 1)) >= 0
   10857      6291315 :               && INTVAL (XEXP (varop, 1)) < GET_MODE_PRECISION (int_varop_mode)
   10858      6291315 :               && HWI_COMPUTABLE_MODE_P (int_result_mode)
   10859     12572073 :               && HWI_COMPUTABLE_MODE_P (int_mode))
   10860              :             {
   10861      6260545 :               enum rtx_code first_code = GET_CODE (varop);
   10862      6260545 :               unsigned int first_count = INTVAL (XEXP (varop, 1));
   10863      6260545 :               unsigned HOST_WIDE_INT mask;
   10864      6260545 :               rtx mask_rtx;
   10865              : 
   10866              :               /* We have one common special case.  We can't do any merging if
   10867              :                  the inner code is an ASHIFTRT of a smaller mode.  However, if
   10868              :                  we have (ashift:M1 (subreg:M1 (ashiftrt:M2 FOO C1) 0) C2)
   10869              :                  with C2 == GET_MODE_BITSIZE (M1) - GET_MODE_BITSIZE (M2),
   10870              :                  we can convert it to
   10871              :                  (ashiftrt:M1 (ashift:M1 (and:M1 (subreg:M1 FOO 0) C3) C2) C1).
   10872              :                  This simplifies certain SIGN_EXTEND operations.  */
   10873      6260545 :               if (code == ASHIFT && first_code == ASHIFTRT
   10874      6260545 :                   && count == (GET_MODE_PRECISION (int_result_mode)
   10875       345846 :                                - GET_MODE_PRECISION (int_varop_mode)))
   10876              :                 {
   10877              :                   /* C3 has the low-order C1 bits zero.  */
   10878              : 
   10879            0 :                   mask = GET_MODE_MASK (int_mode)
   10880            0 :                          & ~((HOST_WIDE_INT_1U << first_count) - 1);
   10881              : 
   10882            0 :                   varop = simplify_and_const_int (NULL_RTX, int_result_mode,
   10883              :                                                   XEXP (varop, 0), mask);
   10884            0 :                   varop = simplify_shift_const (NULL_RTX, ASHIFT,
   10885              :                                                 int_result_mode, varop, count);
   10886            0 :                   count = first_count;
   10887            0 :                   code = ASHIFTRT;
   10888            0 :                   continue;
   10889              :                 }
   10890              : 
   10891              :               /* If this was (ashiftrt (ashift foo C1) C2) and FOO has more
   10892              :                  than C1 high-order bits equal to the sign bit, we can convert
   10893              :                  this to either an ASHIFT or an ASHIFTRT depending on the
   10894              :                  two counts.
   10895              : 
   10896              :                  We cannot do this if VAROP's mode is not SHIFT_UNIT_MODE.  */
   10897              : 
   10898      6262045 :               if (code == ASHIFTRT && first_code == ASHIFT
   10899      3016799 :                   && int_varop_mode == shift_unit_mode
   10900      9270593 :                   && (num_sign_bit_copies (XEXP (varop, 0), shift_unit_mode)
   10901              :                       > first_count))
   10902              :                 {
   10903         1500 :                   varop = XEXP (varop, 0);
   10904         1500 :                   count -= first_count;
   10905         1500 :                   if (count < 0)
   10906              :                     {
   10907            4 :                       count = -count;
   10908            4 :                       code = ASHIFT;
   10909              :                     }
   10910              : 
   10911         1500 :                   continue;
   10912              :                 }
   10913              : 
   10914              :               /* There are some cases we can't do.  If CODE is ASHIFTRT,
   10915              :                  we can only do this if FIRST_CODE is also ASHIFTRT.
   10916              : 
   10917              :                  We can't do the case when CODE is ROTATE and FIRST_CODE is
   10918              :                  ASHIFTRT.
   10919              : 
   10920              :                  If the mode of this shift is not the mode of the outer shift,
   10921              :                  we can't do this if either shift is a right shift or ROTATE.
   10922              : 
   10923              :                  Finally, we can't do any of these if the mode is too wide
   10924              :                  unless the codes are the same.
   10925              : 
   10926              :                  Handle the case where the shift codes are the same
   10927              :                  first.  */
   10928              : 
   10929      6259045 :               if (code == first_code)
   10930              :                 {
   10931        31360 :                   if (int_varop_mode != int_result_mode
   10932        31360 :                       && (code == ASHIFTRT || code == LSHIFTRT
   10933          709 :                           || code == ROTATE))
   10934              :                     break;
   10935              : 
   10936        30699 :                   count += first_count;
   10937        30699 :                   varop = XEXP (varop, 0);
   10938        30699 :                   continue;
   10939              :                 }
   10940              : 
   10941      6227685 :               if (code == ASHIFTRT
   10942      3212338 :                   || (code == ROTATE && first_code == ASHIFTRT)
   10943      3212308 :                   || GET_MODE_PRECISION (int_mode) > HOST_BITS_PER_WIDE_INT
   10944      9439993 :                   || (int_varop_mode != int_result_mode
   10945        81670 :                       && (first_code == ASHIFTRT || first_code == LSHIFTRT
   10946        81670 :                           || first_code == ROTATE
   10947        26842 :                           || code == ROTATE)))
   10948              :                 break;
   10949              : 
   10950              :               /* To compute the mask to apply after the shift, shift the
   10951              :                  nonzero bits of the inner shift the same way the
   10952              :                  outer shift will.  */
   10953              : 
   10954      3157480 :               mask_rtx = gen_int_mode (nonzero_bits (varop, int_varop_mode),
   10955              :                                        int_result_mode);
   10956      3157480 :               rtx count_rtx = gen_int_shift_amount (int_result_mode, count);
   10957      3157480 :               mask_rtx
   10958      3157480 :                 = simplify_const_binary_operation (code, int_result_mode,
   10959              :                                                    mask_rtx, count_rtx);
   10960              : 
   10961              :               /* Give up if we can't compute an outer operation to use.  */
   10962      3157480 :               if (mask_rtx == 0
   10963      3157480 :                   || !CONST_INT_P (mask_rtx)
   10964      6314960 :                   || ! merge_outer_ops (&outer_op, &outer_const, AND,
   10965              :                                         INTVAL (mask_rtx),
   10966              :                                         int_result_mode, &complement_p))
   10967              :                 break;
   10968              : 
   10969              :               /* If the shifts are in the same direction, we add the
   10970              :                  counts.  Otherwise, we subtract them.  */
   10971      3131168 :               if ((code == ASHIFTRT || code == LSHIFTRT)
   10972      3131168 :                   == (first_code == ASHIFTRT || first_code == LSHIFTRT))
   10973        12084 :                 count += first_count;
   10974              :               else
   10975      3119084 :                 count -= first_count;
   10976              : 
   10977              :               /* If COUNT is positive, the new shift is usually CODE,
   10978              :                  except for the two exceptions below, in which case it is
   10979              :                  FIRST_CODE.  If the count is negative, FIRST_CODE should
   10980              :                  always be used  */
   10981      3131168 :               if (count > 0
   10982       715573 :                   && ((first_code == ROTATE && code == ASHIFT)
   10983       715067 :                       || (first_code == ASHIFTRT && code == LSHIFTRT)))
   10984              :                 code = first_code;
   10985      3119090 :               else if (count < 0)
   10986       324913 :                 code = first_code, count = -count;
   10987              : 
   10988      3131168 :               varop = XEXP (varop, 0);
   10989      3131168 :               continue;
   10990      3131168 :             }
   10991              : 
   10992              :           /* If we have (A << B << C) for any shift, we can convert this to
   10993              :              (A << C << B).  This wins if A is a constant.  Only try this if
   10994              :              B is not a constant.  */
   10995              : 
   10996        50983 :           else if (GET_CODE (varop) == code
   10997         5285 :                    && CONST_INT_P (XEXP (varop, 0))
   10998         1047 :                    && !CONST_INT_P (XEXP (varop, 1)))
   10999              :             {
   11000              :               /* For ((unsigned) (cstULL >> count)) >> cst2 we have to make
   11001              :                  sure the result will be masked.  See PR70222.  */
   11002         1047 :               if (code == LSHIFTRT
   11003            7 :                   && int_mode != int_result_mode
   11004         1054 :                   && !merge_outer_ops (&outer_op, &outer_const, AND,
   11005            7 :                                        GET_MODE_MASK (int_result_mode)
   11006            7 :                                        >> orig_count, int_result_mode,
   11007              :                                        &complement_p))
   11008              :                 break;
   11009              :               /* For ((int) (cstLL >> count)) >> cst2 just give up.  Queuing
   11010              :                  up outer sign extension (often left and right shift) is
   11011              :                  hardly more efficient than the original.  See PR70429.
   11012              :                  Similarly punt for rotates with different modes.
   11013              :                  See PR97386.  */
   11014         1047 :               if ((code == ASHIFTRT || code == ROTATE)
   11015         1047 :                   && int_mode != int_result_mode)
   11016              :                 break;
   11017              : 
   11018         1033 :               rtx count_rtx = gen_int_shift_amount (int_result_mode, count);
   11019         1033 :               rtx new_rtx = simplify_const_binary_operation (code, int_mode,
   11020              :                                                              XEXP (varop, 0),
   11021              :                                                              count_rtx);
   11022         1033 :               varop = gen_rtx_fmt_ee (code, int_mode, new_rtx, XEXP (varop, 1));
   11023         1033 :               count = 0;
   11024         1033 :               continue;
   11025         1033 :             }
   11026              :           break;
   11027              : 
   11028        59749 :         case NOT:
   11029              :           /* The following rules apply only to scalars.  */
   11030        59749 :           if (shift_mode != shift_unit_mode)
   11031              :             break;
   11032              : 
   11033              :           /* Make this fit the case below.  */
   11034        59732 :           varop = gen_rtx_XOR (mode, XEXP (varop, 0), constm1_rtx);
   11035        59732 :           continue;
   11036              : 
   11037       733461 :         case IOR:
   11038       733461 :         case AND:
   11039       733461 :         case XOR:
   11040              :           /* The following rules apply only to scalars.  */
   11041       733461 :           if (shift_mode != shift_unit_mode)
   11042              :             break;
   11043       729344 :           int_varop_mode = as_a <scalar_int_mode> (GET_MODE (varop));
   11044       729344 :           int_result_mode = as_a <scalar_int_mode> (result_mode);
   11045              : 
   11046              :           /* If we have (xshiftrt (ior (plus X (const_int -1)) X) C)
   11047              :              with C the size of VAROP - 1 and the shift is logical if
   11048              :              STORE_FLAG_VALUE is 1 and arithmetic if STORE_FLAG_VALUE is -1,
   11049              :              we have an (le X 0) operation.   If we have an arithmetic shift
   11050              :              and STORE_FLAG_VALUE is 1 or we have a logical shift with
   11051              :              STORE_FLAG_VALUE of -1, we have a (neg (le X 0)) operation.  */
   11052              : 
   11053       189162 :           if (GET_CODE (varop) == IOR && GET_CODE (XEXP (varop, 0)) == PLUS
   11054         1560 :               && XEXP (XEXP (varop, 0), 1) == constm1_rtx
   11055              :               && (STORE_FLAG_VALUE == 1 || STORE_FLAG_VALUE == -1)
   11056          140 :               && (code == LSHIFTRT || code == ASHIFTRT)
   11057          140 :               && count == (GET_MODE_PRECISION (int_varop_mode) - 1)
   11058       729484 :               && rtx_equal_p (XEXP (XEXP (varop, 0), 0), XEXP (varop, 1)))
   11059              :             {
   11060           56 :               count = 0;
   11061           56 :               varop = gen_rtx_LE (int_varop_mode, XEXP (varop, 1),
   11062              :                                   const0_rtx);
   11063              : 
   11064           56 :               if (STORE_FLAG_VALUE == 1 ? code == ASHIFTRT : code == LSHIFTRT)
   11065           56 :                 varop = gen_rtx_NEG (int_varop_mode, varop);
   11066              : 
   11067           56 :               continue;
   11068              :             }
   11069              : 
   11070              :           /* If we have (shift (logical)), move the logical to the outside
   11071              :              to allow it to possibly combine with another logical and the
   11072              :              shift to combine with another shift.  This also canonicalizes to
   11073              :              what a ZERO_EXTRACT looks like.  Also, some machines have
   11074              :              (and (shift)) insns.  */
   11075              : 
   11076      1162227 :           if (CONST_INT_P (XEXP (varop, 1))
   11077              :               /* We can't do this if we have (ashiftrt (xor))  and the
   11078              :                  constant has its sign bit set in shift_unit_mode with
   11079              :                  shift_unit_mode wider than result_mode.  */
   11080       434275 :               && !(code == ASHIFTRT && GET_CODE (varop) == XOR
   11081         8747 :                    && int_result_mode != shift_unit_mode
   11082            0 :                    && trunc_int_for_mode (INTVAL (XEXP (varop, 1)),
   11083              :                                           shift_unit_mode) < 0)
   11084       434275 :               && (new_rtx = simplify_const_binary_operation
   11085       434275 :                   (code, int_result_mode,
   11086       434275 :                    gen_int_mode (INTVAL (XEXP (varop, 1)), int_result_mode),
   11087       434275 :                    gen_int_shift_amount (int_result_mode, count))) != 0
   11088       434275 :               && CONST_INT_P (new_rtx)
   11089      1163563 :               && merge_outer_ops (&outer_op, &outer_const, GET_CODE (varop),
   11090              :                                   INTVAL (new_rtx), int_result_mode,
   11091              :                                   &complement_p))
   11092              :             {
   11093       432939 :               varop = XEXP (varop, 0);
   11094       432939 :               continue;
   11095              :             }
   11096              : 
   11097              :           /* If we can't do that, try to simplify the shift in each arm of the
   11098              :              logical expression, make a new logical expression, and apply
   11099              :              the inverse distributive law.  This also can't be done for
   11100              :              (ashiftrt (xor)) where we've widened the shift and the constant
   11101              :              changes the sign bit.  */
   11102       296349 :           if (CONST_INT_P (XEXP (varop, 1))
   11103       296349 :               && !(code == ASHIFTRT && GET_CODE (varop) == XOR
   11104           48 :                    && int_result_mode != shift_unit_mode
   11105            0 :                    && trunc_int_for_mode (INTVAL (XEXP (varop, 1)),
   11106              :                                           shift_unit_mode) < 0))
   11107              :             {
   11108         1336 :               rtx lhs = simplify_shift_const (NULL_RTX, code, shift_unit_mode,
   11109              :                                               XEXP (varop, 0), count);
   11110         1336 :               rtx rhs = simplify_shift_const (NULL_RTX, code, shift_unit_mode,
   11111              :                                               XEXP (varop, 1), count);
   11112              : 
   11113         1336 :               varop = simplify_gen_binary (GET_CODE (varop), shift_unit_mode,
   11114              :                                            lhs, rhs);
   11115         1336 :               varop = apply_distributive_law (varop);
   11116              : 
   11117         1336 :               count = 0;
   11118         1336 :               continue;
   11119         1336 :             }
   11120              :           break;
   11121              : 
   11122        34839 :         case EQ:
   11123              :           /* The following rules apply only to scalars.  */
   11124        34839 :           if (shift_mode != shift_unit_mode)
   11125              :             break;
   11126        34839 :           int_result_mode = as_a <scalar_int_mode> (result_mode);
   11127              : 
   11128              :           /* Convert (lshiftrt (eq FOO 0) C) to (xor FOO 1) if STORE_FLAG_VALUE
   11129              :              says that the sign bit can be tested, FOO has mode MODE, C is
   11130              :              GET_MODE_PRECISION (MODE) - 1, and FOO has only its low-order bit
   11131              :              that may be nonzero.  */
   11132        34839 :           if (code == LSHIFTRT
   11133            0 :               && XEXP (varop, 1) == const0_rtx
   11134            0 :               && GET_MODE (XEXP (varop, 0)) == int_result_mode
   11135            0 :               && count == (GET_MODE_PRECISION (int_result_mode) - 1)
   11136        34839 :               && HWI_COMPUTABLE_MODE_P (int_result_mode)
   11137              :               && STORE_FLAG_VALUE == -1
   11138              :               && nonzero_bits (XEXP (varop, 0), int_result_mode) == 1
   11139              :               && merge_outer_ops (&outer_op, &outer_const, XOR, 1,
   11140              :                                   int_result_mode, &complement_p))
   11141              :             {
   11142              :               varop = XEXP (varop, 0);
   11143              :               count = 0;
   11144              :               continue;
   11145              :             }
   11146              :           break;
   11147              : 
   11148        27539 :         case NEG:
   11149              :           /* The following rules apply only to scalars.  */
   11150        27539 :           if (shift_mode != shift_unit_mode)
   11151              :             break;
   11152        27409 :           int_result_mode = as_a <scalar_int_mode> (result_mode);
   11153              : 
   11154              :           /* (lshiftrt (neg A) C) where A is either 0 or 1 and C is one less
   11155              :              than the number of bits in the mode is equivalent to A.  */
   11156        27414 :           if (code == LSHIFTRT
   11157         5801 :               && count == (GET_MODE_PRECISION (int_result_mode) - 1)
   11158        30080 :               && nonzero_bits (XEXP (varop, 0), int_result_mode) == 1)
   11159              :             {
   11160            5 :               varop = XEXP (varop, 0);
   11161            5 :               count = 0;
   11162            5 :               continue;
   11163              :             }
   11164              : 
   11165              :           /* NEG commutes with ASHIFT since it is multiplication.  Move the
   11166              :              NEG outside to allow shifts to combine.  */
   11167        45590 :           if (code == ASHIFT
   11168        27404 :               && merge_outer_ops (&outer_op, &outer_const, NEG, 0,
   11169              :                                   int_result_mode, &complement_p))
   11170              :             {
   11171        18186 :               varop = XEXP (varop, 0);
   11172        18186 :               continue;
   11173              :             }
   11174              :           break;
   11175              : 
   11176      2034219 :         case PLUS:
   11177              :           /* The following rules apply only to scalars.  */
   11178      2034219 :           if (shift_mode != shift_unit_mode)
   11179              :             break;
   11180      1978982 :           int_result_mode = as_a <scalar_int_mode> (result_mode);
   11181              : 
   11182              :           /* (lshiftrt (plus A -1) C) where A is either 0 or 1 and C
   11183              :              is one less than the number of bits in the mode is
   11184              :              equivalent to (xor A 1).  */
   11185      1978982 :           if (code == LSHIFTRT
   11186       410318 :               && count == (GET_MODE_PRECISION (int_result_mode) - 1)
   11187        34320 :               && XEXP (varop, 1) == constm1_rtx
   11188        16692 :               && nonzero_bits (XEXP (varop, 0), int_result_mode) == 1
   11189      1978982 :               && merge_outer_ops (&outer_op, &outer_const, XOR, 1,
   11190              :                                   int_result_mode, &complement_p))
   11191              :             {
   11192            0 :               count = 0;
   11193            0 :               varop = XEXP (varop, 0);
   11194            0 :               continue;
   11195              :             }
   11196              : 
   11197              :           /* If we have (xshiftrt (plus FOO BAR) C), and the only bits
   11198              :              that might be nonzero in BAR are those being shifted out and those
   11199              :              bits are known zero in FOO, we can replace the PLUS with FOO.
   11200              :              Similarly in the other operand order.  This code occurs when
   11201              :              we are computing the size of a variable-size array.  */
   11202              : 
   11203      1982344 :           if ((code == ASHIFTRT || code == LSHIFTRT)
   11204       581785 :               && count < HOST_BITS_PER_WIDE_INT
   11205       580616 :               && nonzero_bits (XEXP (varop, 1), int_result_mode) >> count == 0
   11206      2159187 :               && (nonzero_bits (XEXP (varop, 1), int_result_mode)
   11207       180205 :                   & nonzero_bits (XEXP (varop, 0), int_result_mode)) == 0)
   11208              :             {
   11209         3362 :               varop = XEXP (varop, 0);
   11210         3362 :               continue;
   11211              :             }
   11212      1975652 :           else if ((code == ASHIFTRT || code == LSHIFTRT)
   11213       578423 :                    && count < HOST_BITS_PER_WIDE_INT
   11214       577254 :                    && HWI_COMPUTABLE_MODE_P (int_result_mode)
   11215       576016 :                    && (nonzero_bits (XEXP (varop, 0), int_result_mode)
   11216       576016 :                        >> count) == 0
   11217      2070854 :                    && (nonzero_bits (XEXP (varop, 0), int_result_mode)
   11218        95234 :                        & nonzero_bits (XEXP (varop, 1), int_result_mode)) == 0)
   11219              :             {
   11220           32 :               varop = XEXP (varop, 1);
   11221           32 :               continue;
   11222              :             }
   11223              : 
   11224              :           /* (ashift (plus foo C) N) is (plus (ashift foo N) C').  */
   11225      2359188 :           if (code == ASHIFT
   11226      1388367 :               && CONST_INT_P (XEXP (varop, 1))
   11227       383755 :               && (new_rtx = simplify_const_binary_operation
   11228       383755 :                   (ASHIFT, int_result_mode,
   11229       383755 :                    gen_int_mode (INTVAL (XEXP (varop, 1)), int_result_mode),
   11230       383755 :                    gen_int_shift_amount (int_result_mode, count))) != 0
   11231       383755 :               && CONST_INT_P (new_rtx)
   11232      2359343 :               && merge_outer_ops (&outer_op, &outer_const, PLUS,
   11233              :                                   INTVAL (new_rtx), int_result_mode,
   11234              :                                   &complement_p))
   11235              :             {
   11236       383600 :               varop = XEXP (varop, 0);
   11237       383600 :               continue;
   11238              :             }
   11239              : 
   11240              :           /* Check for 'PLUS signbit', which is the canonical form of 'XOR
   11241              :              signbit', and attempt to change the PLUS to an XOR and move it to
   11242              :              the outer operation as is done above in the AND/IOR/XOR case
   11243              :              leg for shift(logical). See details in logical handling above
   11244              :              for reasoning in doing so.  */
   11245      1600589 :           if (code == LSHIFTRT
   11246       407026 :               && CONST_INT_P (XEXP (varop, 1))
   11247       285384 :               && mode_signbit_p (int_result_mode, XEXP (varop, 1))
   11248         8601 :               && (new_rtx = simplify_const_binary_operation
   11249      1591988 :                   (code, int_result_mode,
   11250         8601 :                    gen_int_mode (INTVAL (XEXP (varop, 1)), int_result_mode),
   11251         8601 :                    gen_int_shift_amount (int_result_mode, count))) != 0
   11252         8601 :               && CONST_INT_P (new_rtx)
   11253      1600589 :               && merge_outer_ops (&outer_op, &outer_const, XOR,
   11254              :                                   INTVAL (new_rtx), int_result_mode,
   11255              :                                   &complement_p))
   11256              :             {
   11257         8601 :               varop = XEXP (varop, 0);
   11258         8601 :               continue;
   11259              :             }
   11260              : 
   11261              :           break;
   11262              : 
   11263       633404 :         case MINUS:
   11264              :           /* The following rules apply only to scalars.  */
   11265       633404 :           if (shift_mode != shift_unit_mode)
   11266              :             break;
   11267       614162 :           int_varop_mode = as_a <scalar_int_mode> (GET_MODE (varop));
   11268              : 
   11269              :           /* If we have (xshiftrt (minus (ashiftrt X C)) X) C)
   11270              :              with C the size of VAROP - 1 and the shift is logical if
   11271              :              STORE_FLAG_VALUE is 1 and arithmetic if STORE_FLAG_VALUE is -1,
   11272              :              we have a (gt X 0) operation.  If the shift is arithmetic with
   11273              :              STORE_FLAG_VALUE of 1 or logical with STORE_FLAG_VALUE == -1,
   11274              :              we have a (neg (gt X 0)) operation.  */
   11275              : 
   11276       614162 :           if ((STORE_FLAG_VALUE == 1 || STORE_FLAG_VALUE == -1)
   11277       614162 :               && GET_CODE (XEXP (varop, 0)) == ASHIFTRT
   11278        12285 :               && count == (GET_MODE_PRECISION (int_varop_mode) - 1)
   11279           48 :               && (code == LSHIFTRT || code == ASHIFTRT)
   11280           13 :               && CONST_INT_P (XEXP (XEXP (varop, 0), 1))
   11281           13 :               && INTVAL (XEXP (XEXP (varop, 0), 1)) == count
   11282       614162 :               && rtx_equal_p (XEXP (XEXP (varop, 0), 0), XEXP (varop, 1)))
   11283              :             {
   11284            0 :               count = 0;
   11285            0 :               varop = gen_rtx_GT (int_varop_mode, XEXP (varop, 1),
   11286              :                                   const0_rtx);
   11287              : 
   11288            0 :               if (STORE_FLAG_VALUE == 1 ? code == ASHIFTRT : code == LSHIFTRT)
   11289            0 :                 varop = gen_rtx_NEG (int_varop_mode, varop);
   11290              : 
   11291            0 :               continue;
   11292              :             }
   11293              :           break;
   11294              : 
   11295          683 :         case TRUNCATE:
   11296              :           /* Change (lshiftrt (truncate (lshiftrt))) to (truncate (lshiftrt))
   11297              :              if the truncate does not affect the value.  */
   11298          683 :           if (code == LSHIFTRT
   11299          525 :               && GET_CODE (XEXP (varop, 0)) == LSHIFTRT
   11300          525 :               && CONST_INT_P (XEXP (XEXP (varop, 0), 1))
   11301          683 :               && (INTVAL (XEXP (XEXP (varop, 0), 1))
   11302          525 :                   >= (GET_MODE_UNIT_PRECISION (GET_MODE (XEXP (varop, 0)))
   11303         1050 :                       - GET_MODE_UNIT_PRECISION (GET_MODE (varop)))))
   11304              :             {
   11305          525 :               rtx varop_inner = XEXP (varop, 0);
   11306          525 :               int new_count = count + INTVAL (XEXP (varop_inner, 1));
   11307          525 :               rtx new_count_rtx = gen_int_shift_amount (GET_MODE (varop_inner),
   11308          525 :                                                         new_count);
   11309          525 :               varop_inner = gen_rtx_LSHIFTRT (GET_MODE (varop_inner),
   11310              :                                               XEXP (varop_inner, 0),
   11311              :                                               new_count_rtx);
   11312          525 :               varop = gen_rtx_TRUNCATE (GET_MODE (varop), varop_inner);
   11313          525 :               count = 0;
   11314          525 :               continue;
   11315          525 :             }
   11316              :           break;
   11317              : 
   11318              :         default:
   11319              :           break;
   11320        59732 :         }
   11321              : 
   11322              :       break;
   11323              :     }
   11324              : 
   11325     25156939 :   shift_mode = result_mode;
   11326     25156939 :   if (shift_mode != mode)
   11327              :     {
   11328              :       /* We only change the modes of scalar shifts.  */
   11329       198079 :       int_mode = as_a <scalar_int_mode> (mode);
   11330       198079 :       int_result_mode = as_a <scalar_int_mode> (result_mode);
   11331       198079 :       shift_mode = try_widen_shift_mode (code, varop, count, int_result_mode,
   11332              :                                          int_mode, outer_op, outer_const);
   11333              :     }
   11334              : 
   11335              :   /* We have now finished analyzing the shift.  The result should be
   11336              :      a shift of type CODE with SHIFT_MODE shifting VAROP COUNT places.  If
   11337              :      OUTER_OP is non-UNKNOWN, it is an operation that needs to be applied
   11338              :      to the result of the shift.  OUTER_CONST is the relevant constant,
   11339              :      but we must turn off all bits turned off in the shift.  */
   11340              : 
   11341     25156939 :   if (outer_op == UNKNOWN
   11342     21250298 :       && orig_code == code && orig_count == count
   11343     21202073 :       && varop == orig_varop
   11344     21030652 :       && shift_mode == GET_MODE (varop))
   11345              :     return NULL_RTX;
   11346              : 
   11347              :   /* Make a SUBREG if necessary.  If we can't make it, fail.  */
   11348      4128938 :   varop = gen_lowpart (shift_mode, varop);
   11349      4128938 :   if (varop == NULL_RTX || GET_CODE (varop) == CLOBBER)
   11350              :     return NULL_RTX;
   11351              : 
   11352              :   /* If we have an outer operation and we just made a shift, it is
   11353              :      possible that we could have simplified the shift were it not
   11354              :      for the outer operation.  So try to do the simplification
   11355              :      recursively.  */
   11356              : 
   11357      4128938 :   if (outer_op != UNKNOWN)
   11358      3906641 :     x = simplify_shift_const_1 (code, shift_mode, varop, count);
   11359              :   else
   11360              :     x = NULL_RTX;
   11361              : 
   11362      3906641 :   if (x == NULL_RTX)
   11363      4093214 :     x = simplify_gen_binary (code, shift_mode, varop,
   11364      4093214 :                              gen_int_shift_amount (shift_mode, count));
   11365              : 
   11366              :   /* If we were doing an LSHIFTRT in a wider mode than it was originally,
   11367              :      turn off all the bits that the shift would have turned off.  */
   11368      4128938 :   if (orig_code == LSHIFTRT && result_mode != shift_mode)
   11369              :     /* We only change the modes of scalar shifts.  */
   11370        27661 :     x = simplify_and_const_int (NULL_RTX, as_a <scalar_int_mode> (shift_mode),
   11371        27661 :                                 x, GET_MODE_MASK (result_mode) >> orig_count);
   11372              : 
   11373              :   /* Do the remainder of the processing in RESULT_MODE.  */
   11374      4128938 :   x = gen_lowpart_or_truncate (result_mode, x);
   11375              : 
   11376              :   /* If COMPLEMENT_P is set, we have to complement X before doing the outer
   11377              :      operation.  */
   11378      4128938 :   if (complement_p)
   11379        24499 :     x = simplify_gen_unary (NOT, result_mode, x, result_mode);
   11380              : 
   11381      4128938 :   if (outer_op != UNKNOWN)
   11382              :     {
   11383      3906641 :       int_result_mode = as_a <scalar_int_mode> (result_mode);
   11384              : 
   11385      3906641 :       if (GET_RTX_CLASS (outer_op) != RTX_UNARY
   11386      3906641 :           && GET_MODE_PRECISION (int_result_mode) < HOST_BITS_PER_WIDE_INT)
   11387      1386796 :         outer_const = trunc_int_for_mode (outer_const, int_result_mode);
   11388              : 
   11389      3906641 :       if (outer_op == AND)
   11390      3437427 :         x = simplify_and_const_int (NULL_RTX, int_result_mode, x, outer_const);
   11391       469214 :       else if (outer_op == SET)
   11392              :         {
   11393              :           /* This means that we have determined that the result is
   11394              :              equivalent to a constant.  This should be rare.  */
   11395            0 :           if (!side_effects_p (x))
   11396            0 :             x = GEN_INT (outer_const);
   11397              :         }
   11398       469214 :       else if (GET_RTX_CLASS (outer_op) == RTX_UNARY)
   11399        18186 :         x = simplify_gen_unary (outer_op, int_result_mode, x, int_result_mode);
   11400              :       else
   11401       451028 :         x = simplify_gen_binary (outer_op, int_result_mode, x,
   11402              :                                  GEN_INT (outer_const));
   11403              :     }
   11404              : 
   11405              :   return x;
   11406              : }
   11407              : 
   11408              : /* Simplify a shift of VAROP by COUNT bits.  CODE says what kind of shift.
   11409              :    The result of the shift is RESULT_MODE.  If we cannot simplify it,
   11410              :    return X or, if it is NULL, synthesize the expression with
   11411              :    simplify_gen_binary.  Otherwise, return a simplified value.
   11412              : 
   11413              :    The shift is normally computed in the widest mode we find in VAROP, as
   11414              :    long as it isn't a different number of words than RESULT_MODE.  Exceptions
   11415              :    are ASHIFTRT and ROTATE, which are always done in their original mode.  */
   11416              : 
   11417              : static rtx
   11418     21250570 : simplify_shift_const (rtx x, enum rtx_code code, machine_mode result_mode,
   11419              :                       rtx varop, int count)
   11420              : {
   11421     21250570 :   rtx tem = simplify_shift_const_1 (code, result_mode, varop, count);
   11422     21250570 :   if (tem)
   11423              :     return tem;
   11424              : 
   11425     17157356 :   if (!x)
   11426      5374561 :     x = simplify_gen_binary (code, GET_MODE (varop), varop,
   11427      5374561 :                              gen_int_shift_amount (GET_MODE (varop), count));
   11428     17157356 :   if (GET_MODE (x) != result_mode)
   11429            0 :     x = gen_lowpart (result_mode, x);
   11430              :   return x;
   11431              : }
   11432              : 
   11433              : 
   11434              : /* A subroutine of recog_for_combine.  See there for arguments and
   11435              :    return value.  */
   11436              : 
   11437              : static int
   11438     49984089 : recog_for_combine_1 (rtx *pnewpat, rtx_insn *insn, rtx *pnotes,
   11439              :                      unsigned old_nregs, unsigned new_nregs)
   11440              : {
   11441     49984089 :   rtx pat = *pnewpat;
   11442     49984089 :   rtx pat_without_clobbers;
   11443     49984089 :   int insn_code_number;
   11444     49984089 :   int num_clobbers_to_add = 0;
   11445     49984089 :   int i;
   11446     49984089 :   rtx notes = NULL_RTX;
   11447     49984089 :   rtx old_notes, old_pat;
   11448     49984089 :   int old_icode;
   11449              : 
   11450              :   /* If PAT is a PARALLEL, check to see if it contains the CLOBBER
   11451              :      we use to indicate that something didn't match.  If we find such a
   11452              :      thing, force rejection.  */
   11453     49984089 :   if (GET_CODE (pat) == PARALLEL)
   11454     54176540 :     for (i = XVECLEN (pat, 0) - 1; i >= 0; i--)
   11455     37365264 :       if (GET_CODE (XVECEXP (pat, 0, i)) == CLOBBER
   11456      7604794 :           && XEXP (XVECEXP (pat, 0, i), 0) == const0_rtx)
   11457              :         return -1;
   11458              : 
   11459     49982089 :   old_pat = PATTERN (insn);
   11460     49982089 :   old_notes = REG_NOTES (insn);
   11461     49982089 :   PATTERN (insn) = pat;
   11462     49982089 :   REG_NOTES (insn) = NULL_RTX;
   11463              : 
   11464     49982089 :   insn_code_number = recog (pat, insn, &num_clobbers_to_add);
   11465     49982089 :   if (dump_file && (dump_flags & TDF_DETAILS))
   11466              :     {
   11467          277 :       if (insn_code_number < 0)
   11468          177 :         fputs ("Failed to match this instruction:\n", dump_file);
   11469              :       else
   11470          100 :         fputs ("Successfully matched this instruction:\n", dump_file);
   11471          277 :       print_rtl_single (dump_file, pat);
   11472              :     }
   11473              : 
   11474              :   /* If it isn't, there is the possibility that we previously had an insn
   11475              :      that clobbered some register as a side effect, but the combined
   11476              :      insn doesn't need to do that.  So try once more without the clobbers
   11477              :      unless this represents an ASM insn.  */
   11478              : 
   11479     39834289 :   if (insn_code_number < 0 && ! check_asm_operands (pat)
   11480     89814001 :       && GET_CODE (pat) == PARALLEL)
   11481              :     {
   11482              :       int pos;
   11483              : 
   11484     52651885 :       for (pos = 0, i = 0; i < XVECLEN (pat, 0); i++)
   11485     36327474 :         if (GET_CODE (XVECEXP (pat, 0, i)) != CLOBBER)
   11486              :           {
   11487     29141869 :             if (i != pos)
   11488      2455911 :               SUBST (XVECEXP (pat, 0, pos), XVECEXP (pat, 0, i));
   11489     29141869 :             pos++;
   11490              :           }
   11491              : 
   11492     16324411 :       SUBST_INT (XVECLEN (pat, 0), pos);
   11493              : 
   11494     16324411 :       if (pos == 1)
   11495      4806164 :         pat = XVECEXP (pat, 0, 0);
   11496              : 
   11497     16324411 :       PATTERN (insn) = pat;
   11498     16324411 :       insn_code_number = recog (pat, insn, &num_clobbers_to_add);
   11499     16324411 :       if (dump_file && (dump_flags & TDF_DETAILS))
   11500              :         {
   11501           82 :           if (insn_code_number < 0)
   11502           81 :             fputs ("Failed to match this instruction:\n", dump_file);
   11503              :           else
   11504            1 :             fputs ("Successfully matched this instruction:\n", dump_file);
   11505           82 :           print_rtl_single (dump_file, pat);
   11506              :         }
   11507              :     }
   11508              : 
   11509     49982089 :   pat_without_clobbers = pat;
   11510              : 
   11511     49982089 :   PATTERN (insn) = old_pat;
   11512     49982089 :   REG_NOTES (insn) = old_notes;
   11513              : 
   11514              :   /* Recognize all noop sets, these will be killed by followup pass.  */
   11515     49982089 :   if (insn_code_number < 0 && GET_CODE (pat) == SET && set_noop_p (pat))
   11516       243445 :     insn_code_number = NOOP_MOVE_INSN_CODE, num_clobbers_to_add = 0;
   11517              : 
   11518              :   /* If we had any clobbers to add, make a new pattern than contains
   11519              :      them.  Then check to make sure that all of them are dead.  */
   11520     49982089 :   if (num_clobbers_to_add)
   11521              :     {
   11522      1669155 :       rtx newpat = gen_rtx_PARALLEL (VOIDmode,
   11523              :                                      rtvec_alloc (GET_CODE (pat) == PARALLEL
   11524              :                                                   ? (XVECLEN (pat, 0)
   11525              :                                                      + num_clobbers_to_add)
   11526              :                                                   : num_clobbers_to_add + 1));
   11527              : 
   11528      1669155 :       if (GET_CODE (pat) == PARALLEL)
   11529         1455 :         for (i = 0; i < XVECLEN (pat, 0); i++)
   11530          970 :           XVECEXP (newpat, 0, i) = XVECEXP (pat, 0, i);
   11531              :       else
   11532      1668670 :         XVECEXP (newpat, 0, 0) = pat;
   11533              : 
   11534      1669155 :       add_clobbers (newpat, insn_code_number);
   11535              : 
   11536      3213577 :       for (i = XVECLEN (newpat, 0) - num_clobbers_to_add;
   11537      3213577 :            i < XVECLEN (newpat, 0); i++)
   11538              :         {
   11539      1691203 :           if (REG_P (XEXP (XVECEXP (newpat, 0, i), 0))
   11540      1691203 :               && ! reg_dead_at_p (XEXP (XVECEXP (newpat, 0, i), 0), insn))
   11541              :             return -1;
   11542      1544422 :           if (GET_CODE (XEXP (XVECEXP (newpat, 0, i), 0)) != SCRATCH)
   11543              :             {
   11544      1494364 :               gcc_assert (REG_P (XEXP (XVECEXP (newpat, 0, i), 0)));
   11545      1494364 :               notes = alloc_reg_note (REG_UNUSED,
   11546              :                                       XEXP (XVECEXP (newpat, 0, i), 0), notes);
   11547              :             }
   11548              :         }
   11549              :       pat = newpat;
   11550              :     }
   11551              : 
   11552     49835308 :   if (insn_code_number >= 0
   11553     49835308 :       && insn_code_number != NOOP_MOVE_INSN_CODE)
   11554              :     {
   11555              :       /* Create the reg dead notes if needed for the regs that were created via split.   */
   11556     10324735 :       for (; old_nregs < new_nregs; old_nregs++)
   11557         2255 :         notes = alloc_reg_note (REG_DEAD, regno_reg_rtx[old_nregs], notes);
   11558     10322480 :       old_pat = PATTERN (insn);
   11559     10322480 :       old_notes = REG_NOTES (insn);
   11560     10322480 :       old_icode = INSN_CODE (insn);
   11561     10322480 :       PATTERN (insn) = pat;
   11562     10322480 :       REG_NOTES (insn) = notes;
   11563     10322480 :       INSN_CODE (insn) = insn_code_number;
   11564              : 
   11565              :       /* Do not accept an insn if hard register constraints are used.  For
   11566              :          example, assume that the first insn is combined into the last one:
   11567              : 
   11568              :          r100=...
   11569              :          %5=...
   11570              :          r101=exp(r100)
   11571              : 
   11572              :          If the resulting insn has an operand which is constrained to hard
   11573              :          register %5, then this introduces a conflict since register %5 is live
   11574              :          at this point.  Therefore, skip for now.  This is a sledge hammer
   11575              :          approach.  Ideally we would skip based on the fact whether a
   11576              :          combination crosses a hard register assignment and the corresponding
   11577              :          hard register is also referred by a single register constraint of the
   11578              :          resulting insn.  */
   11579     10322480 :       bool has_hard_reg_cstr = false;
   11580     10322480 :       extract_insn (insn);
   11581     35203817 :       for (int nop = recog_data.n_operands - 1; nop >= 0; --nop)
   11582     24881337 :         if (strchr (recog_data.constraints[nop], '{'))
   11583              :           {
   11584              :             has_hard_reg_cstr = true;
   11585              :             break;
   11586              :           }
   11587              : 
   11588              :       /* Don't accept hard register constraints.  Allow targets to reject
   11589              :          combined insn.  */
   11590     10322480 :       if (has_hard_reg_cstr || !targetm.legitimate_combined_insn (insn))
   11591              :         {
   11592         3667 :           if (dump_file && (dump_flags & TDF_DETAILS))
   11593              :             {
   11594            0 :               if (has_hard_reg_cstr)
   11595            0 :                 fputs ("Instruction makes use of hard register constraints.",
   11596              :                        dump_file);
   11597              :               else
   11598            0 :                 fputs ("Instruction not appropriate for target.",
   11599              :                        dump_file);
   11600              :             }
   11601              : 
   11602              :           /* Callers expect recog_for_combine to strip
   11603              :              clobbers from the pattern on failure.  */
   11604              :           pat = pat_without_clobbers;
   11605              :           notes = NULL_RTX;
   11606              : 
   11607              :           insn_code_number = -1;
   11608              :         }
   11609              : 
   11610     10322480 :       PATTERN (insn) = old_pat;
   11611     10322480 :       REG_NOTES (insn) = old_notes;
   11612     10322480 :       INSN_CODE (insn) = old_icode;
   11613              :     }
   11614              : 
   11615     49835308 :   *pnewpat = pat;
   11616     49835308 :   *pnotes = notes;
   11617              : 
   11618     49835308 :   return insn_code_number;
   11619              : }
   11620              : 
   11621              : /* Change every ZERO_EXTRACT and ZERO_EXTEND of a SUBREG that can be
   11622              :    expressed as an AND and maybe an LSHIFTRT, to that formulation.
   11623              :    Return whether anything was so changed.  */
   11624              : 
   11625              : static bool
   11626     50139840 : change_zero_ext (rtx pat)
   11627              : {
   11628     50139840 :   bool changed = false;
   11629     50139840 :   rtx *src = &SET_SRC (pat);
   11630              : 
   11631     50139840 :   subrtx_ptr_iterator::array_type array;
   11632    349850049 :   FOR_EACH_SUBRTX_PTR (iter, array, src, NONCONST)
   11633              :     {
   11634    299710209 :       rtx x = **iter;
   11635    299710209 :       scalar_int_mode mode, inner_mode;
   11636    299710209 :       if (!is_a <scalar_int_mode> (GET_MODE (x), &mode))
   11637    299710209 :         continue;
   11638    157824002 :       int size;
   11639              : 
   11640    157824002 :       if (GET_CODE (x) == ZERO_EXTRACT
   11641       812530 :           && CONST_INT_P (XEXP (x, 1))
   11642       812516 :           && CONST_INT_P (XEXP (x, 2))
   11643       769999 :           && is_a <scalar_int_mode> (GET_MODE (XEXP (x, 0)), &inner_mode)
   11644    158593997 :           && GET_MODE_PRECISION (inner_mode) <= GET_MODE_PRECISION (mode))
   11645              :         {
   11646       769977 :           size = INTVAL (XEXP (x, 1));
   11647              : 
   11648       769977 :           int start = INTVAL (XEXP (x, 2));
   11649       769977 :           if (BITS_BIG_ENDIAN)
   11650              :             start = GET_MODE_PRECISION (inner_mode) - size - start;
   11651              : 
   11652       769977 :           if (start != 0)
   11653       648785 :             x = gen_rtx_LSHIFTRT (inner_mode, XEXP (x, 0),
   11654              :                                   gen_int_shift_amount (inner_mode, start));
   11655              :           else
   11656              :             x = XEXP (x, 0);
   11657              : 
   11658       769977 :           if (mode != inner_mode)
   11659              :             {
   11660          154 :               if (REG_P (x) && HARD_REGISTER_P (x)
   11661       212061 :                   && !can_change_dest_mode (x, 0, mode))
   11662            0 :                 continue;
   11663              : 
   11664       212061 :               x = gen_lowpart_SUBREG (mode, x);
   11665              :             }
   11666              :         }
   11667    157054025 :       else if (GET_CODE (x) == ZERO_EXTEND
   11668      2348991 :                && GET_CODE (XEXP (x, 0)) == SUBREG
   11669       445695 :                && SCALAR_INT_MODE_P (GET_MODE (SUBREG_REG (XEXP (x, 0))))
   11670       437992 :                && !paradoxical_subreg_p (XEXP (x, 0))
   11671    157492017 :                && subreg_lowpart_p (XEXP (x, 0)))
   11672              :         {
   11673       308662 :           inner_mode = as_a <scalar_int_mode> (GET_MODE (XEXP (x, 0)));
   11674       308662 :           size = GET_MODE_PRECISION (inner_mode);
   11675       308662 :           x = SUBREG_REG (XEXP (x, 0));
   11676       308662 :           if (GET_MODE (x) != mode)
   11677              :             {
   11678        18252 :               if (REG_P (x) && HARD_REGISTER_P (x)
   11679        21144 :                   && !can_change_dest_mode (x, 0, mode))
   11680            0 :                 continue;
   11681              : 
   11682        21144 :               x = gen_lowpart_SUBREG (mode, x);
   11683              :             }
   11684              :         }
   11685    313490653 :       else if (GET_CODE (x) == ZERO_EXTEND
   11686      2040329 :                && REG_P (XEXP (x, 0))
   11687      1041747 :                && HARD_REGISTER_P (XEXP (x, 0))
   11688    156745436 :                && can_change_dest_mode (XEXP (x, 0), 0, mode))
   11689              :         {
   11690           73 :           inner_mode = as_a <scalar_int_mode> (GET_MODE (XEXP (x, 0)));
   11691           73 :           size = GET_MODE_PRECISION (inner_mode);
   11692           73 :           x = gen_rtx_REG (mode, REGNO (XEXP (x, 0)));
   11693              :         }
   11694              :       else
   11695    156745290 :         continue;
   11696              : 
   11697      1534003 :       if (!(GET_CODE (x) == LSHIFTRT
   11698       455291 :             && CONST_INT_P (XEXP (x, 1))
   11699       455291 :             && size + INTVAL (XEXP (x, 1)) == GET_MODE_PRECISION (mode)))
   11700              :         {
   11701       891844 :           wide_int mask = wi::mask (size, false, GET_MODE_PRECISION (mode));
   11702       891844 :           x = gen_rtx_AND (mode, x, immed_wide_int_const (mask, mode));
   11703       891844 :         }
   11704              : 
   11705      1078712 :       SUBST (**iter, x);
   11706      1078712 :       changed = true;
   11707              :     }
   11708              : 
   11709     50139840 :   if (changed)
   11710      9965524 :     FOR_EACH_SUBRTX_PTR (iter, array, src, NONCONST)
   11711      8904214 :       maybe_swap_commutative_operands (**iter);
   11712              : 
   11713     50139840 :   rtx *dst = &SET_DEST (pat);
   11714     50139840 :   scalar_int_mode mode;
   11715     50139840 :   if (GET_CODE (*dst) == ZERO_EXTRACT
   11716         9086 :       && REG_P (XEXP (*dst, 0))
   11717          390 :       && is_a <scalar_int_mode> (GET_MODE (XEXP (*dst, 0)), &mode)
   11718          390 :       && CONST_INT_P (XEXP (*dst, 1))
   11719     50140230 :       && CONST_INT_P (XEXP (*dst, 2)))
   11720              :     {
   11721          291 :       rtx reg = XEXP (*dst, 0);
   11722          291 :       int width = INTVAL (XEXP (*dst, 1));
   11723          291 :       int offset = INTVAL (XEXP (*dst, 2));
   11724          291 :       int reg_width = GET_MODE_PRECISION (mode);
   11725          291 :       if (BITS_BIG_ENDIAN)
   11726              :         offset = reg_width - width - offset;
   11727              : 
   11728          291 :       rtx x, y, z, w;
   11729          291 :       wide_int mask = wi::shifted_mask (offset, width, true, reg_width);
   11730          291 :       wide_int mask2 = wi::shifted_mask (offset, width, false, reg_width);
   11731          291 :       x = gen_rtx_AND (mode, reg, immed_wide_int_const (mask, mode));
   11732          291 :       if (offset)
   11733          235 :         y = gen_rtx_ASHIFT (mode, SET_SRC (pat), GEN_INT (offset));
   11734              :       else
   11735           56 :         y = SET_SRC (pat);
   11736          291 :       z = gen_rtx_AND (mode, y, immed_wide_int_const (mask2, mode));
   11737          291 :       w = gen_rtx_IOR (mode, x, z);
   11738          291 :       SUBST (SET_DEST (pat), reg);
   11739          291 :       SUBST (SET_SRC (pat), w);
   11740              : 
   11741          291 :       changed = true;
   11742          291 :     }
   11743              : 
   11744     50139840 :   return changed;
   11745     50139840 : }
   11746              : 
   11747              : /* Like recog, but we receive the address of a pointer to a new pattern.
   11748              :    We try to match the rtx that the pointer points to.
   11749              :    If that fails, we may try to modify or replace the pattern,
   11750              :    storing the replacement into the same pointer object.
   11751              : 
   11752              :    Modifications include deletion or addition of CLOBBERs.  If the
   11753              :    instruction will still not match, we change ZERO_EXTEND and ZERO_EXTRACT
   11754              :    to the equivalent AND and perhaps LSHIFTRT patterns, and try with that
   11755              :    (and undo if that fails).
   11756              : 
   11757              :    PNOTES is a pointer to a location where any REG_UNUSED notes added for
   11758              :    the CLOBBERs are placed.
   11759              :    If OLD_NREGS != NEW_NREGS, then PNOTES also includes REG_DEAD notes added.
   11760              : 
   11761              :    The value is the final insn code from the pattern ultimately matched,
   11762              :    or -1.  */
   11763              : 
   11764              : static int
   11765     48682995 : recog_for_combine (rtx *pnewpat, rtx_insn *insn, rtx *pnotes,
   11766              :                    unsigned int old_nregs, unsigned int new_nregs)
   11767              : {
   11768     48682995 :   rtx pat = *pnewpat;
   11769     48682995 :   int insn_code_number = recog_for_combine_1 (pnewpat, insn, pnotes,
   11770              :                                               old_nregs, new_nregs);
   11771     48682995 :   if (insn_code_number >= 0 || check_asm_operands (pat))
   11772              :     return insn_code_number;
   11773              : 
   11774     38282428 :   void *marker = get_undo_marker ();
   11775     38282428 :   bool changed = false;
   11776              : 
   11777     38282428 :   if (GET_CODE (pat) == SET)
   11778              :     {
   11779              :       /* For an unrecognized single set of a constant, try placing it in
   11780              :          the constant pool, if this function already uses one.  */
   11781     22540735 :       rtx src = SET_SRC (pat);
   11782     22540735 :       if (CONSTANT_P (src)
   11783       472218 :           && !CONST_INT_P (src)
   11784       421527 :           && crtl->uses_const_pool
   11785       369319 :           && SET_DEST (pat) != pc_rtx)
   11786              :         {
   11787       369317 :           machine_mode mode = GET_MODE (src);
   11788       369317 :           if (mode == VOIDmode)
   11789         1346 :             mode = GET_MODE (SET_DEST (pat));
   11790       369317 :           src = force_const_mem (mode, src);
   11791       369317 :           if (src)
   11792              :             {
   11793       369307 :               SUBST (SET_SRC (pat), src);
   11794       369307 :               changed = true;
   11795              :             }
   11796              :         }
   11797              :       else
   11798     22171418 :         changed = change_zero_ext (pat);
   11799              :     }
   11800     15741693 :   else if (GET_CODE (pat) == PARALLEL)
   11801              :     {
   11802              :       int i;
   11803     43953049 :       for (i = 0; i < XVECLEN (pat, 0); i++)
   11804              :         {
   11805     28227040 :           rtx set = XVECEXP (pat, 0, i);
   11806     28227040 :           if (GET_CODE (set) == SET)
   11807     27968422 :             changed |= change_zero_ext (set);
   11808              :         }
   11809              :     }
   11810              : 
   11811     38266734 :   if (changed)
   11812              :     {
   11813      1301094 :       insn_code_number = recog_for_combine_1 (pnewpat, insn, pnotes,
   11814              :                                               old_nregs, new_nregs);
   11815              : 
   11816      1301094 :       if (insn_code_number < 0)
   11817      1137026 :         undo_to_marker (marker);
   11818              :     }
   11819              : 
   11820              :   return insn_code_number;
   11821              : }
   11822              : 
   11823              : /* Like gen_lowpart_general but for use by combine.  In combine it
   11824              :    is not possible to create any new pseudoregs.  However, it is
   11825              :    safe to create invalid memory addresses, because combine will
   11826              :    try to recognize them and all they will do is make the combine
   11827              :    attempt fail.
   11828              : 
   11829              :    If for some reason this cannot do its job, an rtx
   11830              :    (clobber (const_int 0)) is returned.
   11831              :    An insn containing that will not be recognized.  */
   11832              : 
   11833              : static rtx
   11834    162327112 : gen_lowpart_for_combine (machine_mode omode, rtx x)
   11835              : {
   11836    162327112 :   machine_mode imode = GET_MODE (x);
   11837    162327112 :   rtx result;
   11838              : 
   11839    162327112 :   if (omode == imode)
   11840              :     return x;
   11841              : 
   11842              :   /* We can only support MODE being wider than a word if X is a
   11843              :      constant integer or has a mode the same size.  */
   11844     57657355 :   if (maybe_gt (GET_MODE_SIZE (omode), UNITS_PER_WORD)
   11845     27328812 :       && ! (CONST_SCALAR_INT_P (x)
   11846     10348820 :             || known_eq (GET_MODE_SIZE (imode), GET_MODE_SIZE (omode))))
   11847      3188316 :     goto fail;
   11848              : 
   11849              :   /* X might be a paradoxical (subreg (mem)).  In that case, gen_lowpart
   11850              :      won't know what to do.  So we will strip off the SUBREG here and
   11851              :      process normally.  */
   11852     24140496 :   if (GET_CODE (x) == SUBREG && MEM_P (SUBREG_REG (x)))
   11853              :     {
   11854        13644 :       x = SUBREG_REG (x);
   11855              : 
   11856              :       /* For use in case we fall down into the address adjustments
   11857              :          further below, we need to adjust the known mode and size of
   11858              :          x; imode and isize, since we just adjusted x.  */
   11859        13644 :       imode = GET_MODE (x);
   11860              : 
   11861        13644 :       if (imode == omode)
   11862              :         return x;
   11863              :     }
   11864              : 
   11865     24131536 :   result = gen_lowpart_common (omode, x);
   11866              : 
   11867     24131536 :   if (result)
   11868              :     return result;
   11869              : 
   11870     10186895 :   if (MEM_P (x))
   11871              :     {
   11872              :       /* Refuse to work on a volatile memory ref or one with a mode-dependent
   11873              :          address.  */
   11874      2039418 :       if (MEM_VOLATILE_P (x)
   11875      4026732 :           || mode_dependent_address_p (XEXP (x, 0), MEM_ADDR_SPACE (x)))
   11876        52138 :         goto fail;
   11877              : 
   11878              :       /* If we want to refer to something bigger than the original memref,
   11879              :          generate a paradoxical subreg instead.  That will force a reload
   11880              :          of the original memref X.  */
   11881      1987280 :       if (paradoxical_subreg_p (omode, imode)
   11882      1987280 :           && validate_subreg (omode, GET_MODE (x), x, 0))
   11883      1776078 :         return gen_rtx_SUBREG (omode, x, 0);
   11884              : 
   11885       211202 :       poly_int64 offset = byte_lowpart_offset (omode, imode);
   11886       211202 :       return adjust_address_nv (x, omode, offset);
   11887              :     }
   11888              : 
   11889              :   /* If X is a comparison operator, rewrite it in a new mode.  This
   11890              :      probably won't match, but may allow further simplifications.  */
   11891      8147477 :   else if (COMPARISON_P (x)
   11892       150714 :            && SCALAR_INT_MODE_P (imode)
   11893        54314 :            && SCALAR_INT_MODE_P (omode))
   11894        54303 :     return gen_rtx_fmt_ee (GET_CODE (x), omode, XEXP (x, 0), XEXP (x, 1));
   11895              : 
   11896              :   /* If we couldn't simplify X any other way, just enclose it in a
   11897              :      SUBREG.  Normally, this SUBREG won't match, but some patterns may
   11898              :      include an explicit SUBREG or we may simplify it further in combine.  */
   11899              :   else
   11900              :     {
   11901      8093174 :       rtx res;
   11902              : 
   11903      8093174 :       if (imode == VOIDmode)
   11904              :         {
   11905            8 :           imode = int_mode_for_mode (omode).require ();
   11906            8 :           x = gen_lowpart_common (imode, x);
   11907            8 :           if (x == NULL)
   11908            0 :             goto fail;
   11909              :         }
   11910      8093174 :       res = lowpart_subreg (omode, x, imode);
   11911      8093174 :       if (res)
   11912              :         return res;
   11913              :     }
   11914              : 
   11915        16386 :  fail:
   11916      3256840 :   return gen_rtx_CLOBBER (omode, const0_rtx);
   11917              : }
   11918              : 
   11919              : /* Like gen_lowpart_for_combine but returns NULL_RTX
   11920              :    for an error instead of CLOBBER.
   11921              :    Note no_emit is not called directly from combine but rather from
   11922              :    simplify_rtx and is expecting a NULL on failure rather than
   11923              :    a CLOBBER.  */
   11924              : 
   11925              : static rtx
   11926      1602174 : gen_lowpart_for_combine_no_emit (machine_mode omode, rtx x)
   11927              : {
   11928      1602174 :   rtx tem = gen_lowpart_for_combine (omode, x);
   11929      1602174 :   if (!tem || GET_CODE (tem) == CLOBBER)
   11930        16811 :     return NULL_RTX;
   11931              :   return tem;
   11932              : }
   11933              : 
   11934              : 
   11935              : /* Try to simplify a comparison between OP0 and a constant OP1,
   11936              :    where CODE is the comparison code that will be tested, into a
   11937              :    (CODE OP0 const0_rtx) form.
   11938              : 
   11939              :    The result is a possibly different comparison code to use.
   11940              :    *POP0 and *POP1 may be updated.  */
   11941              : 
   11942              : static enum rtx_code
   11943     16181685 : simplify_compare_const (enum rtx_code code, machine_mode mode,
   11944              :                         rtx *pop0, rtx *pop1)
   11945              : {
   11946     16181685 :   scalar_int_mode int_mode;
   11947     16181685 :   rtx op0 = *pop0;
   11948     16181685 :   HOST_WIDE_INT const_op = INTVAL (*pop1);
   11949              : 
   11950              :   /* Get the constant we are comparing against and turn off all bits
   11951              :      not on in our mode.  */
   11952     16181685 :   if (mode != VOIDmode)
   11953     15781487 :     const_op = trunc_int_for_mode (const_op, mode);
   11954              : 
   11955              :   /* If we are comparing against a constant power of two and the value
   11956              :      being compared can only have that single bit nonzero (e.g., it was
   11957              :      `and'ed with that bit), we can replace this with a comparison
   11958              :      with zero.  */
   11959     16181685 :   if (const_op
   11960      4275342 :       && (code == EQ || code == NE || code == GEU || code == LTU
   11961              :           /* This optimization is incorrect for signed >= INT_MIN or
   11962              :              < INT_MIN, those are always true or always false.  */
   11963        26195 :           || ((code == GE || code == LT) && const_op > 0))
   11964      2880818 :       && is_a <scalar_int_mode> (mode, &int_mode)
   11965      2880818 :       && GET_MODE_PRECISION (int_mode) - 1 < HOST_BITS_PER_WIDE_INT
   11966      2861694 :       && pow2p_hwi (const_op & GET_MODE_MASK (int_mode))
   11967     17102247 :       && (nonzero_bits (op0, int_mode)
   11968       920562 :           == (unsigned HOST_WIDE_INT) (const_op & GET_MODE_MASK (int_mode))))
   11969              :     {
   11970         5108 :       code = (code == EQ || code == GE || code == GEU ? NE : EQ);
   11971              :       const_op = 0;
   11972              :     }
   11973              : 
   11974              :   /* Similarly, if we are comparing a value known to be either -1 or
   11975              :      0 with -1, change it to the opposite comparison against zero.  */
   11976         2459 :   if (const_op == -1
   11977       258934 :       && (code == EQ || code == NE || code == GT || code == LE
   11978              :           || code == GEU || code == LTU)
   11979     16425637 :       && is_a <scalar_int_mode> (mode, &int_mode)
   11980     16432859 :       && num_sign_bit_copies (op0, int_mode) == GET_MODE_PRECISION (int_mode))
   11981              :     {
   11982        12330 :       code = (code == EQ || code == LE || code == GEU ? NE : EQ);
   11983              :       const_op = 0;
   11984              :     }
   11985              : 
   11986              :   /* Do some canonicalizations based on the comparison code.  We prefer
   11987              :      comparisons against zero and then prefer equality comparisons.
   11988              :      If we can reduce the size of a constant, we will do that too.  */
   11989     16169355 :   switch (code)
   11990              :     {
   11991       274073 :     case LT:
   11992              :       /* < C is equivalent to <= (C - 1) */
   11993       274073 :       if (const_op > 0)
   11994              :         {
   11995         5375 :           const_op -= 1;
   11996         5375 :           code = LE;
   11997              :           /* ... fall through to LE case below.  */
   11998       466753 :           gcc_fallthrough ();
   11999              :         }
   12000              :       else
   12001              :         break;
   12002              : 
   12003       466753 :     case LE:
   12004              :       /* <= C is equivalent to < (C + 1); we do this for C < 0  */
   12005       466753 :       if (const_op < 0)
   12006              :         {
   12007           52 :           const_op += 1;
   12008           52 :           code = LT;
   12009              :         }
   12010              : 
   12011              :       /* If we are doing a <= 0 comparison on a value known to have
   12012              :          a zero sign bit, we can replace this with == 0.  */
   12013       466701 :       else if (const_op == 0
   12014       324059 :                && is_a <scalar_int_mode> (mode, &int_mode)
   12015       324059 :                && GET_MODE_PRECISION (int_mode) - 1 < HOST_BITS_PER_WIDE_INT
   12016       790760 :                && (nonzero_bits (op0, int_mode)
   12017       324059 :                    & (HOST_WIDE_INT_1U << (GET_MODE_PRECISION (int_mode) - 1)))
   12018       324059 :                == 0)
   12019              :         code = EQ;
   12020              :       break;
   12021              : 
   12022       241342 :     case GE:
   12023              :       /* >= C is equivalent to > (C - 1).  */
   12024       241342 :       if (const_op > 0)
   12025              :         {
   12026         1264 :           const_op -= 1;
   12027         1264 :           code = GT;
   12028              :           /* ... fall through to GT below.  */
   12029       264338 :           gcc_fallthrough ();
   12030              :         }
   12031              :       else
   12032              :         break;
   12033              : 
   12034       264338 :     case GT:
   12035              :       /* > C is equivalent to >= (C + 1); we do this for C < 0.  */
   12036       264338 :       if (const_op < 0)
   12037              :         {
   12038          322 :           const_op += 1;
   12039          322 :           code = GE;
   12040              :         }
   12041              : 
   12042              :       /* If we are doing a > 0 comparison on a value known to have
   12043              :          a zero sign bit, we can replace this with != 0.  */
   12044       264016 :       else if (const_op == 0
   12045       133551 :                && is_a <scalar_int_mode> (mode, &int_mode)
   12046       133551 :                && GET_MODE_PRECISION (int_mode) - 1 < HOST_BITS_PER_WIDE_INT
   12047       397567 :                && (nonzero_bits (op0, int_mode)
   12048       133551 :                    & (HOST_WIDE_INT_1U << (GET_MODE_PRECISION (int_mode) - 1)))
   12049       133551 :                == 0)
   12050              :         code = NE;
   12051              :       break;
   12052              : 
   12053        97801 :     case LTU:
   12054              :       /* < C is equivalent to <= (C - 1).  */
   12055        97801 :       if (const_op > 0)
   12056              :         {
   12057        88129 :           const_op -= 1;
   12058        88129 :           code = LEU;
   12059              :           /* ... fall through ...  */
   12060        88129 :           gcc_fallthrough ();
   12061              :         }
   12062              :       /* (unsigned) < 0x80000000 is equivalent to >= 0.  */
   12063         9672 :       else if (is_a <scalar_int_mode> (mode, &int_mode)
   12064         9672 :                && GET_MODE_PRECISION (int_mode) - 1 < HOST_BITS_PER_WIDE_INT
   12065         8899 :                && (((unsigned HOST_WIDE_INT) const_op & GET_MODE_MASK (int_mode))
   12066         8899 :                    == HOST_WIDE_INT_1U << (GET_MODE_PRECISION (int_mode) - 1)))
   12067              :         {
   12068              :           const_op = 0;
   12069              :           code = GE;
   12070              :           break;
   12071              :         }
   12072              :       else
   12073              :         break;
   12074              : 
   12075       695600 :     case LEU:
   12076              :       /* unsigned <= 0 is equivalent to == 0 */
   12077       695600 :       if (const_op == 0)
   12078              :         code = EQ;
   12079              :       /* (unsigned) <= 0x7fffffff is equivalent to >= 0.  */
   12080       695142 :       else if (is_a <scalar_int_mode> (mode, &int_mode)
   12081       695142 :                && GET_MODE_PRECISION (int_mode) - 1 < HOST_BITS_PER_WIDE_INT
   12082       693201 :                && ((unsigned HOST_WIDE_INT) const_op
   12083              :                    == ((HOST_WIDE_INT_1U
   12084       693201 :                         << (GET_MODE_PRECISION (int_mode) - 1)) - 1)))
   12085              :         {
   12086              :           const_op = 0;
   12087              :           code = GE;
   12088              :         }
   12089              :       break;
   12090              : 
   12091        31977 :     case GEU:
   12092              :       /* >= C is equivalent to > (C - 1).  */
   12093        31977 :       if (const_op > 1)
   12094              :         {
   12095        23219 :           const_op -= 1;
   12096        23219 :           code = GTU;
   12097              :           /* ... fall through ...  */
   12098        23219 :           gcc_fallthrough ();
   12099              :         }
   12100              : 
   12101              :       /* (unsigned) >= 0x80000000 is equivalent to < 0.  */
   12102         8758 :       else if (is_a <scalar_int_mode> (mode, &int_mode)
   12103         8758 :                && GET_MODE_PRECISION (int_mode) - 1 < HOST_BITS_PER_WIDE_INT
   12104         7498 :                && (((unsigned HOST_WIDE_INT) const_op & GET_MODE_MASK (int_mode))
   12105         7498 :                    == HOST_WIDE_INT_1U << (GET_MODE_PRECISION (int_mode) - 1)))
   12106              :         {
   12107              :           const_op = 0;
   12108              :           code = LT;
   12109              :           break;
   12110              :         }
   12111              :       else
   12112              :         break;
   12113              : 
   12114       523874 :     case GTU:
   12115              :       /* unsigned > 0 is equivalent to != 0 */
   12116       523874 :       if (const_op == 0)
   12117              :         code = NE;
   12118              :       /* (unsigned) > 0x7fffffff is equivalent to < 0.  */
   12119       523874 :       else if (is_a <scalar_int_mode> (mode, &int_mode)
   12120       523874 :                && GET_MODE_PRECISION (int_mode) - 1 < HOST_BITS_PER_WIDE_INT
   12121       522695 :                && ((unsigned HOST_WIDE_INT) const_op
   12122              :                    == (HOST_WIDE_INT_1U
   12123       522695 :                        << (GET_MODE_PRECISION (int_mode) - 1)) - 1))
   12124              :         {
   12125              :           const_op = 0;
   12126              :           code = LT;
   12127              :         }
   12128              :       break;
   12129              : 
   12130              :     default:
   12131              :       break;
   12132              :     }
   12133              : 
   12134              :   /* Narrow non-symmetric comparison of memory and constant as e.g.
   12135              :      x0...x7 <= 0x3fffffffffffffff into x0 <= 0x3f where x0 is the most
   12136              :      significant byte.  Likewise, transform x0...x7 >= 0x4000000000000000 into
   12137              :      x0 >= 0x40.  */
   12138     15468915 :   if ((code == LEU || code == LTU || code == GEU || code == GTU)
   12139      1234191 :       && is_a <scalar_int_mode> (GET_MODE (op0), &int_mode)
   12140      1234158 :       && HWI_COMPUTABLE_MODE_P (int_mode)
   12141      1229005 :       && MEM_P (op0)
   12142        82630 :       && !MEM_VOLATILE_P (op0)
   12143              :       /* The optimization makes only sense for constants which are big enough
   12144              :          so that we have a chance to chop off something at all.  */
   12145        81753 :       && ((unsigned HOST_WIDE_INT) const_op & GET_MODE_MASK (int_mode)) > 0xff
   12146              :       /* Ensure that we do not overflow during normalization.  */
   12147        22207 :       && (code != GTU
   12148         4179 :           || ((unsigned HOST_WIDE_INT) const_op & GET_MODE_MASK (int_mode))
   12149              :              < HOST_WIDE_INT_M1U)
   12150     16203892 :       && trunc_int_for_mode (const_op, int_mode) == const_op)
   12151              :     {
   12152        22207 :       unsigned HOST_WIDE_INT n
   12153        22207 :         = (unsigned HOST_WIDE_INT) const_op & GET_MODE_MASK (int_mode);
   12154        22207 :       enum rtx_code adjusted_code;
   12155              : 
   12156              :       /* Normalize code to either LEU or GEU.  */
   12157        22207 :       if (code == LTU)
   12158              :         {
   12159          116 :           --n;
   12160          116 :           adjusted_code = LEU;
   12161              :         }
   12162        22091 :       else if (code == GTU)
   12163              :         {
   12164         4179 :           ++n;
   12165         4179 :           adjusted_code = GEU;
   12166              :         }
   12167              :       else
   12168              :         adjusted_code = code;
   12169              : 
   12170        22207 :       scalar_int_mode narrow_mode_iter;
   12171        68910 :       FOR_EACH_MODE_UNTIL (narrow_mode_iter, int_mode)
   12172              :         {
   12173        47366 :           unsigned nbits = GET_MODE_PRECISION (int_mode)
   12174        47366 :                            - GET_MODE_PRECISION (narrow_mode_iter);
   12175        47366 :           unsigned HOST_WIDE_INT mask = (HOST_WIDE_INT_1U << nbits) - 1;
   12176        47366 :           unsigned HOST_WIDE_INT lower_bits = n & mask;
   12177        47366 :           if ((adjusted_code == LEU && lower_bits == mask)
   12178        47110 :               || (adjusted_code == GEU && lower_bits == 0))
   12179              :             {
   12180          663 :               n >>= nbits;
   12181          663 :               break;
   12182              :             }
   12183              :         }
   12184              : 
   12185        22207 :       if (narrow_mode_iter < int_mode)
   12186              :         {
   12187          663 :           if (dump_file && (dump_flags & TDF_DETAILS))
   12188              :             {
   12189           12 :               fprintf (
   12190              :                 dump_file, "narrow comparison from mode %s to %s: (MEM %s "
   12191              :                 HOST_WIDE_INT_PRINT_HEX ") to (MEM %s "
   12192           12 :                 HOST_WIDE_INT_PRINT_HEX ").\n", GET_MODE_NAME (int_mode),
   12193           12 :                 GET_MODE_NAME (narrow_mode_iter), GET_RTX_NAME (code),
   12194           12 :                 (unsigned HOST_WIDE_INT) const_op & GET_MODE_MASK (int_mode),
   12195           12 :                 GET_RTX_NAME (adjusted_code), n);
   12196              :             }
   12197          663 :           poly_int64 offset = (BYTES_BIG_ENDIAN
   12198          663 :                                ? 0
   12199          663 :                                : (GET_MODE_SIZE (int_mode)
   12200          663 :                                   - GET_MODE_SIZE (narrow_mode_iter)));
   12201          663 :           *pop0 = adjust_address_nv (op0, narrow_mode_iter, offset);
   12202          663 :           *pop1 = gen_int_mode (n, narrow_mode_iter);
   12203          663 :           return adjusted_code;
   12204              :         }
   12205              :     }
   12206              : 
   12207     16181022 :   *pop1 = GEN_INT (const_op);
   12208     16181022 :   return code;
   12209              : }
   12210              : 
   12211              : /* Simplify a comparison between *POP0 and *POP1 where CODE is the
   12212              :    comparison code that will be tested.
   12213              : 
   12214              :    The result is a possibly different comparison code to use.  *POP0 and
   12215              :    *POP1 may be updated.
   12216              : 
   12217              :    It is possible that we might detect that a comparison is either always
   12218              :    true or always false.  However, we do not perform general constant
   12219              :    folding in combine, so this knowledge isn't useful.  Such tautologies
   12220              :    should have been detected earlier.  Hence we ignore all such cases.  */
   12221              : 
   12222              : static enum rtx_code
   12223     24439984 : simplify_comparison (enum rtx_code code, rtx *pop0, rtx *pop1)
   12224              : {
   12225     24439984 :   rtx op0 = *pop0;
   12226     24439984 :   rtx op1 = *pop1;
   12227     24439984 :   rtx tem, tem1;
   12228     24439984 :   int i;
   12229     24439984 :   scalar_int_mode mode, inner_mode, tmode;
   12230     24439984 :   opt_scalar_int_mode tmode_iter;
   12231              : 
   12232              :   /* Try a few ways of applying the same transformation to both operands.  */
   12233     24440321 :   while (1)
   12234              :     {
   12235              :       /* The test below this one won't handle SIGN_EXTENDs on these machines,
   12236              :          so check specially.  */
   12237     24440321 :       if (!WORD_REGISTER_OPERATIONS
   12238     24440321 :           && code != GTU && code != GEU && code != LTU && code != LEU
   12239     21236912 :           && GET_CODE (op0) == ASHIFTRT && GET_CODE (op1) == ASHIFTRT
   12240         1474 :           && GET_CODE (XEXP (op0, 0)) == ASHIFT
   12241         1095 :           && GET_CODE (XEXP (op1, 0)) == ASHIFT
   12242          725 :           && GET_CODE (XEXP (XEXP (op0, 0), 0)) == SUBREG
   12243          724 :           && GET_CODE (XEXP (XEXP (op1, 0), 0)) == SUBREG
   12244          724 :           && is_a <scalar_int_mode> (GET_MODE (op0), &mode)
   12245              :           && (is_a <scalar_int_mode>
   12246          724 :               (GET_MODE (SUBREG_REG (XEXP (XEXP (op0, 0), 0))), &inner_mode))
   12247          724 :           && inner_mode == GET_MODE (SUBREG_REG (XEXP (XEXP (op1, 0), 0)))
   12248          724 :           && CONST_INT_P (XEXP (op0, 1))
   12249          724 :           && XEXP (op0, 1) == XEXP (op1, 1)
   12250           91 :           && XEXP (op0, 1) == XEXP (XEXP (op0, 0), 1)
   12251           91 :           && XEXP (op0, 1) == XEXP (XEXP (op1, 0), 1)
   12252           91 :           && (INTVAL (XEXP (op0, 1))
   12253           91 :               == (GET_MODE_PRECISION (mode)
   12254           91 :                   - GET_MODE_PRECISION (inner_mode))))
   12255              :         {
   12256           91 :           op0 = SUBREG_REG (XEXP (XEXP (op0, 0), 0));
   12257           91 :           op1 = SUBREG_REG (XEXP (XEXP (op1, 0), 0));
   12258              :         }
   12259              : 
   12260              :       /* If both operands are the same constant shift, see if we can ignore the
   12261              :          shift.  We can if the shift is a rotate or if the bits shifted out of
   12262              :          this shift are known to be zero for both inputs and if the type of
   12263              :          comparison is compatible with the shift.  */
   12264     24440321 :       if (GET_CODE (op0) == GET_CODE (op1)
   12265      3499988 :           && HWI_COMPUTABLE_MODE_P (GET_MODE (op0))
   12266      3142453 :           && ((GET_CODE (op0) == ROTATE && (code == NE || code == EQ))
   12267      3142453 :               || ((GET_CODE (op0) == LSHIFTRT || GET_CODE (op0) == ASHIFT)
   12268          930 :                   && (code != GT && code != LT && code != GE && code != LE))
   12269      3141575 :               || (GET_CODE (op0) == ASHIFTRT
   12270         1425 :                   && (code != GTU && code != LTU
   12271         1417 :                       && code != GEU && code != LEU)))
   12272         2258 :           && CONST_INT_P (XEXP (op0, 1))
   12273         2232 :           && INTVAL (XEXP (op0, 1)) >= 0
   12274         2232 :           && INTVAL (XEXP (op0, 1)) < HOST_BITS_PER_WIDE_INT
   12275     24442553 :           && XEXP (op0, 1) == XEXP (op1, 1))
   12276              :         {
   12277         1158 :           machine_mode mode = GET_MODE (op0);
   12278         1158 :           unsigned HOST_WIDE_INT mask = GET_MODE_MASK (mode);
   12279         1158 :           int shift_count = INTVAL (XEXP (op0, 1));
   12280              : 
   12281         1158 :           if (GET_CODE (op0) == LSHIFTRT || GET_CODE (op0) == ASHIFTRT)
   12282          495 :             mask &= (mask >> shift_count) << shift_count;
   12283          663 :           else if (GET_CODE (op0) == ASHIFT)
   12284          663 :             mask = (mask & (mask << shift_count)) >> shift_count;
   12285              : 
   12286         1158 :           if ((nonzero_bits (XEXP (op0, 0), mode) & ~mask) == 0
   12287         1158 :               && (nonzero_bits (XEXP (op1, 0), mode) & ~mask) == 0)
   12288          171 :             op0 = XEXP (op0, 0), op1 = XEXP (op1, 0);
   12289              :           else
   12290              :             break;
   12291              :         }
   12292              : 
   12293              :       /* If both operands are AND's of a paradoxical SUBREG by constant, the
   12294              :          SUBREGs are of the same mode, and, in both cases, the AND would
   12295              :          be redundant if the comparison was done in the narrower mode,
   12296              :          do the comparison in the narrower mode (e.g., we are AND'ing with 1
   12297              :          and the operand's possibly nonzero bits are 0xffffff01; in that case
   12298              :          if we only care about QImode, we don't need the AND).  This case
   12299              :          occurs if the output mode of an scc insn is not SImode and
   12300              :          STORE_FLAG_VALUE == 1 (e.g., the 386).
   12301              : 
   12302              :          Similarly, check for a case where the AND's are ZERO_EXTEND
   12303              :          operations from some narrower mode even though a SUBREG is not
   12304              :          present.  */
   12305              : 
   12306     24439163 :       else if (GET_CODE (op0) == AND && GET_CODE (op1) == AND
   12307         2717 :                && CONST_INT_P (XEXP (op0, 1))
   12308         2635 :                && CONST_INT_P (XEXP (op1, 1)))
   12309              :         {
   12310         2593 :           rtx inner_op0 = XEXP (op0, 0);
   12311         2593 :           rtx inner_op1 = XEXP (op1, 0);
   12312         2593 :           HOST_WIDE_INT c0 = INTVAL (XEXP (op0, 1));
   12313         2593 :           HOST_WIDE_INT c1 = INTVAL (XEXP (op1, 1));
   12314         2593 :           bool changed = false;
   12315              : 
   12316         2593 :           if (paradoxical_subreg_p (inner_op0)
   12317         1024 :               && GET_CODE (inner_op1) == SUBREG
   12318          482 :               && HWI_COMPUTABLE_MODE_P (GET_MODE (SUBREG_REG (inner_op0)))
   12319          482 :               && (GET_MODE (SUBREG_REG (inner_op0))
   12320          482 :                   == GET_MODE (SUBREG_REG (inner_op1)))
   12321          198 :               && ((~c0) & nonzero_bits (SUBREG_REG (inner_op0),
   12322              :                                         GET_MODE (SUBREG_REG (inner_op0)))) == 0
   12323         1711 :               && ((~c1) & nonzero_bits (SUBREG_REG (inner_op1),
   12324          128 :                                         GET_MODE (SUBREG_REG (inner_op1)))) == 0)
   12325              :             {
   12326          118 :               op0 = SUBREG_REG (inner_op0);
   12327          118 :               op1 = SUBREG_REG (inner_op1);
   12328              : 
   12329              :               /* The resulting comparison is always unsigned since we masked
   12330              :                  off the original sign bit.  */
   12331          118 :               code = unsigned_condition (code);
   12332              : 
   12333          118 :               changed = true;
   12334              :             }
   12335              : 
   12336         2475 :           else if (c0 == c1)
   12337         5188 :             FOR_EACH_MODE_UNTIL (tmode,
   12338              :                                  as_a <scalar_int_mode> (GET_MODE (op0)))
   12339         3103 :               if ((unsigned HOST_WIDE_INT) c0 == GET_MODE_MASK (tmode))
   12340              :                 {
   12341           37 :                   op0 = gen_lowpart_or_truncate (tmode, inner_op0);
   12342           37 :                   op1 = gen_lowpart_or_truncate (tmode, inner_op1);
   12343           37 :                   code = unsigned_condition (code);
   12344           37 :                   changed = true;
   12345           37 :                   break;
   12346              :                 }
   12347              : 
   12348         2240 :           if (! changed)
   12349              :             break;
   12350              :         }
   12351              : 
   12352              :       /* If both operands are NOT, we can strip off the outer operation
   12353              :          and adjust the comparison code for swapped operands; similarly for
   12354              :          NEG, except that this must be an equality comparison.  */
   12355     24436570 :       else if ((GET_CODE (op0) == NOT && GET_CODE (op1) == NOT)
   12356     24436570 :                || (GET_CODE (op0) == NEG && GET_CODE (op1) == NEG
   12357           11 :                    && (code == EQ || code == NE)))
   12358           11 :         op0 = XEXP (op0, 0), op1 = XEXP (op1, 0), code = swap_condition (code);
   12359              : 
   12360              :       else
   12361              :         break;
   12362              :     }
   12363              : 
   12364              :   /* If the first operand is a constant, swap the operands and adjust the
   12365              :      comparison code appropriately, but don't do this if the second operand
   12366              :      is already a constant integer.  */
   12367     24439984 :   if (swap_commutative_operands_p (op0, op1))
   12368              :     {
   12369      1519325 :       std::swap (op0, op1);
   12370      1519325 :       code = swap_condition (code);
   12371              :     }
   12372              : 
   12373              :   /* We now enter a loop during which we will try to simplify the comparison.
   12374              :      For the most part, we only are concerned with comparisons with zero,
   12375              :      but some things may really be comparisons with zero but not start
   12376              :      out looking that way.  */
   12377              : 
   12378     25566384 :   while (CONST_INT_P (op1))
   12379              :     {
   12380     16694845 :       machine_mode raw_mode = GET_MODE (op0);
   12381     16694845 :       scalar_int_mode int_mode;
   12382     16694845 :       int equality_comparison_p;
   12383     16694845 :       int sign_bit_comparison_p;
   12384     16694845 :       int unsigned_comparison_p;
   12385     16694845 :       HOST_WIDE_INT const_op;
   12386              : 
   12387              :       /* We only want to handle integral modes.  This catches VOIDmode,
   12388              :          CCmode, and the floating-point modes.  An exception is that we
   12389              :          can handle VOIDmode if OP0 is a COMPARE or a comparison
   12390              :          operation.  */
   12391              : 
   12392     16694845 :       if (GET_MODE_CLASS (raw_mode) != MODE_INT
   12393      1730325 :           && ! (raw_mode == VOIDmode
   12394       400228 :                 && (GET_CODE (op0) == COMPARE || COMPARISON_P (op0))))
   12395              :         break;
   12396              : 
   12397              :       /* Try to simplify the compare to constant, possibly changing the
   12398              :          comparison op, and/or changing op1 to zero.  */
   12399     15364718 :       code = simplify_compare_const (code, raw_mode, &op0, &op1);
   12400     15364718 :       const_op = INTVAL (op1);
   12401              : 
   12402              :       /* Compute some predicates to simplify code below.  */
   12403              : 
   12404     15364718 :       equality_comparison_p = (code == EQ || code == NE);
   12405     15364718 :       sign_bit_comparison_p = ((code == LT || code == GE) && const_op == 0);
   12406     15364718 :       unsigned_comparison_p = (code == LTU || code == LEU || code == GTU
   12407     15364718 :                                || code == GEU);
   12408              : 
   12409              :       /* If this is a sign bit comparison and we can do arithmetic in
   12410              :          MODE, say that we will only be needing the sign bit of OP0.  */
   12411     15364718 :       if (sign_bit_comparison_p
   12412       462401 :           && is_a <scalar_int_mode> (raw_mode, &int_mode)
   12413     15827119 :           && HWI_COMPUTABLE_MODE_P (int_mode))
   12414       462009 :         op0 = force_to_mode (op0, int_mode,
   12415              :                              HOST_WIDE_INT_1U
   12416       462009 :                              << (GET_MODE_PRECISION (int_mode) - 1), false);
   12417              : 
   12418     15364718 :       if (COMPARISON_P (op0))
   12419              :         {
   12420              :           /* We can't do anything if OP0 is a condition code value, rather
   12421              :              than an actual data value.  */
   12422       724296 :           if (const_op != 0
   12423       724296 :               || GET_MODE_CLASS (GET_MODE (XEXP (op0, 0))) == MODE_CC)
   12424              :             break;
   12425              : 
   12426              :           /* Get the two operands being compared.  */
   12427       136829 :           if (GET_CODE (XEXP (op0, 0)) == COMPARE)
   12428            0 :             tem = XEXP (XEXP (op0, 0), 0), tem1 = XEXP (XEXP (op0, 0), 1);
   12429              :           else
   12430       136829 :             tem = XEXP (op0, 0), tem1 = XEXP (op0, 1);
   12431              : 
   12432              :           /* Check for the cases where we simply want the result of the
   12433              :              earlier test or the opposite of that result.  */
   12434       136829 :           if (code == NE || code == EQ
   12435       136829 :               || (val_signbit_known_set_p (raw_mode, STORE_FLAG_VALUE)
   12436            0 :                   && (code == LT || code == GE)))
   12437              :             {
   12438       136829 :               enum rtx_code new_code;
   12439       136829 :               if (code == LT || code == NE)
   12440       136829 :                 new_code = GET_CODE (op0);
   12441              :               else
   12442            0 :                 new_code = reversed_comparison_code (op0, NULL);
   12443              : 
   12444       136829 :               if (new_code != UNKNOWN)
   12445              :                 {
   12446       136829 :                   code = new_code;
   12447       136829 :                   op0 = tem;
   12448       136829 :                   op1 = tem1;
   12449      1126400 :                   continue;
   12450              :                 }
   12451              :             }
   12452              :           break;
   12453              :         }
   12454              : 
   12455     14640422 :       if (raw_mode == VOIDmode)
   12456              :         break;
   12457     14640422 :       scalar_int_mode mode = as_a <scalar_int_mode> (raw_mode);
   12458              : 
   12459              :       /* Now try cases based on the opcode of OP0.  If none of the cases
   12460              :          does a "continue", we exit this loop immediately after the
   12461              :          switch.  */
   12462              : 
   12463     14640422 :       unsigned int mode_width = GET_MODE_PRECISION (mode);
   12464     14640422 :       unsigned HOST_WIDE_INT mask = GET_MODE_MASK (mode);
   12465     14640422 :       switch (GET_CODE (op0))
   12466              :         {
   12467       390518 :         case ZERO_EXTRACT:
   12468              :           /* If we are extracting a single bit from a variable position in
   12469              :              a constant that has only a single bit set and are comparing it
   12470              :              with zero, we can convert this into an equality comparison
   12471              :              between the position and the location of the single bit.  */
   12472              :           /* Except we can't if SHIFT_COUNT_TRUNCATED is set, since we might
   12473              :              have already reduced the shift count modulo the word size.  */
   12474       390518 :           if (!SHIFT_COUNT_TRUNCATED
   12475       390518 :               && CONST_INT_P (XEXP (op0, 0))
   12476         9328 :               && XEXP (op0, 1) == const1_rtx
   12477         9310 :               && equality_comparison_p && const_op == 0
   12478       399828 :               && (i = exact_log2 (UINTVAL (XEXP (op0, 0)))) >= 0)
   12479              :             {
   12480            0 :               if (BITS_BIG_ENDIAN)
   12481              :                 i = BITS_PER_WORD - 1 - i;
   12482              : 
   12483            0 :               op0 = XEXP (op0, 2);
   12484            0 :               op1 = GEN_INT (i);
   12485            0 :               const_op = i;
   12486              : 
   12487              :               /* Result is nonzero iff shift count is equal to I.  */
   12488            0 :               code = reverse_condition (code);
   12489            0 :               continue;
   12490              :             }
   12491              : 
   12492              :           /* fall through */
   12493              : 
   12494       390522 :         case SIGN_EXTRACT:
   12495       390522 :           tem = expand_compound_operation (op0);
   12496       390522 :           if (tem != op0)
   12497              :             {
   12498       357645 :               op0 = tem;
   12499       357645 :               continue;
   12500              :             }
   12501              :           break;
   12502              : 
   12503        28805 :         case NOT:
   12504              :           /* If testing for equality, we can take the NOT of the constant.  */
   12505        41174 :           if (equality_comparison_p
   12506        28805 :               && (tem = simplify_unary_operation (NOT, mode, op1, mode)) != 0)
   12507              :             {
   12508        12369 :               op0 = XEXP (op0, 0);
   12509        12369 :               op1 = tem;
   12510        12369 :               continue;
   12511              :             }
   12512              : 
   12513              :           /* If just looking at the sign bit, reverse the sense of the
   12514              :              comparison.  */
   12515        16436 :           if (sign_bit_comparison_p)
   12516              :             {
   12517        16096 :               op0 = XEXP (op0, 0);
   12518        16096 :               code = (code == GE ? LT : GE);
   12519        16096 :               continue;
   12520              :             }
   12521              :           break;
   12522              : 
   12523       245120 :         case NEG:
   12524              :           /* If testing for equality, we can take the NEG of the constant.  */
   12525       486631 :           if (equality_comparison_p
   12526       245120 :               && (tem = simplify_unary_operation (NEG, mode, op1, mode)) != 0)
   12527              :             {
   12528       241511 :               op0 = XEXP (op0, 0);
   12529       241511 :               op1 = tem;
   12530       241511 :               continue;
   12531              :             }
   12532              : 
   12533              :           /* The remaining cases only apply to comparisons with zero.  */
   12534         3609 :           if (const_op != 0)
   12535              :             break;
   12536              : 
   12537              :           /* When X is ABS or is known positive,
   12538              :              (neg X) is < 0 if and only if X != 0.  */
   12539              : 
   12540         3023 :           if (sign_bit_comparison_p
   12541         2985 :               && (GET_CODE (XEXP (op0, 0)) == ABS
   12542         2978 :                   || (mode_width <= HOST_BITS_PER_WIDE_INT
   12543         2978 :                       && (nonzero_bits (XEXP (op0, 0), mode)
   12544         2978 :                           & (HOST_WIDE_INT_1U << (mode_width - 1)))
   12545         2978 :                          == 0)))
   12546              :             {
   12547           38 :               op0 = XEXP (op0, 0);
   12548           38 :               code = (code == LT ? NE : EQ);
   12549           38 :               continue;
   12550              :             }
   12551              : 
   12552              :           /* If we have NEG of something whose two high-order bits are the
   12553              :              same, we know that "(-a) < 0" is equivalent to "a > 0".  */
   12554         2947 :           if (num_sign_bit_copies (op0, mode) >= 2)
   12555              :             {
   12556           22 :               op0 = XEXP (op0, 0);
   12557           22 :               code = swap_condition (code);
   12558           22 :               continue;
   12559              :             }
   12560              :           break;
   12561              : 
   12562          146 :         case ROTATE:
   12563              :           /* If we are testing equality and our count is a constant, we
   12564              :              can perform the inverse operation on our RHS.  */
   12565          146 :           if (equality_comparison_p && CONST_INT_P (XEXP (op0, 1))
   12566          146 :               && (tem = simplify_binary_operation (ROTATERT, mode,
   12567              :                                                    op1, XEXP (op0, 1))) != 0)
   12568              :             {
   12569            0 :               op0 = XEXP (op0, 0);
   12570            0 :               op1 = tem;
   12571            0 :               continue;
   12572              :             }
   12573              : 
   12574              :           /* If we are doing a < 0 or >= 0 comparison, it means we are testing
   12575              :              a particular bit.  Convert it to an AND of a constant of that
   12576              :              bit.  This will be converted into a ZERO_EXTRACT.  */
   12577          146 :           if (const_op == 0 && sign_bit_comparison_p
   12578            0 :               && CONST_INT_P (XEXP (op0, 1))
   12579            0 :               && mode_width <= HOST_BITS_PER_WIDE_INT
   12580            0 :               && UINTVAL (XEXP (op0, 1)) < mode_width)
   12581              :             {
   12582            0 :               op0 = simplify_and_const_int (NULL_RTX, mode, XEXP (op0, 0),
   12583              :                                             (HOST_WIDE_INT_1U
   12584              :                                              << (mode_width - 1
   12585            0 :                                                  - INTVAL (XEXP (op0, 1)))));
   12586            0 :               code = (code == LT ? NE : EQ);
   12587            0 :               continue;
   12588              :             }
   12589              : 
   12590              :           /* Fall through.  */
   12591              : 
   12592         1554 :         case ABS:
   12593              :           /* ABS is ignorable inside an equality comparison with zero.  */
   12594         1554 :           if (const_op == 0 && equality_comparison_p)
   12595              :             {
   12596            1 :               op0 = XEXP (op0, 0);
   12597            1 :               continue;
   12598              :             }
   12599              :           break;
   12600              : 
   12601         1763 :         case SIGN_EXTEND:
   12602              :           /* Can simplify (compare (zero/sign_extend FOO) CONST) to
   12603              :              (compare FOO CONST) if CONST fits in FOO's mode and we
   12604              :              are either testing inequality or have an unsigned
   12605              :              comparison with ZERO_EXTEND or a signed comparison with
   12606              :              SIGN_EXTEND.  But don't do it if we don't have a compare
   12607              :              insn of the given mode, since we'd have to revert it
   12608              :              later on, and then we wouldn't know whether to sign- or
   12609              :              zero-extend.  */
   12610         1763 :           if (is_int_mode (GET_MODE (XEXP (op0, 0)), &mode)
   12611         1763 :               && ! unsigned_comparison_p
   12612          996 :               && HWI_COMPUTABLE_MODE_P (mode)
   12613          996 :               && trunc_int_for_mode (const_op, mode) == const_op
   12614          996 :               && have_insn_for (COMPARE, mode))
   12615              :             {
   12616          996 :               op0 = XEXP (op0, 0);
   12617          996 :               continue;
   12618              :             }
   12619              :           break;
   12620              : 
   12621       493284 :         case SUBREG:
   12622              :           /* Check for the case where we are comparing A - C1 with C2, that is
   12623              : 
   12624              :                (subreg:MODE (plus (A) (-C1))) op (C2)
   12625              : 
   12626              :              with C1 a constant, and try to lift the SUBREG, i.e. to do the
   12627              :              comparison in the wider mode.  One of the following two conditions
   12628              :              must be true in order for this to be valid:
   12629              : 
   12630              :                1. The mode extension results in the same bit pattern being added
   12631              :                   on both sides and the comparison is equality or unsigned.  As
   12632              :                   C2 has been truncated to fit in MODE, the pattern can only be
   12633              :                   all 0s or all 1s.
   12634              : 
   12635              :                2. The mode extension results in the sign bit being copied on
   12636              :                   each side.
   12637              : 
   12638              :              The difficulty here is that we have predicates for A but not for
   12639              :              (A - C1) so we need to check that C1 is within proper bounds so
   12640              :              as to perturb A as little as possible.  */
   12641              : 
   12642       493284 :           if (mode_width <= HOST_BITS_PER_WIDE_INT
   12643       493208 :               && subreg_lowpart_p (op0)
   12644       462269 :               && is_a <scalar_int_mode> (GET_MODE (SUBREG_REG (op0)),
   12645              :                                          &inner_mode)
   12646       460154 :               && GET_MODE_PRECISION (inner_mode) > mode_width
   12647       460154 :               && GET_CODE (SUBREG_REG (op0)) == PLUS
   12648       493284 :               && CONST_INT_P (XEXP (SUBREG_REG (op0), 1)))
   12649              :             {
   12650            0 :               rtx a = XEXP (SUBREG_REG (op0), 0);
   12651            0 :               HOST_WIDE_INT c1 = -INTVAL (XEXP (SUBREG_REG (op0), 1));
   12652              : 
   12653            0 :               if ((c1 > 0
   12654            0 :                    && (unsigned HOST_WIDE_INT) c1
   12655            0 :                        < HOST_WIDE_INT_1U << (mode_width - 1)
   12656            0 :                    && (equality_comparison_p || unsigned_comparison_p)
   12657              :                    /* (A - C1) zero-extends if it is positive and sign-extends
   12658              :                       if it is negative, C2 both zero- and sign-extends.  */
   12659            0 :                    && (((nonzero_bits (a, inner_mode)
   12660            0 :                          & ~GET_MODE_MASK (mode)) == 0
   12661            0 :                         && const_op >= 0)
   12662              :                        /* (A - C1) sign-extends if it is positive and 1-extends
   12663              :                           if it is negative, C2 both sign- and 1-extends.  */
   12664            0 :                        || (num_sign_bit_copies (a, inner_mode)
   12665            0 :                            > (unsigned int) (GET_MODE_PRECISION (inner_mode)
   12666            0 :                                              - mode_width)
   12667            0 :                            && const_op < 0)))
   12668            0 :                   || ((unsigned HOST_WIDE_INT) c1
   12669            0 :                        < HOST_WIDE_INT_1U << (mode_width - 2)
   12670              :                       /* (A - C1) always sign-extends, like C2.  */
   12671            0 :                       && num_sign_bit_copies (a, inner_mode)
   12672            0 :                          > (unsigned int) (GET_MODE_PRECISION (inner_mode)
   12673            0 :                                            - (mode_width - 1))))
   12674              :                 {
   12675            0 :                   op0 = SUBREG_REG (op0);
   12676            0 :                   continue;
   12677              :                 }
   12678              :             }
   12679              : 
   12680              :           /* If the inner mode is narrower and we are extracting the low part,
   12681              :              we can treat the SUBREG as if it were a ZERO_EXTEND ...  */
   12682       493284 :           if (paradoxical_subreg_p (op0))
   12683              :             {
   12684              :               if (WORD_REGISTER_OPERATIONS
   12685              :                   && is_a <scalar_int_mode> (GET_MODE (SUBREG_REG (op0)),
   12686              :                                              &inner_mode)
   12687              :                   && GET_MODE_PRECISION (inner_mode) < BITS_PER_WORD
   12688              :                   /* On WORD_REGISTER_OPERATIONS targets the bits
   12689              :                      beyond sub_mode aren't considered undefined,
   12690              :                      so optimize only if it is a MEM load when MEM loads
   12691              :                      zero extend, because then the upper bits are all zero.  */
   12692              :                   && !(MEM_P (SUBREG_REG (op0))
   12693              :                        && load_extend_op (inner_mode) == ZERO_EXTEND))
   12694              :                 break;
   12695              :               /* FALLTHROUGH to case ZERO_EXTEND */
   12696              :             }
   12697       493284 :           else if (subreg_lowpart_p (op0)
   12698       462345 :                    && GET_MODE_CLASS (mode) == MODE_INT
   12699       462345 :                    && is_int_mode (GET_MODE (SUBREG_REG (op0)), &inner_mode)
   12700       460154 :                    && (code == NE || code == EQ)
   12701       325062 :                    && GET_MODE_PRECISION (inner_mode) <= HOST_BITS_PER_WIDE_INT
   12702       318816 :                    && !paradoxical_subreg_p (op0)
   12703       812100 :                    && (nonzero_bits (SUBREG_REG (op0), inner_mode)
   12704       318816 :                        & ~GET_MODE_MASK (mode)) == 0)
   12705              :             {
   12706              :               /* Remove outer subregs that don't do anything.  */
   12707        63521 :               tem = gen_lowpart (inner_mode, op1);
   12708              : 
   12709        63521 :               if ((nonzero_bits (tem, inner_mode)
   12710        63521 :                    & ~GET_MODE_MASK (mode)) == 0)
   12711              :                 {
   12712        62801 :                   op0 = SUBREG_REG (op0);
   12713        62801 :                   op1 = tem;
   12714        62801 :                   continue;
   12715              :                 }
   12716              :               break;
   12717              :             }
   12718              :           else
   12719              :             break;
   12720              : 
   12721              :           /* FALLTHROUGH */
   12722              : 
   12723        42183 :         case ZERO_EXTEND:
   12724        42183 :           if (is_int_mode (GET_MODE (XEXP (op0, 0)), &mode)
   12725        42183 :               && (unsigned_comparison_p || equality_comparison_p)
   12726        42145 :               && HWI_COMPUTABLE_MODE_P (mode)
   12727        42145 :               && (unsigned HOST_WIDE_INT) const_op <= GET_MODE_MASK (mode)
   12728        42145 :               && const_op >= 0
   12729        42136 :               && have_insn_for (COMPARE, mode))
   12730              :             {
   12731        42136 :               op0 = XEXP (op0, 0);
   12732        42136 :               continue;
   12733              :             }
   12734              :           break;
   12735              : 
   12736       464206 :         case PLUS:
   12737              :           /* (eq (plus X A) B) -> (eq X (minus B A)).  We can only do
   12738              :              this for equality comparisons due to pathological cases involving
   12739              :              overflows.  */
   12740       520188 :           if (equality_comparison_p
   12741       464206 :               && (tem = simplify_binary_operation (MINUS, mode,
   12742              :                                                    op1, XEXP (op0, 1))) != 0)
   12743              :             {
   12744        55982 :               op0 = XEXP (op0, 0);
   12745        55982 :               op1 = tem;
   12746        55982 :               continue;
   12747              :             }
   12748              : 
   12749              :           /* (plus (abs X) (const_int -1)) is < 0 if and only if X == 0.  */
   12750       408224 :           if (const_op == 0 && XEXP (op0, 1) == constm1_rtx
   12751        15405 :               && GET_CODE (XEXP (op0, 0)) == ABS && sign_bit_comparison_p)
   12752              :             {
   12753            0 :               op0 = XEXP (XEXP (op0, 0), 0);
   12754            0 :               code = (code == LT ? EQ : NE);
   12755            0 :               continue;
   12756              :             }
   12757              :           break;
   12758              : 
   12759       182766 :         case MINUS:
   12760              :           /* We used to optimize signed comparisons against zero, but that
   12761              :              was incorrect.  Unsigned comparisons against zero (GTU, LEU)
   12762              :              arrive here as equality comparisons, or (GEU, LTU) are
   12763              :              optimized away.  No need to special-case them.  */
   12764              : 
   12765              :           /* (eq (minus A B) C) -> (eq A (plus B C)) or
   12766              :              (eq B (minus A C)), whichever simplifies.  We can only do
   12767              :              this for equality comparisons due to pathological cases involving
   12768              :              overflows.  */
   12769       214995 :           if (equality_comparison_p
   12770       182766 :               && (tem = simplify_binary_operation (PLUS, mode,
   12771              :                                                    XEXP (op0, 1), op1)) != 0)
   12772              :             {
   12773        32229 :               op0 = XEXP (op0, 0);
   12774        32229 :               op1 = tem;
   12775        32229 :               continue;
   12776              :             }
   12777              : 
   12778       183164 :           if (equality_comparison_p
   12779       150537 :               && (tem = simplify_binary_operation (MINUS, mode,
   12780              :                                                    XEXP (op0, 0), op1)) != 0)
   12781              :             {
   12782        32627 :               op0 = XEXP (op0, 1);
   12783        32627 :               op1 = tem;
   12784        32627 :               continue;
   12785              :             }
   12786              : 
   12787              :           /* The sign bit of (minus (ashiftrt X C) X), where C is the number
   12788              :              of bits in X minus 1, is one iff X > 0.  */
   12789        15987 :           if (sign_bit_comparison_p && GET_CODE (XEXP (op0, 0)) == ASHIFTRT
   12790          489 :               && CONST_INT_P (XEXP (XEXP (op0, 0), 1))
   12791          489 :               && UINTVAL (XEXP (XEXP (op0, 0), 1)) == mode_width - 1
   12792       117934 :               && rtx_equal_p (XEXP (XEXP (op0, 0), 0), XEXP (op0, 1)))
   12793              :             {
   12794            0 :               op0 = XEXP (op0, 1);
   12795            0 :               code = (code == GE ? LE : GT);
   12796            0 :               continue;
   12797              :             }
   12798              :           break;
   12799              : 
   12800         9015 :         case XOR:
   12801              :           /* (eq (xor A B) C) -> (eq A (xor B C)).  This is a simplification
   12802              :              if C is zero or B is a constant.  */
   12803         9031 :           if (equality_comparison_p
   12804         9015 :               && (tem = simplify_binary_operation (XOR, mode,
   12805              :                                                    XEXP (op0, 1), op1)) != 0)
   12806              :             {
   12807           16 :               op0 = XEXP (op0, 0);
   12808           16 :               op1 = tem;
   12809           16 :               continue;
   12810              :             }
   12811              :           break;
   12812              : 
   12813              : 
   12814       394468 :         case IOR:
   12815              :           /* The sign bit of (ior (plus X (const_int -1)) X) is nonzero
   12816              :              iff X <= 0.  */
   12817         7512 :           if (sign_bit_comparison_p && GET_CODE (XEXP (op0, 0)) == PLUS
   12818         1246 :               && XEXP (XEXP (op0, 0), 1) == constm1_rtx
   12819       394516 :               && rtx_equal_p (XEXP (XEXP (op0, 0), 0), XEXP (op0, 1)))
   12820              :             {
   12821           48 :               op0 = XEXP (op0, 1);
   12822           48 :               code = (code == GE ? GT : LE);
   12823           48 :               continue;
   12824              :             }
   12825              :           break;
   12826              : 
   12827      1693322 :         case AND:
   12828              :           /* Convert (and (xshift 1 X) Y) to (and (lshiftrt Y X) 1).  This
   12829              :              will be converted to a ZERO_EXTRACT later.  */
   12830      1693322 :           if (const_op == 0 && equality_comparison_p
   12831      1575961 :               && GET_CODE (XEXP (op0, 0)) == ASHIFT
   12832        60929 :               && XEXP (XEXP (op0, 0), 0) == const1_rtx)
   12833              :             {
   12834         6839 :               op0 = gen_rtx_LSHIFTRT (mode, XEXP (op0, 1),
   12835              :                                       XEXP (XEXP (op0, 0), 1));
   12836         6839 :               op0 = simplify_and_const_int (NULL_RTX, mode, op0, 1);
   12837         6839 :               continue;
   12838              :             }
   12839              : 
   12840              :           /* If we are comparing (and (lshiftrt X C1) C2) for equality with
   12841              :              zero and X is a comparison and C1 and C2 describe only bits set
   12842              :              in STORE_FLAG_VALUE, we can compare with X.  */
   12843      1686483 :           if (const_op == 0 && equality_comparison_p
   12844      1569122 :               && mode_width <= HOST_BITS_PER_WIDE_INT
   12845      1565231 :               && CONST_INT_P (XEXP (op0, 1))
   12846      1209583 :               && GET_CODE (XEXP (op0, 0)) == LSHIFTRT
   12847       510879 :               && CONST_INT_P (XEXP (XEXP (op0, 0), 1))
   12848       497158 :               && INTVAL (XEXP (XEXP (op0, 0), 1)) >= 0
   12849       497158 :               && INTVAL (XEXP (XEXP (op0, 0), 1)) < HOST_BITS_PER_WIDE_INT)
   12850              :             {
   12851       497158 :               mask = ((INTVAL (XEXP (op0, 1)) & GET_MODE_MASK (mode))
   12852       497158 :                       << INTVAL (XEXP (XEXP (op0, 0), 1)));
   12853       497158 :               if ((~STORE_FLAG_VALUE & mask) == 0
   12854       497158 :                   && (COMPARISON_P (XEXP (XEXP (op0, 0), 0))
   12855            0 :                       || ((tem = get_last_value (XEXP (XEXP (op0, 0), 0))) != 0
   12856            0 :                           && COMPARISON_P (tem))))
   12857              :                 {
   12858            0 :                   op0 = XEXP (XEXP (op0, 0), 0);
   12859            0 :                   continue;
   12860              :                 }
   12861              :             }
   12862              : 
   12863              :           /* If we are doing an equality comparison of an AND of a bit equal
   12864              :              to the sign bit, replace this with a LT or GE comparison of
   12865              :              the underlying value.  */
   12866      1687062 :           if (equality_comparison_p
   12867              :               && const_op == 0
   12868      1569122 :               && CONST_INT_P (XEXP (op0, 1))
   12869      1209894 :               && mode_width <= HOST_BITS_PER_WIDE_INT
   12870      1686483 :               && ((INTVAL (XEXP (op0, 1)) & GET_MODE_MASK (mode))
   12871      1209583 :                   == HOST_WIDE_INT_1U << (mode_width - 1)))
   12872              :             {
   12873          579 :               op0 = XEXP (op0, 0);
   12874          579 :               code = (code == EQ ? GE : LT);
   12875          579 :               continue;
   12876              :             }
   12877              : 
   12878              :           /* If this AND operation is really a ZERO_EXTEND from a narrower
   12879              :              mode, the constant fits within that mode, and this is either an
   12880              :              equality or unsigned comparison, try to do this comparison in
   12881              :              the narrower mode.
   12882              : 
   12883              :              Note that in:
   12884              : 
   12885              :              (ne:DI (and:DI (reg:DI 4) (const_int 0xffffffff)) (const_int 0))
   12886              :              -> (ne:DI (reg:SI 4) (const_int 0))
   12887              : 
   12888              :              unless TARGET_TRULY_NOOP_TRUNCATION allows it or the register is
   12889              :              known to hold a value of the required mode the
   12890              :              transformation is invalid.  */
   12891      1702076 :           if ((equality_comparison_p || unsigned_comparison_p)
   12892      1670015 :               && CONST_INT_P (XEXP (op0, 1))
   12893      3877422 :               && (i = exact_log2 ((UINTVAL (XEXP (op0, 1))
   12894      1303795 :                                    & GET_MODE_MASK (mode))
   12895              :                                   + 1)) >= 0
   12896       903895 :               && const_op >> i == 0
   12897      2589799 :               && int_mode_for_size (i, 1).exists (&tmode))
   12898              :             {
   12899        16172 :               op0 = gen_lowpart_or_truncate (tmode, XEXP (op0, 0));
   12900        16172 :               continue;
   12901              :             }
   12902              : 
   12903              :           /* Convert (ne (and (not X) 1) 0) to (eq (and X 1) 0).  */
   12904      1669732 :           if (const_op == 0 && equality_comparison_p
   12905      1557727 :               && XEXP (op0, 1) == const1_rtx
   12906       681161 :               && GET_CODE (XEXP (op0, 0)) == NOT)
   12907              :             {
   12908         6261 :               op0 = simplify_and_const_int (NULL_RTX, mode,
   12909              :                                             XEXP (XEXP (op0, 0), 0), 1);
   12910         6261 :               code = (code == NE ? EQ : NE);
   12911         6261 :               continue;
   12912              :             }
   12913              : 
   12914              :           /* Convert (ne (and (lshiftrt (not X)) 1) 0) to
   12915              :              (eq (and (lshiftrt X) 1) 0).
   12916              :              Also handle the case where (not X) is expressed using xor.  */
   12917      1663471 :           if (const_op == 0 && equality_comparison_p
   12918      1551466 :               && XEXP (op0, 1) == const1_rtx
   12919       674900 :               && GET_CODE (XEXP (op0, 0)) == LSHIFTRT)
   12920              :             {
   12921       505004 :               rtx shift_op = XEXP (XEXP (op0, 0), 0);
   12922       505004 :               rtx shift_count = XEXP (XEXP (op0, 0), 1);
   12923              : 
   12924       507859 :               if (GET_CODE (shift_op) == NOT
   12925       505004 :                   || (GET_CODE (shift_op) == XOR
   12926         4960 :                       && CONST_INT_P (XEXP (shift_op, 1))
   12927         2855 :                       && CONST_INT_P (shift_count)
   12928         2855 :                       && HWI_COMPUTABLE_MODE_P (mode)
   12929         2855 :                       && (UINTVAL (XEXP (shift_op, 1))
   12930              :                           == HOST_WIDE_INT_1U
   12931         2855 :                                << INTVAL (shift_count))))
   12932              :                 {
   12933         2855 :                   op0
   12934         2855 :                     = gen_rtx_LSHIFTRT (mode, XEXP (shift_op, 0), shift_count);
   12935         2855 :                   op0 = simplify_and_const_int (NULL_RTX, mode, op0, 1);
   12936         2855 :                   code = (code == NE ? EQ : NE);
   12937         2855 :                   continue;
   12938              :                 }
   12939              :             }
   12940              :           break;
   12941              : 
   12942        48732 :         case ASHIFT:
   12943              :           /* If we have (compare (ashift FOO N) (const_int C)) and
   12944              :              the high order N bits of FOO (N+1 if an inequality comparison)
   12945              :              are known to be zero, we can do this by comparing FOO with C
   12946              :              shifted right N bits so long as the low-order N bits of C are
   12947              :              zero.  */
   12948        48732 :           if (CONST_INT_P (XEXP (op0, 1))
   12949        44966 :               && INTVAL (XEXP (op0, 1)) >= 0
   12950        44966 :               && ((INTVAL (XEXP (op0, 1)) + ! equality_comparison_p)
   12951              :                   < HOST_BITS_PER_WIDE_INT)
   12952        44966 :               && (((unsigned HOST_WIDE_INT) const_op
   12953        44966 :                    & ((HOST_WIDE_INT_1U << INTVAL (XEXP (op0, 1)))
   12954              :                       - 1)) == 0)
   12955        38029 :               && mode_width <= HOST_BITS_PER_WIDE_INT
   12956        86741 :               && (nonzero_bits (XEXP (op0, 0), mode)
   12957        38009 :                   & ~(mask >> (INTVAL (XEXP (op0, 1))
   12958        38009 :                                + ! equality_comparison_p))) == 0)
   12959              :             {
   12960              :               /* We must perform a logical shift, not an arithmetic one,
   12961              :                  as we want the top N bits of C to be zero.  */
   12962          768 :               unsigned HOST_WIDE_INT temp = const_op & GET_MODE_MASK (mode);
   12963              : 
   12964          768 :               temp >>= INTVAL (XEXP (op0, 1));
   12965          768 :               op1 = gen_int_mode (temp, mode);
   12966          768 :               op0 = XEXP (op0, 0);
   12967          768 :               continue;
   12968          768 :             }
   12969              : 
   12970              :           /* If we are doing a sign bit comparison, it means we are testing
   12971              :              a particular bit.  Convert it to the appropriate AND.  */
   12972        47964 :           if (sign_bit_comparison_p && CONST_INT_P (XEXP (op0, 1))
   12973         1676 :               && mode_width <= HOST_BITS_PER_WIDE_INT)
   12974              :             {
   12975         3352 :               op0 = simplify_and_const_int (NULL_RTX, mode, XEXP (op0, 0),
   12976              :                                             (HOST_WIDE_INT_1U
   12977              :                                              << (mode_width - 1
   12978         1676 :                                                  - INTVAL (XEXP (op0, 1)))));
   12979         1676 :               code = (code == LT ? NE : EQ);
   12980         1676 :               continue;
   12981              :             }
   12982              : 
   12983              :           /* If this an equality comparison with zero and we are shifting
   12984              :              the low bit to the sign bit, we can convert this to an AND of the
   12985              :              low-order bit.  */
   12986        46288 :           if (const_op == 0 && equality_comparison_p
   12987        15944 :               && CONST_INT_P (XEXP (op0, 1))
   12988        13348 :               && UINTVAL (XEXP (op0, 1)) == mode_width - 1)
   12989              :             {
   12990          402 :               op0 = simplify_and_const_int (NULL_RTX, mode, XEXP (op0, 0), 1);
   12991          402 :               continue;
   12992              :             }
   12993              :           break;
   12994              : 
   12995        44428 :         case ASHIFTRT:
   12996              :           /* If this is an equality comparison with zero, we can do this
   12997              :              as a logical shift, which might be much simpler.  */
   12998        44428 :           if (equality_comparison_p && const_op == 0
   12999        27031 :               && CONST_INT_P (XEXP (op0, 1)))
   13000              :             {
   13001        52540 :               op0 = simplify_shift_const (NULL_RTX, LSHIFTRT, mode,
   13002              :                                           XEXP (op0, 0),
   13003        26270 :                                           INTVAL (XEXP (op0, 1)));
   13004        26270 :               continue;
   13005              :             }
   13006              : 
   13007              :           /* If OP0 is a sign extension and CODE is not an unsigned comparison,
   13008              :              do the comparison in a narrower mode.  */
   13009        23156 :           if (! unsigned_comparison_p
   13010        16910 :               && CONST_INT_P (XEXP (op0, 1))
   13011        16057 :               && GET_CODE (XEXP (op0, 0)) == ASHIFT
   13012         5588 :               && XEXP (op0, 1) == XEXP (XEXP (op0, 0), 1)
   13013         5486 :               && (int_mode_for_size (mode_width - INTVAL (XEXP (op0, 1)), 1)
   13014        18158 :                   .exists (&tmode))
   13015        18158 :               && (((unsigned HOST_WIDE_INT) const_op
   13016         4998 :                    + (GET_MODE_MASK (tmode) >> 1) + 1)
   13017         4998 :                   <= GET_MODE_MASK (tmode)))
   13018              :             {
   13019         4998 :               op0 = gen_lowpart (tmode, XEXP (XEXP (op0, 0), 0));
   13020         4998 :               continue;
   13021              :             }
   13022              : 
   13023              :           /* Likewise if OP0 is a PLUS of a sign extension with a
   13024              :              constant, which is usually represented with the PLUS
   13025              :              between the shifts.  */
   13026        13160 :           if (! unsigned_comparison_p
   13027        11912 :               && CONST_INT_P (XEXP (op0, 1))
   13028        11059 :               && GET_CODE (XEXP (op0, 0)) == PLUS
   13029           54 :               && CONST_INT_P (XEXP (XEXP (op0, 0), 1))
   13030           22 :               && GET_CODE (XEXP (XEXP (op0, 0), 0)) == ASHIFT
   13031            2 :               && XEXP (op0, 1) == XEXP (XEXP (XEXP (op0, 0), 0), 1)
   13032            0 :               && (int_mode_for_size (mode_width - INTVAL (XEXP (op0, 1)), 1)
   13033        13160 :                   .exists (&tmode))
   13034        13160 :               && (((unsigned HOST_WIDE_INT) const_op
   13035            0 :                    + (GET_MODE_MASK (tmode) >> 1) + 1)
   13036            0 :                   <= GET_MODE_MASK (tmode)))
   13037              :             {
   13038            0 :               rtx inner = XEXP (XEXP (XEXP (op0, 0), 0), 0);
   13039            0 :               rtx add_const = XEXP (XEXP (op0, 0), 1);
   13040            0 :               rtx new_const = simplify_gen_binary (ASHIFTRT, mode,
   13041              :                                                    add_const, XEXP (op0, 1));
   13042              : 
   13043            0 :               op0 = simplify_gen_binary (PLUS, tmode,
   13044            0 :                                          gen_lowpart (tmode, inner),
   13045              :                                          new_const);
   13046            0 :               continue;
   13047            0 :             }
   13048              : 
   13049              :           /* FALLTHROUGH */
   13050       132971 :         case LSHIFTRT:
   13051              :           /* If we have (compare (xshiftrt FOO N) (const_int C)) and
   13052              :              the low order N bits of FOO are known to be zero, we can do this
   13053              :              by comparing FOO with C shifted left N bits so long as no
   13054              :              overflow occurs.  Even if the low order N bits of FOO aren't known
   13055              :              to be zero, if the comparison is >= or < we can use the same
   13056              :              optimization and for > or <= by setting all the low
   13057              :              order N bits in the comparison constant.  */
   13058       132971 :           if (CONST_INT_P (XEXP (op0, 1))
   13059       128169 :               && INTVAL (XEXP (op0, 1)) > 0
   13060       128169 :               && INTVAL (XEXP (op0, 1)) < HOST_BITS_PER_WIDE_INT
   13061       127809 :               && mode_width <= HOST_BITS_PER_WIDE_INT
   13062       132971 :               && (((unsigned HOST_WIDE_INT) const_op
   13063       254104 :                    + (GET_CODE (op0) != LSHIFTRT
   13064       127052 :                       ? ((GET_MODE_MASK (mode) >> INTVAL (XEXP (op0, 1)) >> 1)
   13065              :                          + 1)
   13066              :                       : 0))
   13067       127052 :                   <= GET_MODE_MASK (mode) >> INTVAL (XEXP (op0, 1))))
   13068              :             {
   13069       126878 :               unsigned HOST_WIDE_INT low_bits
   13070       126878 :                 = (nonzero_bits (XEXP (op0, 0), mode)
   13071       126878 :                    & ((HOST_WIDE_INT_1U
   13072       126878 :                        << INTVAL (XEXP (op0, 1))) - 1));
   13073       126878 :               if (low_bits == 0 || !equality_comparison_p)
   13074              :                 {
   13075              :                   /* If the shift was logical, then we must make the condition
   13076              :                      unsigned.  */
   13077        19016 :                   if (GET_CODE (op0) == LSHIFTRT)
   13078        16330 :                     code = unsigned_condition (code);
   13079              : 
   13080        19016 :                   const_op = (unsigned HOST_WIDE_INT) const_op
   13081        19016 :                               << INTVAL (XEXP (op0, 1));
   13082        19016 :                   if (low_bits != 0
   13083         3017 :                       && (code == GT || code == GTU
   13084         1029 :                           || code == LE || code == LEU))
   13085         2949 :                     const_op
   13086         2949 :                       |= ((HOST_WIDE_INT_1 << INTVAL (XEXP (op0, 1))) - 1);
   13087        19016 :                   op1 = GEN_INT (const_op);
   13088        19016 :                   op0 = XEXP (op0, 0);
   13089        19016 :                   continue;
   13090              :                 }
   13091              :             }
   13092              : 
   13093              :           /* If we are using this shift to extract just the sign bit, we
   13094              :              can replace this with an LT or GE comparison.  */
   13095       113955 :           if (const_op == 0
   13096        98525 :               && (equality_comparison_p || sign_bit_comparison_p)
   13097        98489 :               && CONST_INT_P (XEXP (op0, 1))
   13098        94014 :               && UINTVAL (XEXP (op0, 1)) == mode_width - 1)
   13099              :             {
   13100        49218 :               op0 = XEXP (op0, 0);
   13101        49218 :               code = (code == NE || code == GT ? LT : GE);
   13102        49218 :               continue;
   13103              :             }
   13104              :           break;
   13105              : 
   13106              :         default:
   13107              :           break;
   13108              :         }
   13109              : 
   13110              :       break;
   13111              :     }
   13112              : 
   13113              :   /* Now make any compound operations involved in this comparison.  Then,
   13114              :      check for an outermost SUBREG on OP0 that is not doing anything or is
   13115              :      paradoxical.  The latter transformation must only be performed when
   13116              :      it is known that the "extra" bits will be the same in op0 and op1 or
   13117              :      that they don't matter.  There are three cases to consider:
   13118              : 
   13119              :      1. SUBREG_REG (op0) is a register.  In this case the bits are don't
   13120              :      care bits and we can assume they have any convenient value.  So
   13121              :      making the transformation is safe.
   13122              : 
   13123              :      2. SUBREG_REG (op0) is a memory and LOAD_EXTEND_OP is UNKNOWN.
   13124              :      In this case the upper bits of op0 are undefined.  We should not make
   13125              :      the simplification in that case as we do not know the contents of
   13126              :      those bits.
   13127              : 
   13128              :      3. SUBREG_REG (op0) is a memory and LOAD_EXTEND_OP is not UNKNOWN.
   13129              :      In that case we know those bits are zeros or ones.  We must also be
   13130              :      sure that they are the same as the upper bits of op1.
   13131              : 
   13132              :      We can never remove a SUBREG for a non-equality comparison because
   13133              :      the sign bit is in a different place in the underlying object.  */
   13134              : 
   13135     24439984 :   rtx_code op0_mco_code = SET;
   13136     24439984 :   if (op1 == const0_rtx)
   13137     11665042 :     op0_mco_code = code == NE || code == EQ ? EQ : COMPARE;
   13138              : 
   13139     24439984 :   op0 = make_compound_operation (op0, op0_mco_code);
   13140     24439984 :   op1 = make_compound_operation (op1, SET);
   13141              : 
   13142       611516 :   if (GET_CODE (op0) == SUBREG && subreg_lowpart_p (op0)
   13143       578781 :       && is_int_mode (GET_MODE (op0), &mode)
   13144       548332 :       && is_int_mode (GET_MODE (SUBREG_REG (op0)), &inner_mode)
   13145     24984548 :       && (code == NE || code == EQ))
   13146              :     {
   13147       289024 :       if (paradoxical_subreg_p (op0))
   13148              :         {
   13149              :           /* For paradoxical subregs, allow case 1 as above.  Case 3 isn't
   13150              :              implemented.  */
   13151            0 :           if (REG_P (SUBREG_REG (op0)))
   13152              :             {
   13153            0 :               op0 = SUBREG_REG (op0);
   13154            0 :               op1 = gen_lowpart (inner_mode, op1);
   13155              :             }
   13156              :         }
   13157       289024 :       else if (GET_MODE_PRECISION (inner_mode) <= HOST_BITS_PER_WIDE_INT
   13158       289024 :                && (nonzero_bits (SUBREG_REG (op0), inner_mode)
   13159       281349 :                    & ~GET_MODE_MASK (mode)) == 0)
   13160              :         {
   13161        13858 :           tem = gen_lowpart (inner_mode, op1);
   13162              : 
   13163        13858 :           if ((nonzero_bits (tem, inner_mode) & ~GET_MODE_MASK (mode)) == 0)
   13164         4536 :             op0 = SUBREG_REG (op0), op1 = tem;
   13165              :         }
   13166              :     }
   13167              : 
   13168              :   /* We now do the opposite procedure: Some machines don't have compare
   13169              :      insns in all modes.  If OP0's mode is an integer mode smaller than a
   13170              :      word and we can't do a compare in that mode, see if there is a larger
   13171              :      mode for which we can do the compare.  There are a number of cases in
   13172              :      which we can use the wider mode.  */
   13173              : 
   13174     24439984 :   if (is_int_mode (GET_MODE (op0), &mode)
   13175     25126552 :       && GET_MODE_SIZE (mode) < UNITS_PER_WORD
   13176      8940262 :       && ! have_insn_for (COMPARE, mode))
   13177            0 :     FOR_EACH_WIDER_MODE (tmode_iter, mode)
   13178              :       {
   13179            0 :         tmode = tmode_iter.require ();
   13180            0 :         if (!HWI_COMPUTABLE_MODE_P (tmode))
   13181              :           break;
   13182            0 :         if (have_insn_for (COMPARE, tmode))
   13183              :           {
   13184            0 :             int zero_extended;
   13185              : 
   13186              :             /* If this is a test for negative, we can make an explicit
   13187              :                test of the sign bit.  Test this first so we can use
   13188              :                a paradoxical subreg to extend OP0.  */
   13189              : 
   13190            0 :             if (op1 == const0_rtx && (code == LT || code == GE)
   13191            0 :                 && HWI_COMPUTABLE_MODE_P (mode))
   13192              :               {
   13193            0 :                 unsigned HOST_WIDE_INT sign
   13194            0 :                   = HOST_WIDE_INT_1U << (GET_MODE_BITSIZE (mode) - 1);
   13195            0 :                 op0 = simplify_gen_binary (AND, tmode,
   13196            0 :                                            gen_lowpart (tmode, op0),
   13197            0 :                                            gen_int_mode (sign, tmode));
   13198            0 :                 code = (code == LT) ? NE : EQ;
   13199              :                 break;
   13200              :               }
   13201              : 
   13202              :             /* If the only nonzero bits in OP0 and OP1 are those in the
   13203              :                narrower mode and this is an equality or unsigned comparison,
   13204              :                we can use the wider mode.  Similarly for sign-extended
   13205              :                values, in which case it is true for all comparisons.  */
   13206            0 :             zero_extended = ((code == EQ || code == NE
   13207            0 :                               || code == GEU || code == GTU
   13208            0 :                               || code == LEU || code == LTU)
   13209            0 :                              && (nonzero_bits (op0, tmode)
   13210            0 :                                  & ~GET_MODE_MASK (mode)) == 0
   13211            0 :                              && ((CONST_INT_P (op1)
   13212            0 :                                   || (nonzero_bits (op1, tmode)
   13213            0 :                                       & ~GET_MODE_MASK (mode)) == 0)));
   13214              : 
   13215            0 :             if (zero_extended
   13216            0 :                 || ((num_sign_bit_copies (op0, tmode)
   13217            0 :                      > (unsigned int) (GET_MODE_PRECISION (tmode)
   13218            0 :                                        - GET_MODE_PRECISION (mode)))
   13219            0 :                     && (num_sign_bit_copies (op1, tmode)
   13220            0 :                         > (unsigned int) (GET_MODE_PRECISION (tmode)
   13221            0 :                                           - GET_MODE_PRECISION (mode)))))
   13222              :               {
   13223              :                 /* If OP0 is an AND and we don't have an AND in MODE either,
   13224              :                    make a new AND in the proper mode.  */
   13225            0 :                 if (GET_CODE (op0) == AND
   13226            0 :                     && !have_insn_for (AND, mode))
   13227            0 :                   op0 = simplify_gen_binary (AND, tmode,
   13228            0 :                                              gen_lowpart (tmode,
   13229              :                                                           XEXP (op0, 0)),
   13230            0 :                                              gen_lowpart (tmode,
   13231              :                                                           XEXP (op0, 1)));
   13232              :                 else
   13233              :                   {
   13234            0 :                     if (zero_extended)
   13235              :                       {
   13236            0 :                         op0 = simplify_gen_unary (ZERO_EXTEND, tmode,
   13237              :                                                   op0, mode);
   13238            0 :                         op1 = simplify_gen_unary (ZERO_EXTEND, tmode,
   13239              :                                                   op1, mode);
   13240              :                       }
   13241              :                     else
   13242              :                       {
   13243            0 :                         op0 = simplify_gen_unary (SIGN_EXTEND, tmode,
   13244              :                                                   op0, mode);
   13245            0 :                         op1 = simplify_gen_unary (SIGN_EXTEND, tmode,
   13246              :                                                   op1, mode);
   13247              :                       }
   13248              :                     break;
   13249              :                   }
   13250              :               }
   13251              :           }
   13252              :       }
   13253              : 
   13254              :   /* We may have changed the comparison operands.  Re-canonicalize.  */
   13255     24439984 :   if (swap_commutative_operands_p (op0, op1))
   13256              :     {
   13257        61950 :       std::swap (op0, op1);
   13258        61950 :       code = swap_condition (code);
   13259              :     }
   13260              : 
   13261              :   /* If this machine only supports a subset of valid comparisons, see if we
   13262              :      can convert an unsupported one into a supported one.  */
   13263     24439984 :   target_canonicalize_comparison (&code, &op0, &op1, 0);
   13264              : 
   13265     24439984 :   *pop0 = op0;
   13266     24439984 :   *pop1 = op1;
   13267              : 
   13268     24439984 :   return code;
   13269              : }
   13270              : 
   13271              : /* Utility function for record_value_for_reg.  Count number of
   13272              :    rtxs in X.  */
   13273              : static int
   13274         2702 : count_rtxs (rtx x)
   13275              : {
   13276         2702 :   enum rtx_code code = GET_CODE (x);
   13277         2702 :   const char *fmt;
   13278         2702 :   int i, j, ret = 1;
   13279              : 
   13280         2702 :   if (GET_RTX_CLASS (code) == RTX_BIN_ARITH
   13281         2702 :       || GET_RTX_CLASS (code) == RTX_COMM_ARITH)
   13282              :     {
   13283           83 :       rtx x0 = XEXP (x, 0);
   13284           83 :       rtx x1 = XEXP (x, 1);
   13285              : 
   13286           83 :       if (x0 == x1)
   13287            0 :         return 1 + 2 * count_rtxs (x0);
   13288              : 
   13289           83 :       if ((GET_RTX_CLASS (GET_CODE (x1)) == RTX_BIN_ARITH
   13290           83 :            || GET_RTX_CLASS (GET_CODE (x1)) == RTX_COMM_ARITH)
   13291            0 :           && (x0 == XEXP (x1, 0) || x0 == XEXP (x1, 1)))
   13292            0 :         return 2 + 2 * count_rtxs (x0)
   13293            0 :                + count_rtxs (x == XEXP (x1, 0)
   13294            0 :                              ? XEXP (x1, 1) : XEXP (x1, 0));
   13295              : 
   13296           83 :       if ((GET_RTX_CLASS (GET_CODE (x0)) == RTX_BIN_ARITH
   13297           83 :            || GET_RTX_CLASS (GET_CODE (x0)) == RTX_COMM_ARITH)
   13298            0 :           && (x1 == XEXP (x0, 0) || x1 == XEXP (x0, 1)))
   13299            0 :         return 2 + 2 * count_rtxs (x1)
   13300            0 :                + count_rtxs (x == XEXP (x0, 0)
   13301            0 :                              ? XEXP (x0, 1) : XEXP (x0, 0));
   13302              :     }
   13303              : 
   13304         2702 :   fmt = GET_RTX_FORMAT (code);
   13305         6073 :   for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
   13306         3371 :     if (fmt[i] == 'e')
   13307         1588 :       ret += count_rtxs (XEXP (x, i));
   13308         1783 :     else if (fmt[i] == 'E')
   13309          208 :       for (j = 0; j < XVECLEN (x, i); j++)
   13310          156 :         ret += count_rtxs (XVECEXP (x, i, j));
   13311              : 
   13312              :   return ret;
   13313              : }
   13314              : 
   13315              : /* Utility function for following routine.  Called when X is part of a value
   13316              :    being stored into last_set_value.  Sets last_set_table_tick
   13317              :    for each register mentioned.  Similar to mention_regs in cse.cc  */
   13318              : 
   13319              : static void
   13320    234340146 : update_table_tick (rtx x)
   13321              : {
   13322    235035481 :   enum rtx_code code = GET_CODE (x);
   13323    235035481 :   const char *fmt = GET_RTX_FORMAT (code);
   13324    235035481 :   int i, j;
   13325              : 
   13326    235035481 :   if (code == REG)
   13327              :     {
   13328     83681061 :       unsigned int regno = REGNO (x);
   13329     83681061 :       unsigned int endregno = END_REGNO (x);
   13330     83681061 :       unsigned int r;
   13331              : 
   13332    167473418 :       for (r = regno; r < endregno; r++)
   13333              :         {
   13334     83792357 :           reg_stat_type *rsp = &reg_stat[r];
   13335     83792357 :           rsp->last_set_table_tick = label_tick;
   13336              :         }
   13337              : 
   13338              :       return;
   13339              :     }
   13340              : 
   13341    391133375 :   for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
   13342    240598859 :     if (fmt[i] == 'e')
   13343              :       {
   13344              :         /* Check for identical subexpressions.  If x contains
   13345              :            identical subexpression we only have to traverse one of
   13346              :            them.  */
   13347    139392494 :         if (i == 0 && ARITHMETIC_P (x))
   13348              :           {
   13349              :             /* Note that at this point x1 has already been
   13350              :                processed.  */
   13351     52668743 :             rtx x0 = XEXP (x, 0);
   13352     52668743 :             rtx x1 = XEXP (x, 1);
   13353              : 
   13354              :             /* If x0 and x1 are identical then there is no need to
   13355              :                process x0.  */
   13356     52668743 :             if (x0 == x1)
   13357              :               break;
   13358              : 
   13359              :             /* If x0 is identical to a subexpression of x1 then while
   13360              :                processing x1, x0 has already been processed.  Thus we
   13361              :                are done with x.  */
   13362     52544319 :             if (ARITHMETIC_P (x1)
   13363       236429 :                 && (x0 == XEXP (x1, 0) || x0 == XEXP (x1, 1)))
   13364              :               break;
   13365              : 
   13366              :             /* If x1 is identical to a subexpression of x0 then we
   13367              :                still have to process the rest of x0.  */
   13368     52544174 :             if (ARITHMETIC_P (x0)
   13369      9470897 :                 && (x1 == XEXP (x0, 0) || x1 == XEXP (x0, 1)))
   13370              :               {
   13371       695335 :                 update_table_tick (XEXP (x0, x1 == XEXP (x0, 0) ? 1 : 0));
   13372       695335 :                 break;
   13373              :               }
   13374              :           }
   13375              : 
   13376    138572590 :         update_table_tick (XEXP (x, i));
   13377              :       }
   13378    101206365 :     else if (fmt[i] == 'E')
   13379     10380021 :       for (j = 0; j < XVECLEN (x, i); j++)
   13380      7673126 :         update_table_tick (XVECEXP (x, i, j));
   13381              : }
   13382              : 
   13383              : /* Record that REG is set to VALUE in insn INSN.  If VALUE is zero, we
   13384              :    are saying that the register is clobbered and we no longer know its
   13385              :    value.  If INSN is zero, don't update reg_stat[].last_set; this is
   13386              :    only permitted with VALUE also zero and is used to invalidate the
   13387              :    register.  */
   13388              : 
   13389              : static void
   13390    115350554 : record_value_for_reg (rtx reg, rtx_insn *insn, rtx value)
   13391              : {
   13392    115350554 :   unsigned int regno = REGNO (reg);
   13393    115350554 :   unsigned int endregno = END_REGNO (reg);
   13394    115350554 :   unsigned int i;
   13395    115350554 :   reg_stat_type *rsp;
   13396              : 
   13397              :   /* If VALUE contains REG and we have a previous value for REG, substitute
   13398              :      the previous value.  */
   13399    115350554 :   if (value && insn && reg_overlap_mentioned_p (reg, value))
   13400              :     {
   13401      6304886 :       rtx tem;
   13402              : 
   13403              :       /* Set things up so get_last_value is allowed to see anything set up to
   13404              :          our insn.  */
   13405      6304886 :       subst_low_luid = DF_INSN_LUID (insn);
   13406      6304886 :       tem = get_last_value (reg);
   13407              : 
   13408              :       /* If TEM is simply a binary operation with two CLOBBERs as operands,
   13409              :          it isn't going to be useful and will take a lot of time to process,
   13410              :          so just use the CLOBBER.  */
   13411              : 
   13412      6304886 :       if (tem)
   13413              :         {
   13414      2492295 :           if (ARITHMETIC_P (tem)
   13415      2258343 :               && GET_CODE (XEXP (tem, 0)) == CLOBBER
   13416      1102711 :               && GET_CODE (XEXP (tem, 1)) == CLOBBER)
   13417              :             tem = XEXP (tem, 0);
   13418      2490839 :           else if (count_occurrences (value, reg, 1) >= 2)
   13419              :             {
   13420              :               /* If there are two or more occurrences of REG in VALUE,
   13421              :                  prevent the value from growing too much.  */
   13422          958 :               if (count_rtxs (tem) > param_max_last_value_rtl)
   13423            0 :                 tem = gen_rtx_CLOBBER (GET_MODE (tem), const0_rtx);
   13424              :             }
   13425              : 
   13426      2492295 :           value = replace_rtx (copy_rtx (value), reg, tem);
   13427              :         }
   13428              :     }
   13429              : 
   13430              :   /* For each register modified, show we don't know its value, that
   13431              :      we don't know about its bitwise content, that its value has been
   13432              :      updated, and that we don't know the location of the death of the
   13433              :      register.  */
   13434    231057677 :   for (i = regno; i < endregno; i++)
   13435              :     {
   13436    115707123 :       rsp = &reg_stat[i];
   13437              : 
   13438    115707123 :       if (insn)
   13439    106245569 :         rsp->last_set = insn;
   13440              : 
   13441    115707123 :       rsp->last_set_value = 0;
   13442    115707123 :       rsp->last_set_mode = VOIDmode;
   13443    115707123 :       rsp->last_set_nonzero_bits = 0;
   13444    115707123 :       rsp->last_set_sign_bit_copies = 0;
   13445    115707123 :       rsp->last_death = 0;
   13446    115707123 :       rsp->truncated_to_mode = VOIDmode;
   13447              :     }
   13448              : 
   13449              :   /* Mark registers that are being referenced in this value.  */
   13450    115350554 :   if (value)
   13451     88094430 :     update_table_tick (value);
   13452              : 
   13453              :   /* Now update the status of each register being set.
   13454              :      If someone is using this register in this block, set this register
   13455              :      to invalid since we will get confused between the two lives in this
   13456              :      basic block.  This makes using this register always invalid.  In cse, we
   13457              :      scan the table to invalidate all entries using this register, but this
   13458              :      is too much work for us.  */
   13459              : 
   13460    231057677 :   for (i = regno; i < endregno; i++)
   13461              :     {
   13462    115707123 :       rsp = &reg_stat[i];
   13463    115707123 :       rsp->last_set_label = label_tick;
   13464    115707123 :       if (!insn
   13465    106245569 :           || (value && rsp->last_set_table_tick >= label_tick_ebb_start))
   13466     20137126 :         rsp->last_set_invalid = true;
   13467              :       else
   13468     95569997 :         rsp->last_set_invalid = false;
   13469              :     }
   13470              : 
   13471              :   /* The value being assigned might refer to X (like in "x++;").  In that
   13472              :      case, we must replace it with (clobber (const_int 0)) to prevent
   13473              :      infinite loops.  */
   13474    115350554 :   rsp = &reg_stat[regno];
   13475    115350554 :   if (value && !get_last_value_validate (&value, insn, label_tick, false))
   13476              :     {
   13477     11371155 :       value = copy_rtx (value);
   13478     11371155 :       if (!get_last_value_validate (&value, insn, label_tick, true))
   13479            0 :         value = 0;
   13480              :     }
   13481              : 
   13482              :   /* For the main register being modified, update the value, the mode, the
   13483              :      nonzero bits, and the number of sign bit copies.  */
   13484              : 
   13485    115350554 :   rsp->last_set_value = value;
   13486              : 
   13487    115350554 :   if (value)
   13488              :     {
   13489     88094430 :       machine_mode mode = GET_MODE (reg);
   13490     88094430 :       subst_low_luid = DF_INSN_LUID (insn);
   13491     88094430 :       rsp->last_set_mode = mode;
   13492     88094430 :       if (GET_MODE_CLASS (mode) == MODE_INT
   13493     88094430 :           && HWI_COMPUTABLE_MODE_P (mode))
   13494     66371390 :         mode = nonzero_bits_mode;
   13495     88094430 :       rsp->last_set_nonzero_bits = nonzero_bits (value, mode);
   13496     88094430 :       rsp->last_set_sign_bit_copies
   13497     88094430 :         = num_sign_bit_copies (value, GET_MODE (reg));
   13498              :     }
   13499    115350554 : }
   13500              : 
   13501              : /* Called via note_stores from record_dead_and_set_regs to handle one
   13502              :    SET or CLOBBER in an insn.  DATA is the instruction in which the
   13503              :    set is occurring.  */
   13504              : 
   13505              : static void
   13506    138012178 : record_dead_and_set_regs_1 (rtx dest, const_rtx setter, void *data)
   13507              : {
   13508    138012178 :   rtx_insn *record_dead_insn = (rtx_insn *) data;
   13509              : 
   13510    138012178 :   if (GET_CODE (dest) == SUBREG)
   13511            5 :     dest = SUBREG_REG (dest);
   13512              : 
   13513    138012178 :   if (!record_dead_insn)
   13514              :     {
   13515      4290293 :       if (REG_P (dest))
   13516      4290293 :         record_value_for_reg (dest, NULL, NULL_RTX);
   13517              :       return;
   13518              :     }
   13519              : 
   13520    133721885 :   if (REG_P (dest))
   13521              :     {
   13522              :       /* If we are setting the whole register, we know its value.  */
   13523    106070147 :       if (GET_CODE (setter) == SET && dest == SET_DEST (setter))
   13524     87911130 :         record_value_for_reg (dest, record_dead_insn, SET_SRC (setter));
   13525              :       /* We can handle a SUBREG if it's the low part, but we must be
   13526              :          careful with paradoxical SUBREGs on RISC architectures because
   13527              :          we cannot strip e.g. an extension around a load and record the
   13528              :          naked load since the RTL middle-end considers that the upper bits
   13529              :          are defined according to LOAD_EXTEND_OP.  */
   13530     18159017 :       else if (GET_CODE (setter) == SET
   13531       632292 :                && GET_CODE (SET_DEST (setter)) == SUBREG
   13532       619833 :                && SUBREG_REG (SET_DEST (setter)) == dest
   13533       987787 :                && known_le (GET_MODE_PRECISION (GET_MODE (dest)),
   13534              :                             BITS_PER_WORD)
   13535     18293672 :                && subreg_lowpart_p (SET_DEST (setter)))
   13536              :         {
   13537       134655 :           if (WORD_REGISTER_OPERATIONS
   13538              :               && word_register_operation_p (SET_SRC (setter))
   13539              :               && paradoxical_subreg_p (SET_DEST (setter)))
   13540              :             record_value_for_reg (dest, record_dead_insn, SET_SRC (setter));
   13541       134655 :           else if (!partial_subreg_p (SET_DEST (setter)))
   13542       122523 :             record_value_for_reg (dest, record_dead_insn,
   13543       122523 :                                   gen_lowpart (GET_MODE (dest),
   13544       122523 :                                                SET_SRC (setter)));
   13545              :           else
   13546              :             {
   13547        12132 :               record_value_for_reg (dest, record_dead_insn,
   13548        12132 :                                     gen_lowpart (GET_MODE (dest),
   13549        12132 :                                                  SET_SRC (setter)));
   13550              : 
   13551        12132 :               unsigned HOST_WIDE_INT mask;
   13552        12132 :               reg_stat_type *rsp = &reg_stat[REGNO (dest)];
   13553        12132 :               mask = GET_MODE_MASK (GET_MODE (SET_DEST (setter)));
   13554        12132 :               rsp->last_set_nonzero_bits |= ~mask;
   13555        12132 :               rsp->last_set_sign_bit_copies = 1;
   13556              :             }
   13557              :         }
   13558              :       /* Otherwise show that we don't know the value.  */
   13559              :       else
   13560     18024362 :         record_value_for_reg (dest, record_dead_insn, NULL_RTX);
   13561              :     }
   13562     27651738 :   else if (MEM_P (dest)
   13563              :            /* Ignore pushes, they clobber nothing.  */
   13564     27651738 :            && ! push_operand (dest, GET_MODE (dest)))
   13565     14319774 :     mem_last_set = DF_INSN_LUID (record_dead_insn);
   13566              : }
   13567              : 
   13568              : /* Update the records of when each REG was most recently set or killed
   13569              :    for the things done by INSN.  This is the last thing done in processing
   13570              :    INSN in the combiner loop.
   13571              : 
   13572              :    We update reg_stat[], in particular fields last_set, last_set_value,
   13573              :    last_set_mode, last_set_nonzero_bits, last_set_sign_bit_copies,
   13574              :    last_death, and also the similar information mem_last_set (which insn
   13575              :    most recently modified memory) and last_call_luid (which insn was the
   13576              :    most recent subroutine call).  */
   13577              : 
   13578              : static void
   13579    179087163 : record_dead_and_set_regs (rtx_insn *insn)
   13580              : {
   13581    179087163 :   rtx link;
   13582    179087163 :   unsigned int i;
   13583              : 
   13584    318579802 :   for (link = REG_NOTES (insn); link; link = XEXP (link, 1))
   13585              :     {
   13586    139492639 :       if (REG_NOTE_KIND (link) == REG_DEAD
   13587     79659247 :           && REG_P (XEXP (link, 0)))
   13588              :         {
   13589     79659247 :           unsigned int regno = REGNO (XEXP (link, 0));
   13590     79659247 :           unsigned int endregno = END_REGNO (XEXP (link, 0));
   13591              : 
   13592    159524469 :           for (i = regno; i < endregno; i++)
   13593              :             {
   13594     79865222 :               reg_stat_type *rsp;
   13595              : 
   13596     79865222 :               rsp = &reg_stat[i];
   13597     79865222 :               rsp->last_death = insn;
   13598              :             }
   13599              :         }
   13600     59833392 :       else if (REG_NOTE_KIND (link) == REG_INC)
   13601            0 :         record_value_for_reg (XEXP (link, 0), insn, NULL_RTX);
   13602              :     }
   13603              : 
   13604    179087163 :   if (auto call = dyn_cast<rtx_call_insn *> (insn))
   13605              :     {
   13606      9598322 :       HARD_REG_SET callee_clobbers
   13607      9598322 :         = insn_callee_abi (call).full_and_partial_reg_clobbers ();
   13608      9598322 :       hard_reg_set_iterator hrsi;
   13609    812587158 :       EXECUTE_IF_SET_IN_HARD_REG_SET (callee_clobbers, 0, i, hrsi)
   13610              :         {
   13611    802988836 :           reg_stat_type *rsp;
   13612              : 
   13613              :           /* ??? We could try to preserve some information from the last
   13614              :              set of register I if the call doesn't actually clobber
   13615              :              (reg:last_set_mode I), which might be true for ABIs with
   13616              :              partial clobbers.  However, it would be difficult to
   13617              :              update last_set_nonzero_bits and last_sign_bit_copies
   13618              :              to account for the part of I that actually was clobbered.
   13619              :              It wouldn't help much anyway, since we rarely see this
   13620              :              situation before RA.  */
   13621    802988836 :           rsp = &reg_stat[i];
   13622    802988836 :           rsp->last_set_invalid = true;
   13623    802988836 :           rsp->last_set = call;
   13624    802988836 :           rsp->last_set_value = 0;
   13625    802988836 :           rsp->last_set_mode = VOIDmode;
   13626    802988836 :           rsp->last_set_nonzero_bits = 0;
   13627    802988836 :           rsp->last_set_sign_bit_copies = 0;
   13628    802988836 :           rsp->last_death = 0;
   13629    802988836 :           rsp->truncated_to_mode = VOIDmode;
   13630              :         }
   13631              : 
   13632      9598322 :       last_call_luid = mem_last_set = DF_INSN_LUID (call);
   13633              : 
   13634              :       /* We can't combine into a call pattern.  Remember, though, that
   13635              :          the return value register is set at this LUID.  We could
   13636              :          still replace a register with the return value from the
   13637              :          wrong subroutine call!  */
   13638      9598322 :       note_stores (call, record_dead_and_set_regs_1, NULL_RTX);
   13639              :     }
   13640              :   else
   13641    169488841 :     note_stores (insn, record_dead_and_set_regs_1, insn);
   13642    179087163 : }
   13643              : 
   13644              : /* If a SUBREG has the promoted bit set, it is in fact a property of the
   13645              :    register present in the SUBREG, so for each such SUBREG go back and
   13646              :    adjust nonzero and sign bit information of the registers that are
   13647              :    known to have some zero/sign bits set.
   13648              : 
   13649              :    This is needed because when combine blows the SUBREGs away, the
   13650              :    information on zero/sign bits is lost and further combines can be
   13651              :    missed because of that.  */
   13652              : 
   13653              : static void
   13654         7373 : record_promoted_value (rtx_insn *insn, rtx subreg)
   13655              : {
   13656         7373 :   struct insn_link *links;
   13657         7373 :   rtx set;
   13658         7373 :   unsigned int regno = REGNO (SUBREG_REG (subreg));
   13659         7373 :   machine_mode mode = GET_MODE (subreg);
   13660              : 
   13661         7373 :   if (!HWI_COMPUTABLE_MODE_P (mode))
   13662              :     return;
   13663              : 
   13664         8684 :   for (links = LOG_LINKS (insn); links;)
   13665              :     {
   13666         7787 :       reg_stat_type *rsp;
   13667              : 
   13668         7787 :       insn = links->insn;
   13669         7787 :       set = single_set (insn);
   13670              : 
   13671         7787 :       if (! set || !REG_P (SET_DEST (set))
   13672         7783 :           || REGNO (SET_DEST (set)) != regno
   13673        14405 :           || GET_MODE (SET_DEST (set)) != GET_MODE (SUBREG_REG (subreg)))
   13674              :         {
   13675         1169 :           links = links->next;
   13676         1169 :           continue;
   13677              :         }
   13678              : 
   13679         6618 :       rsp = &reg_stat[regno];
   13680         6618 :       if (rsp->last_set == insn)
   13681              :         {
   13682         6618 :           if (SUBREG_PROMOTED_UNSIGNED_P (subreg))
   13683         6618 :             rsp->last_set_nonzero_bits &= GET_MODE_MASK (mode);
   13684              :         }
   13685              : 
   13686         6618 :       if (REG_P (SET_SRC (set)))
   13687              :         {
   13688          142 :           regno = REGNO (SET_SRC (set));
   13689          142 :           links = LOG_LINKS (insn);
   13690              :         }
   13691              :       else
   13692              :         break;
   13693              :     }
   13694              : }
   13695              : 
   13696              : /* Check if X, a register, is known to contain a value already
   13697              :    truncated to MODE.  In this case we can use a subreg to refer to
   13698              :    the truncated value even though in the generic case we would need
   13699              :    an explicit truncation.  */
   13700              : 
   13701              : static bool
   13702            0 : reg_truncated_to_mode (machine_mode mode, const_rtx x)
   13703              : {
   13704            0 :   reg_stat_type *rsp = &reg_stat[REGNO (x)];
   13705            0 :   machine_mode truncated = rsp->truncated_to_mode;
   13706              : 
   13707            0 :   if (truncated == 0
   13708            0 :       || rsp->truncation_label < label_tick_ebb_start)
   13709              :     return false;
   13710            0 :   if (!partial_subreg_p (mode, truncated))
   13711              :     return true;
   13712            0 :   if (TRULY_NOOP_TRUNCATION_MODES_P (mode, truncated))
   13713              :     return true;
   13714              :   return false;
   13715              : }
   13716              : 
   13717              : /* If X is a hard reg or a subreg record the mode that the register is
   13718              :    accessed in.  For non-TARGET_TRULY_NOOP_TRUNCATION targets we might be
   13719              :    able to turn a truncate into a subreg using this information.  Return true
   13720              :    if traversing X is complete.  */
   13721              : 
   13722              : static bool
   13723    202344462 : record_truncated_value (rtx x)
   13724              : {
   13725    202344462 :   machine_mode truncated_mode;
   13726    202344462 :   reg_stat_type *rsp;
   13727              : 
   13728    202344462 :   if (GET_CODE (x) == SUBREG && REG_P (SUBREG_REG (x)))
   13729              :     {
   13730      1886022 :       machine_mode original_mode = GET_MODE (SUBREG_REG (x));
   13731      1886022 :       truncated_mode = GET_MODE (x);
   13732              : 
   13733      1886022 :       if (!partial_subreg_p (truncated_mode, original_mode))
   13734              :         return true;
   13735              : 
   13736      1128076 :       truncated_mode = GET_MODE (x);
   13737      1128076 :       if (TRULY_NOOP_TRUNCATION_MODES_P (truncated_mode, original_mode))
   13738              :         return true;
   13739              : 
   13740            0 :       x = SUBREG_REG (x);
   13741            0 :     }
   13742              :   /* ??? For hard-regs we now record everything.  We might be able to
   13743              :      optimize this using last_set_mode.  */
   13744    200458440 :   else if (REG_P (x) && REGNO (x) < FIRST_PSEUDO_REGISTER)
   13745     21250546 :     truncated_mode = GET_MODE (x);
   13746              :   else
   13747              :     return false;
   13748              : 
   13749     21250546 :   rsp = &reg_stat[REGNO (x)];
   13750     21250546 :   if (rsp->truncated_to_mode == 0
   13751      9872773 :       || rsp->truncation_label < label_tick_ebb_start
   13752     29854397 :       || partial_subreg_p (truncated_mode, rsp->truncated_to_mode))
   13753              :     {
   13754     12647337 :       rsp->truncated_to_mode = truncated_mode;
   13755     12647337 :       rsp->truncation_label = label_tick;
   13756              :     }
   13757              : 
   13758              :   return true;
   13759              : }
   13760              : 
   13761              : /* Callback for note_uses.  Find hardregs and subregs of pseudos and
   13762              :    the modes they are used in.  This can help turning TRUNCATEs into
   13763              :    SUBREGs.  */
   13764              : 
   13765              : static void
   13766     77566740 : record_truncated_values (rtx *loc, void *data ATTRIBUTE_UNUSED)
   13767              : {
   13768     77566740 :   subrtx_var_iterator::array_type array;
   13769    279911202 :   FOR_EACH_SUBRTX_VAR (iter, array, *loc, NONCONST)
   13770    202344462 :     if (record_truncated_value (*iter))
   13771     23136568 :       iter.skip_subrtxes ();
   13772     77566740 : }
   13773              : 
   13774              : /* Scan X for promoted SUBREGs.  For each one found,
   13775              :    note what it implies to the registers used in it.  */
   13776              : 
   13777              : static void
   13778    368902469 : check_promoted_subreg (rtx_insn *insn, rtx x)
   13779              : {
   13780    368902469 :   if (GET_CODE (x) == SUBREG
   13781      2278834 :       && SUBREG_PROMOTED_VAR_P (x)
   13782    368909842 :       && REG_P (SUBREG_REG (x)))
   13783         7373 :     record_promoted_value (insn, x);
   13784              :   else
   13785              :     {
   13786    368895096 :       const char *format = GET_RTX_FORMAT (GET_CODE (x));
   13787    368895096 :       int i, j;
   13788              : 
   13789    887863142 :       for (i = 0; i < GET_RTX_LENGTH (GET_CODE (x)); i++)
   13790    518968046 :         switch (format[i])
   13791              :           {
   13792    275864170 :           case 'e':
   13793    275864170 :             check_promoted_subreg (insn, XEXP (x, i));
   13794    275864170 :             break;
   13795     12007428 :           case 'V':
   13796     12007428 :           case 'E':
   13797     12007428 :             if (XVEC (x, i) != 0)
   13798     37373204 :               for (j = 0; j < XVECLEN (x, i); j++)
   13799     25365776 :                 check_promoted_subreg (insn, XVECEXP (x, i, j));
   13800              :             break;
   13801              :           }
   13802              :     }
   13803    368902469 : }
   13804              : 
   13805              : /* Verify that all the registers and memory references mentioned in *LOC are
   13806              :    still valid.  *LOC was part of a value set in INSN when label_tick was
   13807              :    equal to TICK.  Return false if some are not.  If REPLACE is true, replace
   13808              :    the invalid references with (clobber (const_int 0)) and return true.  This
   13809              :    replacement is useful because we often can get useful information about
   13810              :    the form of a value (e.g., if it was produced by a shift that always
   13811              :    produces -1 or 0) even though we don't know exactly what registers it
   13812              :    was produced from.  */
   13813              : 
   13814              : static bool
   13815    484593259 : get_last_value_validate (rtx *loc, rtx_insn *insn, int tick, bool replace)
   13816              : {
   13817    484593822 :   rtx x = *loc;
   13818    484593822 :   const char *fmt = GET_RTX_FORMAT (GET_CODE (x));
   13819    484593822 :   int len = GET_RTX_LENGTH (GET_CODE (x));
   13820    484593822 :   int i, j;
   13821              : 
   13822    484593822 :   if (REG_P (x))
   13823              :     {
   13824    162578230 :       unsigned int regno = REGNO (x);
   13825    162578230 :       unsigned int endregno = END_REGNO (x);
   13826    162578230 :       unsigned int j;
   13827              : 
   13828    301197798 :       for (j = regno; j < endregno; j++)
   13829              :         {
   13830    162603745 :           reg_stat_type *rsp = &reg_stat[j];
   13831    162603745 :           if (rsp->last_set_invalid
   13832              :               /* If this is a pseudo-register that was only set once and not
   13833              :                  live at the beginning of the function, it is always valid.  */
   13834    270213963 :               || (! (regno >= FIRST_PSEUDO_REGISTER
   13835    123773126 :                      && regno < reg_n_sets_max
   13836    123752115 :                      && REG_N_SETS (regno) == 1
   13837    215220436 :                      && (!REGNO_REG_SET_P
   13838              :                          (DF_LR_IN (ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb),
   13839              :                           regno)))
   13840     31235670 :                   && rsp->last_set_label > tick))
   13841              :           {
   13842     23984177 :             if (replace)
   13843     12122407 :               *loc = gen_rtx_CLOBBER (GET_MODE (x), const0_rtx);
   13844              :             return replace;
   13845              :           }
   13846              :         }
   13847              : 
   13848              :       return true;
   13849              :     }
   13850              :   /* If this is a memory reference, make sure that there were no stores after
   13851              :      it that might have clobbered the value.  We don't have alias info, so we
   13852              :      assume any store invalidates it.  Moreover, we only have local UIDs, so
   13853              :      we also assume that there were stores in the intervening basic blocks.  */
   13854     35798644 :   else if (MEM_P (x) && !MEM_READONLY_P (x)
   13855    355665180 :            && (tick != label_tick || DF_INSN_LUID (insn) <= mem_last_set))
   13856              :     {
   13857      8077256 :       if (replace)
   13858      4042288 :         *loc = gen_rtx_CLOBBER (GET_MODE (x), const0_rtx);
   13859              :       return replace;
   13860              :     }
   13861              : 
   13862    783614041 :   for (i = 0; i < len; i++)
   13863              :     {
   13864    481849850 :       if (fmt[i] == 'e')
   13865              :         {
   13866              :           /* Check for identical subexpressions.  If x contains
   13867              :              identical subexpression we only have to traverse one of
   13868              :              them.  */
   13869    294723622 :           if (i == 1 && ARITHMETIC_P (x))
   13870              :             {
   13871              :               /* Note that at this point x0 has already been checked
   13872              :                  and found valid.  */
   13873    105633247 :               rtx x0 = XEXP (x, 0);
   13874    105633247 :               rtx x1 = XEXP (x, 1);
   13875              : 
   13876              :               /* If x0 and x1 are identical then x is also valid.  */
   13877    105633247 :               if (x0 == x1)
   13878              :                 return true;
   13879              : 
   13880              :               /* If x1 is identical to a subexpression of x0 then
   13881              :                  while checking x0, x1 has already been checked.  Thus
   13882              :                  it is valid and so as x.  */
   13883    105241175 :               if (ARITHMETIC_P (x0)
   13884     19714264 :                   && (x1 == XEXP (x0, 0) || x1 == XEXP (x0, 1)))
   13885              :                 return true;
   13886              : 
   13887              :               /* If x0 is identical to a subexpression of x1 then x is
   13888              :                  valid iff the rest of x1 is valid.  */
   13889    103164293 :               if (ARITHMETIC_P (x1)
   13890       379471 :                   && (x0 == XEXP (x1, 0) || x0 == XEXP (x1, 1)))
   13891          563 :                 return
   13892          563 :                   get_last_value_validate (&XEXP (x1,
   13893              :                                                   x0 == XEXP (x1, 0) ? 1 : 0),
   13894          563 :                                            insn, tick, replace);
   13895              :             }
   13896              : 
   13897    292254105 :           if (!get_last_value_validate (&XEXP (x, i), insn, tick, replace))
   13898              :             return false;
   13899              :         }
   13900    187126228 :       else if (fmt[i] == 'E')
   13901     33544643 :         for (j = 0; j < XVECLEN (x, i); j++)
   13902     26595254 :           if (!get_last_value_validate (&XVECEXP (x, i, j),
   13903              :                                         insn, tick, replace))
   13904              :             return false;
   13905              :     }
   13906              : 
   13907              :   /* If we haven't found a reason for it to be invalid, it is valid.  */
   13908              :   return true;
   13909              : }
   13910              : 
   13911              : /* Get the last value assigned to X, if known.  Some registers
   13912              :    in the value may be replaced with (clobber (const_int 0)) if their value
   13913              :    is known longer known reliably.  */
   13914              : 
   13915              : static rtx
   13916    238867452 : get_last_value (const_rtx x)
   13917              : {
   13918    238867452 :   unsigned int regno;
   13919    238867452 :   rtx value;
   13920    238867452 :   reg_stat_type *rsp;
   13921              : 
   13922              :   /* If this is a non-paradoxical SUBREG, get the value of its operand and
   13923              :      then convert it to the desired mode.  If this is a paradoxical SUBREG,
   13924              :      we cannot predict what values the "extra" bits might have.  */
   13925    238867452 :   if (GET_CODE (x) == SUBREG
   13926     14228647 :       && subreg_lowpart_p (x)
   13927     13680537 :       && !paradoxical_subreg_p (x)
   13928    247562969 :       && (value = get_last_value (SUBREG_REG (x))) != 0)
   13929      4439598 :     return gen_lowpart (GET_MODE (x), value);
   13930              : 
   13931    234427854 :   if (!REG_P (x))
   13932              :     return 0;
   13933              : 
   13934    203090992 :   regno = REGNO (x);
   13935    203090992 :   rsp = &reg_stat[regno];
   13936    203090992 :   value = rsp->last_set_value;
   13937              : 
   13938              :   /* If we don't have a value, or if it isn't for this basic block and
   13939              :      it's either a hard register, set more than once, or it's a live
   13940              :      at the beginning of the function, return 0.
   13941              : 
   13942              :      Because if it's not live at the beginning of the function then the reg
   13943              :      is always set before being used (is never used without being set).
   13944              :      And, if it's set only once, and it's always set before use, then all
   13945              :      uses must have the same last value, even if it's not from this basic
   13946              :      block.  */
   13947              : 
   13948    203090992 :   if (value == 0
   13949    203090992 :       || (rsp->last_set_label < label_tick_ebb_start
   13950     76659852 :           && (regno < FIRST_PSEUDO_REGISTER
   13951     75800181 :               || regno >= reg_n_sets_max
   13952     75800181 :               || REG_N_SETS (regno) != 1
   13953     16880280 :               || REGNO_REG_SET_P
   13954              :                  (DF_LR_IN (ENTRY_BLOCK_PTR_FOR_FN (cfun)->next_bb), regno))))
   13955              :     return 0;
   13956              : 
   13957              :   /* If the value was set in a later insn than the ones we are processing,
   13958              :      we can't use it even if the register was only set once.  */
   13959     82793326 :   if (rsp->last_set_label == label_tick
   13960     82793326 :       && DF_INSN_LUID (rsp->last_set) >= subst_low_luid)
   13961              :     return 0;
   13962              : 
   13963              :   /* If fewer bits were set than what we are asked for now, we cannot use
   13964              :      the value.  */
   13965     61754154 :   if (maybe_lt (GET_MODE_PRECISION (rsp->last_set_mode),
   13966     61754154 :                 GET_MODE_PRECISION (GET_MODE (x))))
   13967              :     return 0;
   13968              : 
   13969              :   /* If the value has all its registers valid, return it.  */
   13970     61752732 :   if (get_last_value_validate (&value, rsp->last_set,
   13971              :                                rsp->last_set_label, false))
   13972     57227149 :     return value;
   13973              : 
   13974              :   /* Otherwise, make a copy and replace any invalid register with
   13975              :      (clobber (const_int 0)).  If that fails for some reason, return 0.  */
   13976              : 
   13977      4525583 :   value = copy_rtx (value);
   13978      4525583 :   if (get_last_value_validate (&value, rsp->last_set,
   13979              :                                rsp->last_set_label, true))
   13980      4525583 :     return value;
   13981              : 
   13982              :   return 0;
   13983              : }
   13984              : 
   13985              : /* Define three variables used for communication between the following
   13986              :    routines.  */
   13987              : 
   13988              : static unsigned int reg_dead_regno, reg_dead_endregno;
   13989              : static int reg_dead_flag;
   13990              : rtx reg_dead_reg;
   13991              : 
   13992              : /* Function called via note_stores from reg_dead_at_p.
   13993              : 
   13994              :    If DEST is within [reg_dead_regno, reg_dead_endregno), set
   13995              :    reg_dead_flag to 1 if X is a CLOBBER and to -1 it is a SET.  */
   13996              : 
   13997              : static void
   13998       650189 : reg_dead_at_p_1 (rtx dest, const_rtx x, void *data ATTRIBUTE_UNUSED)
   13999              : {
   14000       650189 :   unsigned int regno, endregno;
   14001              : 
   14002       650189 :   if (!REG_P (dest))
   14003              :     return;
   14004              : 
   14005       597761 :   regno = REGNO (dest);
   14006       597761 :   endregno = END_REGNO (dest);
   14007       597761 :   if (reg_dead_endregno > regno && reg_dead_regno < endregno)
   14008       293562 :     reg_dead_flag = (GET_CODE (x) == CLOBBER) ? 1 : -1;
   14009              : }
   14010              : 
   14011              : /* Return true if REG is known to be dead at INSN.
   14012              : 
   14013              :    We scan backwards from INSN.  If we hit a REG_DEAD note or a CLOBBER
   14014              :    referencing REG, it is dead.  If we hit a SET referencing REG, it is
   14015              :    live.  Otherwise, see if it is live or dead at the start of the basic
   14016              :    block we are in.  Hard regs marked as being live in NEWPAT_USED_REGS
   14017              :    must be assumed to be always live.  */
   14018              : 
   14019              : static bool
   14020      1641145 : reg_dead_at_p (rtx reg, rtx_insn *insn)
   14021              : {
   14022      1641145 :   basic_block block;
   14023      1641145 :   unsigned int i;
   14024              : 
   14025              :   /* Set variables for reg_dead_at_p_1.  */
   14026      1641145 :   reg_dead_regno = REGNO (reg);
   14027      1641145 :   reg_dead_endregno = END_REGNO (reg);
   14028      1641145 :   reg_dead_reg = reg;
   14029              : 
   14030      1641145 :   reg_dead_flag = 0;
   14031              : 
   14032              :   /* Check that reg isn't mentioned in NEWPAT_USED_REGS.  For fixed registers
   14033              :      we allow the machine description to decide whether use-and-clobber
   14034              :      patterns are OK.  */
   14035      1641145 :   if (reg_dead_regno < FIRST_PSEUDO_REGISTER)
   14036              :     {
   14037      3282290 :       for (i = reg_dead_regno; i < reg_dead_endregno; i++)
   14038      1641145 :         if (!fixed_regs[i] && TEST_HARD_REG_BIT (newpat_used_regs, i))
   14039              :           return false;
   14040              :     }
   14041              : 
   14042              :   /* Scan backwards until we find a REG_DEAD note, SET, CLOBBER, or
   14043              :      beginning of basic block.  */
   14044      1641145 :   block = BLOCK_FOR_INSN (insn);
   14045       801583 :   for (;;)
   14046              :     {
   14047      2442728 :       if (INSN_P (insn))
   14048              :         {
   14049      2289476 :           if (find_regno_note (insn, REG_UNUSED, reg_dead_regno))
   14050              :             return true;
   14051              : 
   14052       855162 :           note_stores (insn, reg_dead_at_p_1, NULL);
   14053       855162 :           if (reg_dead_flag)
   14054       146781 :             return reg_dead_flag == 1 ? 1 : 0;
   14055              : 
   14056       708381 :           if (find_regno_note (insn, REG_DEAD, reg_dead_regno))
   14057              :             return true;
   14058              :         }
   14059              : 
   14060       831793 :       if (insn == BB_HEAD (block))
   14061              :         break;
   14062              : 
   14063       801583 :       insn = PREV_INSN (insn);
   14064              :     }
   14065              : 
   14066              :   /* Look at live-in sets for the basic block that we were in.  */
   14067        60420 :   for (i = reg_dead_regno; i < reg_dead_endregno; i++)
   14068        30210 :     if (REGNO_REG_SET_P (df_get_live_in (block), i))
   14069              :       return false;
   14070              : 
   14071              :   return true;
   14072              : }
   14073              : 
   14074              : /* Note hard registers in X that are used.  */
   14075              : 
   14076              : static void
   14077    295494114 : mark_used_regs_combine (rtx x)
   14078              : {
   14079    341383421 :   RTX_CODE code = GET_CODE (x);
   14080    341383421 :   unsigned int regno;
   14081    341383421 :   int i;
   14082              : 
   14083    341383421 :   switch (code)
   14084              :     {
   14085              :     case LABEL_REF:
   14086              :     case SYMBOL_REF:
   14087              :     case CONST:
   14088              :     CASE_CONST_ANY:
   14089              :     case PC:
   14090              :     case ADDR_VEC:
   14091              :     case ADDR_DIFF_VEC:
   14092              :     case ASM_INPUT:
   14093              :       return;
   14094              : 
   14095      7601115 :     case CLOBBER:
   14096              :       /* If we are clobbering a MEM, mark any hard registers inside the
   14097              :          address as used.  */
   14098      7601115 :       if (MEM_P (XEXP (x, 0)))
   14099         5683 :         mark_used_regs_combine (XEXP (XEXP (x, 0), 0));
   14100              :       return;
   14101              : 
   14102     77729794 :     case REG:
   14103     77729794 :       regno = REGNO (x);
   14104              :       /* A hard reg in a wide mode may really be multiple registers.
   14105              :          If so, mark all of them just like the first.  */
   14106     77729794 :       if (regno < FIRST_PSEUDO_REGISTER)
   14107              :         {
   14108              :           /* None of this applies to the stack, frame or arg pointers.  */
   14109      9254801 :           if (regno == STACK_POINTER_REGNUM
   14110      9254801 :               || (!HARD_FRAME_POINTER_IS_FRAME_POINTER
   14111              :                   && regno == HARD_FRAME_POINTER_REGNUM)
   14112      8335871 :               || (FRAME_POINTER_REGNUM != ARG_POINTER_REGNUM
   14113      1131128 :                   && regno == ARG_POINTER_REGNUM && fixed_regs[regno])
   14114      7204743 :               || regno == FRAME_POINTER_REGNUM)
   14115              :             return;
   14116              : 
   14117      1787657 :           add_to_hard_reg_set (&newpat_used_regs, GET_MODE (x), regno);
   14118              :         }
   14119              :       return;
   14120              : 
   14121     45883624 :     case SET:
   14122     45883624 :       {
   14123              :         /* If setting a MEM, or a SUBREG of a MEM, then note any hard regs in
   14124              :            the address.  */
   14125     45883624 :         rtx testreg = SET_DEST (x);
   14126              : 
   14127     45883624 :         while (GET_CODE (testreg) == SUBREG
   14128     45899072 :                || GET_CODE (testreg) == ZERO_EXTRACT
   14129     92115733 :                || GET_CODE (testreg) == STRICT_LOW_PART)
   14130       340815 :           testreg = XEXP (testreg, 0);
   14131              : 
   14132     45883624 :         if (MEM_P (testreg))
   14133      4869966 :           mark_used_regs_combine (XEXP (testreg, 0));
   14134              : 
   14135     45883624 :         mark_used_regs_combine (SET_SRC (x));
   14136              :       }
   14137     45883624 :       return;
   14138              : 
   14139    136426601 :     default:
   14140    136426601 :       break;
   14141              :     }
   14142              : 
   14143              :   /* Recursively scan the operands of this expression.  */
   14144              : 
   14145    136426601 :   {
   14146    136426601 :     const char *fmt = GET_RTX_FORMAT (code);
   14147              : 
   14148    395718503 :     for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
   14149              :       {
   14150    259291902 :         if (fmt[i] == 'e')
   14151    210184929 :           mark_used_regs_combine (XEXP (x, i));
   14152     49106973 :         else if (fmt[i] == 'E')
   14153              :           {
   14154              :             int j;
   14155              : 
   14156     67489762 :             for (j = 0; j < XVECLEN (x, i); j++)
   14157     46862754 :               mark_used_regs_combine (XVECEXP (x, i, j));
   14158              :           }
   14159              :       }
   14160              :   }
   14161              : }
   14162              : 
   14163              : /* Remove register number REGNO from the dead registers list of INSN.
   14164              : 
   14165              :    Return the note used to record the death, if there was one.  */
   14166              : 
   14167              : rtx
   14168      3211912 : remove_death (unsigned int regno, rtx_insn *insn)
   14169              : {
   14170      3211912 :   rtx note = find_regno_note (insn, REG_DEAD, regno);
   14171              : 
   14172      3211912 :   if (note)
   14173       522578 :     remove_note (insn, note);
   14174              : 
   14175      3211912 :   return note;
   14176              : }
   14177              : 
   14178              : /* For each register (hardware or pseudo) used within expression X, if its
   14179              :    death is in an instruction with luid between FROM_LUID (inclusive) and
   14180              :    TO_INSN (exclusive), put a REG_DEAD note for that register in the
   14181              :    list headed by PNOTES.
   14182              : 
   14183              :    That said, don't move registers killed by maybe_kill_insn.
   14184              : 
   14185              :    This is done when X is being merged by combination into TO_INSN.  These
   14186              :    notes will then be distributed as needed.  */
   14187              : 
   14188              : static void
   14189     24813521 : move_deaths (rtx x, rtx maybe_kill_insn, int from_luid, rtx_insn *to_insn,
   14190              :              rtx *pnotes)
   14191              : {
   14192     25350353 :   const char *fmt;
   14193     25350353 :   int len, i;
   14194     25350353 :   enum rtx_code code = GET_CODE (x);
   14195              : 
   14196     25350353 :   if (code == REG)
   14197              :     {
   14198      6260776 :       unsigned int regno = REGNO (x);
   14199      6260776 :       rtx_insn *where_dead = reg_stat[regno].last_death;
   14200              : 
   14201              :       /* If we do not know where the register died, it may still die between
   14202              :          FROM_LUID and TO_INSN.  If so, find it.  This is PR83304.  */
   14203      6260776 :       if (!where_dead || DF_INSN_LUID (where_dead) >= DF_INSN_LUID (to_insn))
   14204              :         {
   14205      3400570 :           rtx_insn *insn = prev_real_nondebug_insn (to_insn);
   14206      3400570 :           while (insn
   14207      5103050 :                  && BLOCK_FOR_INSN (insn) == BLOCK_FOR_INSN (to_insn)
   14208      9447169 :                  && DF_INSN_LUID (insn) >= from_luid)
   14209              :             {
   14210      2307534 :               if (dead_or_set_regno_p (insn, regno))
   14211              :                 {
   14212       573546 :                   if (find_regno_note (insn, REG_DEAD, regno))
   14213      6260776 :                     where_dead = insn;
   14214              :                   break;
   14215              :                 }
   14216              : 
   14217      1733988 :               insn = prev_real_nondebug_insn (insn);
   14218              :             }
   14219              :         }
   14220              : 
   14221              :       /* Don't move the register if it gets killed in between from and to.  */
   14222       150369 :       if (maybe_kill_insn && reg_set_p (x, maybe_kill_insn)
   14223      6303065 :           && ! reg_referenced_p (x, maybe_kill_insn))
   14224              :         return;
   14225              : 
   14226      6218487 :       if (where_dead
   14227      3220107 :           && BLOCK_FOR_INSN (where_dead) == BLOCK_FOR_INSN (to_insn)
   14228      3059122 :           && DF_INSN_LUID (where_dead) >= from_luid
   14229      9277380 :           && DF_INSN_LUID (where_dead) < DF_INSN_LUID (to_insn))
   14230              :         {
   14231      2764962 :           rtx note = remove_death (regno, where_dead);
   14232              : 
   14233              :           /* It is possible for the call above to return 0.  This can occur
   14234              :              when last_death points to I2 or I1 that we combined with.
   14235              :              In that case make a new note.
   14236              : 
   14237              :              We must also check for the case where X is a hard register
   14238              :              and NOTE is a death note for a range of hard registers
   14239              :              including X.  In that case, we must put REG_DEAD notes for
   14240              :              the remaining registers in place of NOTE.  */
   14241              : 
   14242      2764962 :           if (note != 0 && regno < FIRST_PSEUDO_REGISTER
   14243      2764962 :               && partial_subreg_p (GET_MODE (x), GET_MODE (XEXP (note, 0))))
   14244              :             {
   14245            0 :               unsigned int deadregno = REGNO (XEXP (note, 0));
   14246            0 :               unsigned int deadend = END_REGNO (XEXP (note, 0));
   14247            0 :               unsigned int ourend = END_REGNO (x);
   14248            0 :               unsigned int i;
   14249              : 
   14250            0 :               for (i = deadregno; i < deadend; i++)
   14251            0 :                 if (i < regno || i >= ourend)
   14252            0 :                   add_reg_note (where_dead, REG_DEAD, regno_reg_rtx[i]);
   14253              :             }
   14254              : 
   14255              :           /* If we didn't find any note, or if we found a REG_DEAD note that
   14256              :              covers only part of the given reg, and we have a multi-reg hard
   14257              :              register, then to be safe we must check for REG_DEAD notes
   14258              :              for each register other than the first.  They could have
   14259              :              their own REG_DEAD notes lying around.  */
   14260      2764962 :           else if ((note == 0
   14261              :                     || (note != 0
   14262        75673 :                         && partial_subreg_p (GET_MODE (XEXP (note, 0)),
   14263        75673 :                                              GET_MODE (x))))
   14264      2689289 :                    && regno < FIRST_PSEUDO_REGISTER
   14265      3104405 :                    && REG_NREGS (x) > 1)
   14266              :             {
   14267            0 :               unsigned int ourend = END_REGNO (x);
   14268            0 :               unsigned int i, offset;
   14269            0 :               rtx oldnotes = 0;
   14270              : 
   14271            0 :               if (note)
   14272            0 :                 offset = hard_regno_nregs (regno, GET_MODE (XEXP (note, 0)));
   14273              :               else
   14274              :                 offset = 1;
   14275              : 
   14276            0 :               for (i = regno + offset; i < ourend; i++)
   14277            0 :                 move_deaths (regno_reg_rtx[i],
   14278              :                              maybe_kill_insn, from_luid, to_insn, &oldnotes);
   14279              :             }
   14280              : 
   14281      2764962 :           if (note != 0 && GET_MODE (XEXP (note, 0)) == GET_MODE (x))
   14282              :             {
   14283        75649 :               XEXP (note, 1) = *pnotes;
   14284        75649 :               *pnotes = note;
   14285              :             }
   14286              :           else
   14287      2689313 :             *pnotes = alloc_reg_note (REG_DEAD, x, *pnotes);
   14288              :         }
   14289              : 
   14290              :       return;
   14291              :     }
   14292              : 
   14293     19089577 :   else if (GET_CODE (x) == SET)
   14294              :     {
   14295      4290863 :       rtx dest = SET_DEST (x);
   14296              : 
   14297      4290863 :       move_deaths (SET_SRC (x), maybe_kill_insn, from_luid, to_insn, pnotes);
   14298              : 
   14299              :       /* In the case of a ZERO_EXTRACT, a STRICT_LOW_PART, or a SUBREG
   14300              :          that accesses one word of a multi-word item, some
   14301              :          piece of everything register in the expression is used by
   14302              :          this insn, so remove any old death.  */
   14303              :       /* ??? So why do we test for equality of the sizes?  */
   14304              : 
   14305      4290863 :       if (GET_CODE (dest) == ZERO_EXTRACT
   14306      4290427 :           || GET_CODE (dest) == STRICT_LOW_PART
   14307      8579539 :           || (GET_CODE (dest) == SUBREG
   14308        78743 :               && !read_modify_subreg_p (dest)))
   14309              :         {
   14310              :           move_deaths (dest, maybe_kill_insn, from_luid, to_insn, pnotes);
   14311              :           return;
   14312              :         }
   14313              : 
   14314              :       /* If this is some other SUBREG, we know it replaces the entire
   14315              :          value, so use that as the destination.  */
   14316      4227721 :       if (GET_CODE (dest) == SUBREG)
   14317        17788 :         dest = SUBREG_REG (dest);
   14318              : 
   14319              :       /* If this is a MEM, adjust deaths of anything used in the address.
   14320              :          For a REG (the only other possibility), the entire value is
   14321              :          being replaced so the old value is not used in this insn.  */
   14322              : 
   14323      4227721 :       if (MEM_P (dest))
   14324       473690 :         move_deaths (XEXP (dest, 0), maybe_kill_insn, from_luid,
   14325              :                      to_insn, pnotes);
   14326              :       return;
   14327              :     }
   14328              : 
   14329     14798714 :   else if (GET_CODE (x) == CLOBBER)
   14330              :     return;
   14331              : 
   14332     14149815 :   len = GET_RTX_LENGTH (code);
   14333     14149815 :   fmt = GET_RTX_FORMAT (code);
   14334              : 
   14335     36859241 :   for (i = 0; i < len; i++)
   14336              :     {
   14337     22709426 :       if (fmt[i] == 'E')
   14338              :         {
   14339      1035298 :           int j;
   14340      3679242 :           for (j = XVECLEN (x, i) - 1; j >= 0; j--)
   14341      2643944 :             move_deaths (XVECEXP (x, i, j), maybe_kill_insn, from_luid,
   14342              :                          to_insn, pnotes);
   14343              :         }
   14344     21674128 :       else if (fmt[i] == 'e')
   14345     13662768 :         move_deaths (XEXP (x, i), maybe_kill_insn, from_luid, to_insn, pnotes);
   14346              :     }
   14347              : }
   14348              : 
   14349              : /* Return true if X is the target of a bit-field assignment in BODY, the
   14350              :    pattern of an insn.  X must be a REG.  */
   14351              : 
   14352              : static bool
   14353      4837337 : reg_bitfield_target_p (rtx x, rtx body)
   14354              : {
   14355      4837337 :   int i;
   14356              : 
   14357      4837337 :   if (GET_CODE (body) == SET)
   14358              :     {
   14359      3512608 :       rtx dest = SET_DEST (body);
   14360      3512608 :       rtx target;
   14361      3512608 :       unsigned int regno, tregno, endregno, endtregno;
   14362              : 
   14363      3512608 :       if (GET_CODE (dest) == ZERO_EXTRACT)
   14364          426 :         target = XEXP (dest, 0);
   14365      3512182 :       else if (GET_CODE (dest) == STRICT_LOW_PART)
   14366         2002 :         target = SUBREG_REG (XEXP (dest, 0));
   14367              :       else
   14368              :         return false;
   14369              : 
   14370         2428 :       if (GET_CODE (target) == SUBREG)
   14371          219 :         target = SUBREG_REG (target);
   14372              : 
   14373         2428 :       if (!REG_P (target))
   14374              :         return false;
   14375              : 
   14376         2349 :       tregno = REGNO (target), regno = REGNO (x);
   14377         2349 :       if (tregno >= FIRST_PSEUDO_REGISTER || regno >= FIRST_PSEUDO_REGISTER)
   14378         2339 :         return target == x;
   14379              : 
   14380           10 :       endtregno = end_hard_regno (GET_MODE (target), tregno);
   14381           10 :       endregno = end_hard_regno (GET_MODE (x), regno);
   14382              : 
   14383           10 :       return endregno > tregno && regno < endtregno;
   14384              :     }
   14385              : 
   14386      1324729 :   else if (GET_CODE (body) == PARALLEL)
   14387      1991611 :     for (i = XVECLEN (body, 0) - 1; i >= 0; i--)
   14388      1341985 :       if (reg_bitfield_target_p (x, XVECEXP (body, 0, i)))
   14389              :         return true;
   14390              : 
   14391              :   return false;
   14392              : }
   14393              : 
   14394              : /* Given a chain of REG_NOTES originally from FROM_INSN, try to place them
   14395              :    as appropriate.  I3 and I2 are the insns resulting from the combination
   14396              :    insns including FROM (I2 may be zero).
   14397              : 
   14398              :    ELIM_I2 and ELIM_I1 are either zero or registers that we know will
   14399              :    not need REG_DEAD notes because they are being substituted for.  This
   14400              :    saves searching in the most common cases.
   14401              : 
   14402              :    Each note in the list is either ignored or placed on some insns, depending
   14403              :    on the type of note.  */
   14404              : 
   14405              : static void
   14406     10062025 : distribute_notes (rtx notes, rtx_insn *from_insn, rtx_insn *i3, rtx_insn *i2,
   14407              :                   rtx elim_i2, rtx elim_i1, rtx elim_i0)
   14408              : {
   14409     10062025 :   rtx note, next_note;
   14410     10062025 :   rtx tem_note;
   14411     10062025 :   rtx_insn *tem_insn;
   14412              : 
   14413     23327244 :   for (note = notes; note; note = next_note)
   14414              :     {
   14415     13265219 :       rtx_insn *place = 0, *place2 = 0;
   14416              : 
   14417     13265219 :       next_note = XEXP (note, 1);
   14418     13265219 :       switch (REG_NOTE_KIND (note))
   14419              :         {
   14420              :         case REG_BR_PROB:
   14421              :         case REG_BR_PRED:
   14422              :           /* Doesn't matter much where we put this, as long as it's somewhere.
   14423              :              It is preferable to keep these notes on branches, which is most
   14424              :              likely to be i3.  */
   14425              :           place = i3;
   14426              :           break;
   14427              : 
   14428            0 :         case REG_NON_LOCAL_GOTO:
   14429            0 :           if (JUMP_P (i3))
   14430              :             place = i3;
   14431              :           else
   14432              :             {
   14433            0 :               gcc_assert (i2 && JUMP_P (i2));
   14434              :               place = i2;
   14435              :             }
   14436              :           break;
   14437              : 
   14438        21259 :         case REG_EH_REGION:
   14439        21259 :           {
   14440              :             /* The landing pad handling needs to be kept in sync with the
   14441              :                prerequisite checking in try_combine.  */
   14442        21259 :             int lp_nr = INTVAL (XEXP (note, 0));
   14443              :             /* A REG_EH_REGION note transferring control can only ever come
   14444              :                from i3.  */
   14445        21259 :             if (lp_nr > 0)
   14446        11601 :               gcc_assert (from_insn == i3);
   14447              :             /* We are making sure there is a single effective REG_EH_REGION
   14448              :                note and it's valid to put it on i3.  */
   14449        21259 :             if (!insn_could_throw_p (from_insn)
   14450        21259 :                 && !(lp_nr == INT_MIN && can_nonlocal_goto (from_insn)))
   14451              :               /* Throw away stray notes on insns that can never throw or
   14452              :                  make a nonlocal goto.  */
   14453              :               ;
   14454              :             else
   14455              :               {
   14456        21182 :                 if (CALL_P (i3))
   14457              :                   place = i3;
   14458              :                 else
   14459              :                   {
   14460         2084 :                     gcc_assert (cfun->can_throw_non_call_exceptions);
   14461              :                     /* If i3 can still trap preserve the note, otherwise we've
   14462              :                        combined things such that we can now prove that the
   14463              :                        instructions can't trap.  Drop the note in this case.  */
   14464         2084 :                     if (may_trap_p (i3))
   14465              :                       place = i3;
   14466              :                   }
   14467              :               }
   14468              :             break;
   14469              :           }
   14470              : 
   14471       126358 :         case REG_ARGS_SIZE:
   14472              :           /* ??? How to distribute between i3-i1.  Assume i3 contains the
   14473              :              entire adjustment.  Assert i3 contains at least some adjust.  */
   14474       126358 :           if (!noop_move_p (i3))
   14475              :             {
   14476       126357 :               poly_int64 old_size, args_size = get_args_size (note);
   14477              :               /* fixup_args_size_notes looks at REG_NORETURN note,
   14478              :                  so ensure the note is placed there first.  */
   14479       126357 :               if (CALL_P (i3))
   14480              :                 {
   14481              :                   rtx *np;
   14482         1661 :                   for (np = &next_note; *np; np = &XEXP (*np, 1))
   14483           20 :                     if (REG_NOTE_KIND (*np) == REG_NORETURN)
   14484              :                       {
   14485            9 :                         rtx n = *np;
   14486            9 :                         *np = XEXP (n, 1);
   14487            9 :                         XEXP (n, 1) = REG_NOTES (i3);
   14488            9 :                         REG_NOTES (i3) = n;
   14489            9 :                         break;
   14490              :                       }
   14491              :                 }
   14492       126357 :               old_size = fixup_args_size_notes (PREV_INSN (i3), i3, args_size);
   14493              :               /* emit_call_1 adds for !ACCUMULATE_OUTGOING_ARGS
   14494              :                  REG_ARGS_SIZE note to all noreturn calls, allow that here.  */
   14495       126357 :               gcc_assert (maybe_ne (old_size, args_size)
   14496              :                           || (CALL_P (i3)
   14497              :                               && !ACCUMULATE_OUTGOING_ARGS
   14498              :                               && find_reg_note (i3, REG_NORETURN, NULL_RTX)));
   14499              :             }
   14500              :           break;
   14501              : 
   14502        81217 :         case REG_NORETURN:
   14503        81217 :         case REG_SETJMP:
   14504        81217 :         case REG_TM:
   14505        81217 :         case REG_CALL_DECL:
   14506        81217 :         case REG_UNTYPED_CALL:
   14507        81217 :         case REG_CALL_NOCF_CHECK:
   14508              :           /* These notes must remain with the call.  It should not be
   14509              :              possible for both I2 and I3 to be a call.  */
   14510        81217 :           if (CALL_P (i3))
   14511              :             place = i3;
   14512              :           else
   14513              :             {
   14514            0 :               gcc_assert (i2 && CALL_P (i2));
   14515              :               place = i2;
   14516              :             }
   14517              :           break;
   14518              : 
   14519      2011294 :         case REG_UNUSED:
   14520              :           /* Any clobbers for i3 may still exist, and so we must process
   14521              :              REG_UNUSED notes from that insn.
   14522              : 
   14523              :              Any clobbers from i2 or i1 can only exist if they were added by
   14524              :              recog_for_combine.  In that case, recog_for_combine created the
   14525              :              necessary REG_UNUSED notes.  Trying to keep any original
   14526              :              REG_UNUSED notes from these insns can cause incorrect output
   14527              :              if it is for the same register as the original i3 dest.
   14528              :              In that case, we will notice that the register is set in i3,
   14529              :              and then add a REG_UNUSED note for the destination of i3, which
   14530              :              is wrong.  However, it is possible to have REG_UNUSED notes from
   14531              :              i2 or i1 for register which were both used and clobbered, so
   14532              :              we keep notes from i2 or i1 if they will turn into REG_DEAD
   14533              :              notes.  */
   14534              : 
   14535              :           /* If this register is set or clobbered between FROM_INSN and I3,
   14536              :              we should not create a note for it.  */
   14537      2011294 :           if (reg_set_between_p (XEXP (note, 0), from_insn, i3))
   14538              :             break;
   14539              : 
   14540              :           /* If this register is set or clobbered in I3, put the note there
   14541              :              unless there is one already.  */
   14542      1920656 :           if (reg_set_p (XEXP (note, 0), PATTERN (i3)))
   14543              :             {
   14544      1146447 :               if (from_insn != i3)
   14545              :                 break;
   14546              : 
   14547       674255 :               if (! (REG_P (XEXP (note, 0))
   14548       674255 :                      ? find_regno_note (i3, REG_UNUSED, REGNO (XEXP (note, 0)))
   14549            0 :                      : find_reg_note (i3, REG_UNUSED, XEXP (note, 0))))
   14550              :                 place = i3;
   14551              :             }
   14552              :           /* Otherwise, if this register is used by I3, then this register
   14553              :              now dies here, so we must put a REG_DEAD note here unless there
   14554              :              is one already.  */
   14555       774209 :           else if (reg_referenced_p (XEXP (note, 0), PATTERN (i3)))
   14556              :             {
   14557         7711 :               if (! (REG_P (XEXP (note, 0))
   14558         7711 :                      ? find_regno_note (i3, REG_DEAD, REGNO (XEXP (note, 0)))
   14559            0 :                      : find_reg_note (i3, REG_DEAD, XEXP (note, 0))))
   14560              :                 {
   14561         7456 :                   PUT_REG_NOTE_KIND (note, REG_DEAD);
   14562         7456 :                   place = i3;
   14563              :                 }
   14564              :             }
   14565              : 
   14566              :           /* A SET or CLOBBER of the REG_UNUSED reg has been removed,
   14567              :              but we can't tell which at this point.  We must reset any
   14568              :              expectations we had about the value that was previously
   14569              :              stored in the reg.  ??? Ideally, we'd adjust REG_N_SETS
   14570              :              and, if appropriate, restore its previous value, but we
   14571              :              don't have enough information for that at this point.  */
   14572              :           else
   14573              :             {
   14574       766498 :               record_value_for_reg (XEXP (note, 0), NULL, NULL_RTX);
   14575              : 
   14576              :               /* Otherwise, if this register is now referenced in i2
   14577              :                  then the register used to be modified in one of the
   14578              :                  original insns.  If it was i3 (say, in an unused
   14579              :                  parallel), it's now completely gone, so the note can
   14580              :                  be discarded.  But if it was modified in i2, i1 or i0
   14581              :                  and we still reference it in i2, then we're
   14582              :                  referencing the previous value, and since the
   14583              :                  register was modified and REG_UNUSED, we know that
   14584              :                  the previous value is now dead.  So, if we only
   14585              :                  reference the register in i2, we change the note to
   14586              :                  REG_DEAD, to reflect the previous value.  However, if
   14587              :                  we're also setting or clobbering the register as
   14588              :                  scratch, we know (because the register was not
   14589              :                  referenced in i3) that it's unused, just as it was
   14590              :                  unused before, and we place the note in i2.  */
   14591        17120 :               if (from_insn != i3 && i2 && INSN_P (i2)
   14592       783618 :                   && reg_referenced_p (XEXP (note, 0), PATTERN (i2)))
   14593              :                 {
   14594           24 :                   if (!reg_set_p (XEXP (note, 0), PATTERN (i2)))
   14595           24 :                     PUT_REG_NOTE_KIND (note, REG_DEAD);
   14596           24 :                   if (! (REG_P (XEXP (note, 0))
   14597           24 :                          ? find_regno_note (i2, REG_NOTE_KIND (note),
   14598           24 :                                             REGNO (XEXP (note, 0)))
   14599            0 :                          : find_reg_note (i2, REG_NOTE_KIND (note),
   14600              :                                           XEXP (note, 0))))
   14601              :                     place = i2;
   14602              :                 }
   14603              :             }
   14604              : 
   14605              :           break;
   14606              : 
   14607       396802 :         case REG_EQUAL:
   14608       396802 :         case REG_EQUIV:
   14609       396802 :         case REG_NOALIAS:
   14610              :           /* These notes say something about results of an insn.  We can
   14611              :              only support them if they used to be on I3 in which case they
   14612              :              remain on I3.  Otherwise they are ignored.
   14613              : 
   14614              :              If the note refers to an expression that is not a constant, we
   14615              :              must also ignore the note since we cannot tell whether the
   14616              :              equivalence is still true.  It might be possible to do
   14617              :              slightly better than this (we only have a problem if I2DEST
   14618              :              or I1DEST is present in the expression), but it doesn't
   14619              :              seem worth the trouble.  */
   14620              : 
   14621       396802 :           if (from_insn == i3
   14622       195966 :               && (XEXP (note, 0) == 0 || CONSTANT_P (XEXP (note, 0))))
   14623              :             place = i3;
   14624              :           break;
   14625              : 
   14626            0 :         case REG_INC:
   14627              :           /* These notes say something about how a register is used.  They must
   14628              :              be present on any use of the register in I2 or I3.  */
   14629            0 :           if (reg_mentioned_p (XEXP (note, 0), PATTERN (i3)))
   14630            0 :             place = i3;
   14631              : 
   14632            0 :           if (i2 && reg_mentioned_p (XEXP (note, 0), PATTERN (i2)))
   14633              :             {
   14634            0 :               if (place)
   14635              :                 place2 = i2;
   14636              :               else
   14637              :                 place = i2;
   14638              :             }
   14639              :           break;
   14640              : 
   14641         7354 :         case REG_LABEL_TARGET:
   14642         7354 :         case REG_LABEL_OPERAND:
   14643              :           /* This can show up in several ways -- either directly in the
   14644              :              pattern, or hidden off in the constant pool with (or without?)
   14645              :              a REG_EQUAL note.  */
   14646              :           /* ??? Ignore the without-reg_equal-note problem for now.  */
   14647         7354 :           if (reg_mentioned_p (XEXP (note, 0), PATTERN (i3))
   14648         7354 :               || ((tem_note = find_reg_note (i3, REG_EQUAL, NULL_RTX))
   14649            0 :                   && GET_CODE (XEXP (tem_note, 0)) == LABEL_REF
   14650            0 :                   && label_ref_label (XEXP (tem_note, 0)) == XEXP (note, 0)))
   14651              :             place = i3;
   14652              : 
   14653         7354 :           if (i2
   14654         7354 :               && (reg_mentioned_p (XEXP (note, 0), PATTERN (i2))
   14655            0 :                   || ((tem_note = find_reg_note (i2, REG_EQUAL, NULL_RTX))
   14656            0 :                       && GET_CODE (XEXP (tem_note, 0)) == LABEL_REF
   14657            0 :                       && label_ref_label (XEXP (tem_note, 0)) == XEXP (note, 0))))
   14658              :             {
   14659            0 :               if (place)
   14660              :                 place2 = i2;
   14661              :               else
   14662              :                 place = i2;
   14663              :             }
   14664              : 
   14665              :           /* For REG_LABEL_TARGET on a JUMP_P, we prefer to put the note
   14666              :              as a JUMP_LABEL or decrement LABEL_NUSES if it's already
   14667              :              there.  */
   14668         7354 :           if (place && JUMP_P (place)
   14669         6102 :               && REG_NOTE_KIND (note) == REG_LABEL_TARGET
   14670            0 :               && (JUMP_LABEL (place) == NULL
   14671            0 :                   || JUMP_LABEL (place) == XEXP (note, 0)))
   14672              :             {
   14673            0 :               rtx label = JUMP_LABEL (place);
   14674              : 
   14675            0 :               if (!label)
   14676            0 :                 JUMP_LABEL (place) = XEXP (note, 0);
   14677            0 :               else if (LABEL_P (label))
   14678            0 :                 LABEL_NUSES (label)--;
   14679              :             }
   14680              : 
   14681         7354 :           if (place2 && JUMP_P (place2)
   14682            0 :               && REG_NOTE_KIND (note) == REG_LABEL_TARGET
   14683            0 :               && (JUMP_LABEL (place2) == NULL
   14684            0 :                   || JUMP_LABEL (place2) == XEXP (note, 0)))
   14685              :             {
   14686            0 :               rtx label = JUMP_LABEL (place2);
   14687              : 
   14688            0 :               if (!label)
   14689            0 :                 JUMP_LABEL (place2) = XEXP (note, 0);
   14690            0 :               else if (LABEL_P (label))
   14691            0 :                 LABEL_NUSES (label)--;
   14692              :               place2 = 0;
   14693              :             }
   14694              :           break;
   14695              : 
   14696              :         case REG_NONNEG:
   14697              :           /* This note says something about the value of a register prior
   14698              :              to the execution of an insn.  It is too much trouble to see
   14699              :              if the note is still correct in all situations.  It is better
   14700              :              to simply delete it.  */
   14701              :           break;
   14702              : 
   14703     10580421 :         case REG_DEAD:
   14704              :           /* If we replaced the right hand side of FROM_INSN with a
   14705              :              REG_EQUAL note, the original use of the dying register
   14706              :              will not have been combined into I3 and I2.  In such cases,
   14707              :              FROM_INSN is guaranteed to be the first of the combined
   14708              :              instructions, so we simply need to search back before
   14709              :              FROM_INSN for the previous use or set of this register,
   14710              :              then alter the notes there appropriately.
   14711              : 
   14712              :              If the register is used as an input in I3, it dies there.
   14713              :              Similarly for I2, if it is nonzero and adjacent to I3.
   14714              : 
   14715              :              If the register is not used as an input in either I3 or I2
   14716              :              and it is not one of the registers we were supposed to eliminate,
   14717              :              there are two possibilities.  We might have a non-adjacent I2
   14718              :              or we might have somehow eliminated an additional register
   14719              :              from a computation.  For example, we might have had A & B where
   14720              :              we discover that B will always be zero.  In this case we will
   14721              :              eliminate the reference to A.
   14722              : 
   14723              :              In both cases, we must search to see if we can find a previous
   14724              :              use of A and put the death note there.  */
   14725              : 
   14726     10580421 :           if (from_insn
   14727      7388082 :               && from_insn == i2mod
   14728     10582065 :               && !reg_overlap_mentioned_p (XEXP (note, 0), i2mod_new_rhs))
   14729              :             tem_insn = from_insn;
   14730              :           else
   14731              :             {
   14732     10579068 :               if (from_insn
   14733      7386729 :                   && CALL_P (from_insn)
   14734     10811830 :                   && find_reg_fusage (from_insn, USE, XEXP (note, 0)))
   14735              :                 place = from_insn;
   14736     10427516 :               else if (i2 && reg_set_p (XEXP (note, 0), PATTERN (i2)))
   14737              :                 {
   14738              :                   /* If the new I2 sets the same register that is marked
   14739              :                      dead in the note, we do not in general know where to
   14740              :                      put the note.  One important case we _can_ handle is
   14741              :                      when the note comes from I3.  */
   14742        39852 :                   if (from_insn == i3)
   14743              :                     place = i3;
   14744              :                   else
   14745              :                     break;
   14746              :                 }
   14747     10387664 :               else if (reg_referenced_p (XEXP (note, 0), PATTERN (i3)))
   14748              :                 place = i3;
   14749       105464 :               else if (i2 != 0 && next_nonnote_nondebug_insn (i2) == i3
   14750      4141191 :                        && reg_referenced_p (XEXP (note, 0), PATTERN (i2)))
   14751              :                 place = i2;
   14752      3992406 :               else if ((rtx_equal_p (XEXP (note, 0), elim_i2)
   14753      3872036 :                         && !(i2mod
   14754        26894 :                              && reg_overlap_mentioned_p (XEXP (note, 0),
   14755              :                                                          i2mod_old_rhs)))
   14756       147274 :                        || rtx_equal_p (XEXP (note, 0), elim_i1)
   14757      4047025 :                        || rtx_equal_p (XEXP (note, 0), elim_i0))
   14758              :                 break;
   14759              :               tem_insn = i3;
   14760              :             }
   14761              : 
   14762       237533 :           if (place == 0)
   14763              :             {
   14764        51323 :               basic_block bb = this_basic_block;
   14765              : 
   14766      1166284 :               for (tem_insn = PREV_INSN (tem_insn); place == 0; tem_insn = PREV_INSN (tem_insn))
   14767              :                 {
   14768      1166284 :                   if (!NONDEBUG_INSN_P (tem_insn))
   14769              :                     {
   14770       823634 :                       if (tem_insn == BB_HEAD (bb))
   14771              :                         break;
   14772       788765 :                       continue;
   14773              :                     }
   14774              : 
   14775              :                   /* If the register is being set at TEM_INSN, see if that is all
   14776              :                      TEM_INSN is doing.  If so, delete TEM_INSN.  Otherwise, make this
   14777              :                      into a REG_UNUSED note instead. Don't delete sets to
   14778              :                      global register vars.  */
   14779       342650 :                   if ((REGNO (XEXP (note, 0)) >= FIRST_PSEUDO_REGISTER
   14780         1438 :                        || !global_regs[REGNO (XEXP (note, 0))])
   14781       344088 :                       && reg_set_p (XEXP (note, 0), PATTERN (tem_insn)))
   14782              :                     {
   14783        15678 :                       rtx set = single_set (tem_insn);
   14784        15678 :                       rtx inner_dest = 0;
   14785              : 
   14786        15678 :                       if (set != 0)
   14787        12064 :                         for (inner_dest = SET_DEST (set);
   14788        12305 :                              (GET_CODE (inner_dest) == STRICT_LOW_PART
   14789        12305 :                               || GET_CODE (inner_dest) == SUBREG
   14790        12305 :                               || GET_CODE (inner_dest) == ZERO_EXTRACT);
   14791          241 :                              inner_dest = XEXP (inner_dest, 0))
   14792              :                           ;
   14793              : 
   14794              :                       /* Verify that it was the set, and not a clobber that
   14795              :                          modified the register.
   14796              : 
   14797              :                          If we cannot delete the setter due to side
   14798              :                          effects, mark the user with an UNUSED note instead
   14799              :                          of deleting it.  */
   14800              : 
   14801        12064 :                       if (set != 0 && ! side_effects_p (SET_SRC (set))
   14802        11691 :                           && rtx_equal_p (XEXP (note, 0), inner_dest))
   14803              :                         {
   14804              :                           /* Move the notes and links of TEM_INSN elsewhere.
   14805              :                              This might delete other dead insns recursively.
   14806              :                              First set the pattern to something that won't use
   14807              :                              any register.  */
   14808        11566 :                           rtx old_notes = REG_NOTES (tem_insn);
   14809              : 
   14810        11566 :                           PATTERN (tem_insn) = pc_rtx;
   14811        11566 :                           REG_NOTES (tem_insn) = NULL;
   14812              : 
   14813        11566 :                           distribute_notes (old_notes, tem_insn, tem_insn, NULL,
   14814              :                                             NULL_RTX, NULL_RTX, NULL_RTX);
   14815        11566 :                           distribute_links (LOG_LINKS (tem_insn));
   14816              : 
   14817        11566 :                           unsigned int regno = REGNO (XEXP (note, 0));
   14818        11566 :                           reg_stat_type *rsp = &reg_stat[regno];
   14819        11566 :                           if (rsp->last_set == tem_insn)
   14820        10157 :                             record_value_for_reg (XEXP (note, 0), NULL, NULL_RTX);
   14821              : 
   14822        11566 :                           SET_INSN_DELETED (tem_insn);
   14823        11566 :                           if (tem_insn == i2)
   14824       326196 :                             i2 = NULL;
   14825              :                         }
   14826              :                       else
   14827              :                         {
   14828         4112 :                           PUT_REG_NOTE_KIND (note, REG_UNUSED);
   14829              : 
   14830              :                           /*  If there isn't already a REG_UNUSED note, put one
   14831              :                               here.  Do not place a REG_DEAD note, even if
   14832              :                               the register is also used here; that would not
   14833              :                               match the algorithm used in lifetime analysis
   14834              :                               and can cause the consistency check in the
   14835              :                               scheduler to fail.  */
   14836         4112 :                           if (! find_regno_note (tem_insn, REG_UNUSED,
   14837         4112 :                                                  REGNO (XEXP (note, 0))))
   14838         2257 :                             place = tem_insn;
   14839              :                           break;
   14840              :                         }
   14841              :                     }
   14842       326972 :                   else if (reg_referenced_p (XEXP (note, 0), PATTERN (tem_insn))
   14843       326972 :                            || (CALL_P (tem_insn)
   14844        14461 :                                && find_reg_fusage (tem_insn, USE, XEXP (note, 0))))
   14845              :                     {
   14846        12342 :                       place = tem_insn;
   14847              : 
   14848              :                       /* If we are doing a 3->2 combination, and we have a
   14849              :                          register which formerly died in i3 and was not used
   14850              :                          by i2, which now no longer dies in i3 and is used in
   14851              :                          i2 but does not die in i2, and place is between i2
   14852              :                          and i3, then we may need to move a link from place to
   14853              :                          i2.  */
   14854         3896 :                       if (i2 && DF_INSN_LUID (place) > DF_INSN_LUID (i2)
   14855           90 :                           && from_insn
   14856           90 :                           && DF_INSN_LUID (from_insn) > DF_INSN_LUID (i2)
   14857        12432 :                           && reg_referenced_p (XEXP (note, 0), PATTERN (i2)))
   14858              :                         {
   14859           90 :                           struct insn_link *links = LOG_LINKS (place);
   14860           90 :                           LOG_LINKS (place) = NULL;
   14861           90 :                           distribute_links (links);
   14862              :                         }
   14863              :                       break;
   14864              :                     }
   14865              : 
   14866       326196 :                   if (tem_insn == BB_HEAD (bb))
   14867              :                     break;
   14868              :                 }
   14869              : 
   14870              :             }
   14871              : 
   14872              :           /* If the register is set or already dead at PLACE, we needn't do
   14873              :              anything with this note if it is still a REG_DEAD note.
   14874              :              We check here if it is set at all, not if is it totally replaced,
   14875              :              which is what `dead_or_set_p' checks, so also check for it being
   14876              :              set partially.  */
   14877              : 
   14878      6634301 :           if (place && REG_NOTE_KIND (note) == REG_DEAD)
   14879              :             {
   14880      6595320 :               unsigned int regno = REGNO (XEXP (note, 0));
   14881      6595320 :               reg_stat_type *rsp = &reg_stat[regno];
   14882              : 
   14883      6595320 :               if (dead_or_set_p (place, XEXP (note, 0))
   14884      6595320 :                   || reg_bitfield_target_p (XEXP (note, 0), PATTERN (place)))
   14885              :                 {
   14886              :                   /* Unless the register previously died in PLACE, clear
   14887              :                      last_death.  [I no longer understand why this is
   14888              :                      being done.] */
   14889      3099994 :                   if (rsp->last_death != place)
   14890       639743 :                     rsp->last_death = 0;
   14891              :                   place = 0;
   14892              :                 }
   14893              :               else
   14894      3495326 :                 rsp->last_death = place;
   14895              : 
   14896              :               /* If this is a death note for a hard reg that is occupying
   14897              :                  multiple registers, ensure that we are still using all
   14898              :                  parts of the object.  If we find a piece of the object
   14899              :                  that is unused, we must arrange for an appropriate REG_DEAD
   14900              :                  note to be added for it.  However, we can't just emit a USE
   14901              :                  and tag the note to it, since the register might actually
   14902              :                  be dead; so we recurse, and the recursive call then finds
   14903              :                  the previous insn that used this register.  */
   14904              : 
   14905      4135069 :               if (place && REG_NREGS (XEXP (note, 0)) > 1)
   14906              :                 {
   14907          773 :                   unsigned int endregno = END_REGNO (XEXP (note, 0));
   14908          773 :                   bool all_used = true;
   14909          773 :                   unsigned int i;
   14910              : 
   14911         2319 :                   for (i = regno; i < endregno; i++)
   14912         1546 :                     if ((! refers_to_regno_p (i, PATTERN (place))
   14913         1546 :                          && ! find_regno_fusage (place, USE, i))
   14914         3092 :                         || dead_or_set_regno_p (place, i))
   14915              :                       {
   14916              :                         all_used = false;
   14917              :                         break;
   14918              :                       }
   14919              : 
   14920          773 :                   if (! all_used)
   14921              :                     {
   14922              :                       /* Put only REG_DEAD notes for pieces that are
   14923              :                          not already dead or set.  */
   14924              : 
   14925            0 :                       for (i = regno; i < endregno;
   14926            0 :                            i += hard_regno_nregs (i, reg_raw_mode[i]))
   14927              :                         {
   14928            0 :                           rtx piece = regno_reg_rtx[i];
   14929            0 :                           basic_block bb = this_basic_block;
   14930              : 
   14931            0 :                           if (! dead_or_set_p (place, piece)
   14932            0 :                               && ! reg_bitfield_target_p (piece,
   14933            0 :                                                           PATTERN (place)))
   14934              :                             {
   14935            0 :                               rtx new_note = alloc_reg_note (REG_DEAD, piece,
   14936              :                                                              NULL_RTX);
   14937              : 
   14938            0 :                               distribute_notes (new_note, place, place,
   14939              :                                                 NULL, NULL_RTX, NULL_RTX,
   14940              :                                                 NULL_RTX);
   14941              :                             }
   14942            0 :                           else if (! refers_to_regno_p (i, PATTERN (place))
   14943            0 :                                    && ! find_regno_fusage (place, USE, i))
   14944            0 :                             for (tem_insn = PREV_INSN (place); ;
   14945            0 :                                  tem_insn = PREV_INSN (tem_insn))
   14946              :                               {
   14947            0 :                                 if (!NONDEBUG_INSN_P (tem_insn))
   14948              :                                   {
   14949            0 :                                     if (tem_insn == BB_HEAD (bb))
   14950              :                                       break;
   14951            0 :                                     continue;
   14952              :                                   }
   14953            0 :                                 if (dead_or_set_p (tem_insn, piece)
   14954            0 :                                     || reg_bitfield_target_p (piece,
   14955            0 :                                                               PATTERN (tem_insn)))
   14956              :                                   {
   14957            0 :                                     add_reg_note (tem_insn, REG_UNUSED, piece);
   14958            0 :                                     break;
   14959              :                                   }
   14960              :                               }
   14961              :                         }
   14962              : 
   14963              :                       place = 0;
   14964              :                     }
   14965              :                 }
   14966              :             }
   14967              :           break;
   14968              : 
   14969            0 :         default:
   14970              :           /* Any other notes should not be present at this point in the
   14971              :              compilation.  */
   14972            0 :           gcc_unreachable ();
   14973              :         }
   14974              : 
   14975      4342044 :       if (place)
   14976              :         {
   14977      4316002 :           XEXP (note, 1) = REG_NOTES (place);
   14978      4316002 :           REG_NOTES (place) = note;
   14979              : 
   14980              :           /* Set added_notes_insn to the earliest insn we added a note to.  */
   14981      4316002 :           if (added_notes_insn == 0
   14982      4316002 :               || DF_INSN_LUID (added_notes_insn) > DF_INSN_LUID (place))
   14983      2853824 :             added_notes_insn = place;
   14984              :         }
   14985              : 
   14986     13265219 :       if (place2)
   14987              :         {
   14988            0 :           add_shallow_copy_of_reg_note (place2, note);
   14989              : 
   14990              :           /* Set added_notes_insn to the earliest insn we added a note to.  */
   14991            0 :           if (added_notes_insn == 0
   14992            0 :               || DF_INSN_LUID (added_notes_insn) > DF_INSN_LUID (place2))
   14993            0 :             added_notes_insn = place2;
   14994              :         }
   14995              :     }
   14996     10062025 : }
   14997              : 
   14998              : /* Similarly to above, distribute the LOG_LINKS that used to be present on
   14999              :    I3, I2, and I1 to new locations.  This is also called to add a link
   15000              :    pointing at I3 when I3's destination is changed.
   15001              : 
   15002              :    If START is nonnull and an insn, we know that the next location for each
   15003              :    link is no earlier than START.  LIMIT is the maximum number of nondebug
   15004              :    instructions that can be scanned when looking for the next use of a
   15005              :    definition.  */
   15006              : 
   15007              : static void
   15008     16327523 : distribute_links (struct insn_link *links, rtx_insn *start, int limit)
   15009              : {
   15010     16327523 :   struct insn_link *link, *next_link;
   15011              : 
   15012     24064334 :   for (link = links; link; link = next_link)
   15013              :     {
   15014      7736811 :       rtx_insn *place = 0;
   15015      7736811 :       rtx_insn *insn;
   15016      7736811 :       rtx set, reg;
   15017              : 
   15018      7736811 :       next_link = link->next;
   15019              : 
   15020              :       /* If the insn that this link points to is a NOTE, ignore it.  */
   15021      7736811 :       if (NOTE_P (link->insn))
   15022      4111950 :         continue;
   15023              : 
   15024      3624861 :       set = 0;
   15025      3624861 :       rtx pat = PATTERN (link->insn);
   15026      3624861 :       if (GET_CODE (pat) == SET)
   15027              :         set = pat;
   15028       639199 :       else if (GET_CODE (pat) == PARALLEL)
   15029              :         {
   15030              :           int i;
   15031       755061 :           for (i = 0; i < XVECLEN (pat, 0); i++)
   15032              :             {
   15033       751412 :               set = XVECEXP (pat, 0, i);
   15034       751412 :               if (GET_CODE (set) != SET)
   15035         3658 :                 continue;
   15036              : 
   15037       747754 :               reg = SET_DEST (set);
   15038       747754 :               while (GET_CODE (reg) == ZERO_EXTRACT
   15039       756413 :                      || GET_CODE (reg) == STRICT_LOW_PART
   15040      1512782 :                      || GET_CODE (reg) == SUBREG)
   15041         8666 :                 reg = XEXP (reg, 0);
   15042              : 
   15043       747754 :               if (!REG_P (reg))
   15044        45213 :                 continue;
   15045              : 
   15046       702541 :               if (REGNO (reg) == link->regno)
   15047              :                 break;
   15048              :             }
   15049       636952 :           if (i == XVECLEN (pat, 0))
   15050         3649 :             continue;
   15051              :         }
   15052              :       else
   15053         2247 :         continue;
   15054              : 
   15055      3618965 :       reg = SET_DEST (set);
   15056              : 
   15057      3618965 :       while (GET_CODE (reg) == ZERO_EXTRACT
   15058      3642257 :              || GET_CODE (reg) == STRICT_LOW_PART
   15059      7284628 :              || GET_CODE (reg) == SUBREG)
   15060        23807 :         reg = XEXP (reg, 0);
   15061              : 
   15062      3618965 :       if (reg == pc_rtx)
   15063          396 :         continue;
   15064              : 
   15065              :       /* A LOG_LINK is defined as being placed on the first insn that uses
   15066              :          a register and points to the insn that sets the register.  Start
   15067              :          searching at the next insn after the target of the link and stop
   15068              :          when we reach a set of the register or the end of the basic block.
   15069              : 
   15070              :          Note that this correctly handles the link that used to point from
   15071              :          I3 to I2.  Also note that not much searching is typically done here
   15072              :          since most links don't point very far away.  */
   15073              : 
   15074      3618569 :       int count = 0;
   15075      3618569 :       insn = start;
   15076      3618569 :       if (!insn || NOTE_P (insn))
   15077      3558980 :         insn = NEXT_INSN (link->insn);
   15078              :       else
   15079        59589 :         count = link->insn_count;
   15080     11951927 :       for (;
   15081     15570496 :            (insn && (this_basic_block->next_bb == EXIT_BLOCK_PTR_FOR_FN (cfun)
   15082     10773714 :                      || BB_HEAD (this_basic_block->next_bb) != insn));
   15083     11951927 :            insn = NEXT_INSN (insn))
   15084     15524273 :         if (DEBUG_INSN_P (insn))
   15085      3269904 :           continue;
   15086     12254369 :         else if (INSN_P (insn) && reg_overlap_mentioned_p (reg, PATTERN (insn)))
   15087              :           {
   15088      3418127 :             if (reg_referenced_p (reg, PATTERN (insn)))
   15089      3418127 :               place = insn;
   15090              :             break;
   15091              :           }
   15092      8836242 :         else if (CALL_P (insn)
   15093      8836242 :                  && find_reg_fusage (insn, USE, reg))
   15094              :           {
   15095              :             place = insn;
   15096              :             break;
   15097              :           }
   15098      8682203 :         else if (INSN_P (insn) && reg_set_p (reg, insn))
   15099              :           break;
   15100      8682023 :         else if (count >= limit)
   15101              :           break;
   15102              :         else
   15103      8682023 :           count += 1;
   15104      3618569 :       link->insn_count = count;
   15105              : 
   15106              :       /* If we found a place to put the link, place it there unless there
   15107              :          is already a link to the same insn as LINK at that point.  */
   15108              : 
   15109      3618569 :       if (place)
   15110              :         {
   15111      3572166 :           struct insn_link *link2;
   15112              : 
   15113      4604120 :           FOR_EACH_LOG_LINK (link2, place)
   15114      1049268 :             if (link2->insn == link->insn && link2->regno == link->regno)
   15115              :               break;
   15116              : 
   15117      3572166 :           if (link2 == NULL)
   15118              :             {
   15119      3554852 :               link->next = LOG_LINKS (place);
   15120      3554852 :               LOG_LINKS (place) = link;
   15121              : 
   15122              :               /* Set added_links_insn to the earliest insn we added a
   15123              :                  link to.  */
   15124      3554852 :               if (added_links_insn == 0
   15125      3554852 :                   || DF_INSN_LUID (added_links_insn) > DF_INSN_LUID (place))
   15126      2818564 :                 added_links_insn = place;
   15127              :             }
   15128              :         }
   15129              :     }
   15130     16327523 : }
   15131              : 
   15132              : /* Check for any register or memory mentioned in EQUIV that is not
   15133              :    mentioned in EXPR.  This is used to restrict EQUIV to "specializations"
   15134              :    of EXPR where some registers may have been replaced by constants.  */
   15135              : 
   15136              : static bool
   15137      2765221 : unmentioned_reg_p (rtx equiv, rtx expr)
   15138              : {
   15139      2765221 :   subrtx_iterator::array_type array;
   15140      7253888 :   FOR_EACH_SUBRTX (iter, array, equiv, NONCONST)
   15141              :     {
   15142      5881493 :       const_rtx x = *iter;
   15143      4034432 :       if ((REG_P (x) || MEM_P (x))
   15144      6266441 :           && !reg_mentioned_p (x, expr))
   15145      1392826 :         return true;
   15146              :     }
   15147      1372395 :   return false;
   15148      2765221 : }
   15149              : 
   15150              : /* Make pseudo-to-pseudo copies after every hard-reg-to-pseudo-copy, because
   15151              :    the reg-to-reg copy can usefully combine with later instructions, but we
   15152              :    do not want to combine the hard reg into later instructions, for that
   15153              :    restricts register allocation.  */
   15154              : static void
   15155      1064963 : make_more_copies (void)
   15156              : {
   15157      1064963 :   basic_block bb;
   15158              : 
   15159     11669262 :   FOR_EACH_BB_FN (bb, cfun)
   15160              :     {
   15161     10604299 :       rtx_insn *insn;
   15162              : 
   15163    140435070 :       FOR_BB_INSNS (bb, insn)
   15164              :         {
   15165    129830771 :           if (!NONDEBUG_INSN_P (insn))
   15166     69394614 :             continue;
   15167              : 
   15168     60436157 :           rtx set = single_set (insn);
   15169     60436157 :           if (!set)
   15170      4095080 :             continue;
   15171              : 
   15172     56341077 :           rtx dest = SET_DEST (set);
   15173     56341077 :           if (!(REG_P (dest) && !HARD_REGISTER_P (dest)))
   15174     32246046 :               continue;
   15175              : 
   15176     24095031 :           rtx src = SET_SRC (set);
   15177     24095031 :           if (!(REG_P (src) && HARD_REGISTER_P (src)))
   15178     21054502 :             continue;
   15179      3040529 :           if (TEST_HARD_REG_BIT (fixed_reg_set, REGNO (src)))
   15180         9980 :             continue;
   15181              : 
   15182      3030549 :           rtx new_reg = gen_reg_rtx (GET_MODE (dest));
   15183              : 
   15184              :           /* The "original" pseudo copies have important attributes
   15185              :              attached, like pointerness.  We want that for these copies
   15186              :              too, for use by insn recognition and later passes.  */
   15187      3030549 :           set_reg_attrs_from_value (new_reg, dest);
   15188              : 
   15189      3030549 :           rtx_insn *new_insn = gen_move_insn (new_reg, src);
   15190      3030549 :           SET_SRC (set) = new_reg;
   15191      3030549 :           emit_insn_before (new_insn, insn);
   15192      3030549 :           df_insn_rescan (insn);
   15193              :         }
   15194              :     }
   15195      1064963 : }
   15196              : 
   15197              : /* Try combining insns through substitution.  */
   15198              : static void
   15199      1064963 : rest_of_handle_combine (void)
   15200              : {
   15201      1064963 :   make_more_copies ();
   15202              : 
   15203      1064963 :   df_set_flags (DF_LR_RUN_DCE + DF_DEFER_INSN_RESCAN);
   15204      1064963 :   df_note_add_problem ();
   15205      1064963 :   df_analyze ();
   15206              : 
   15207      1064963 :   regstat_init_n_sets_and_refs ();
   15208      1064963 :   reg_n_sets_max = max_reg_num ();
   15209              : 
   15210      1064963 :   bool rebuild_jump_labels_after_combine
   15211      1064963 :     = combine_instructions (get_insns (), max_reg_num ());
   15212              : 
   15213              :   /* Combining insns may have turned an indirect jump into a
   15214              :      direct jump.  Rebuild the JUMP_LABEL fields of jumping
   15215              :      instructions.  */
   15216      1064963 :   if (rebuild_jump_labels_after_combine)
   15217              :     {
   15218         2378 :       if (dom_info_available_p (CDI_DOMINATORS))
   15219            0 :         free_dominance_info (CDI_DOMINATORS);
   15220         2378 :       timevar_push (TV_JUMP);
   15221         2378 :       rebuild_jump_labels (get_insns ());
   15222         2378 :       cleanup_cfg (0);
   15223         2378 :       timevar_pop (TV_JUMP);
   15224              :     }
   15225              : 
   15226      1064963 :   regstat_free_n_sets_and_refs ();
   15227      1064963 : }
   15228              : 
   15229              : namespace {
   15230              : 
   15231              : const pass_data pass_data_combine =
   15232              : {
   15233              :   RTL_PASS, /* type */
   15234              :   "combine", /* name */
   15235              :   OPTGROUP_NONE, /* optinfo_flags */
   15236              :   TV_COMBINE, /* tv_id */
   15237              :   PROP_cfglayout, /* properties_required */
   15238              :   0, /* properties_provided */
   15239              :   0, /* properties_destroyed */
   15240              :   0, /* todo_flags_start */
   15241              :   TODO_df_finish, /* todo_flags_finish */
   15242              : };
   15243              : 
   15244              : class pass_combine : public rtl_opt_pass
   15245              : {
   15246              : public:
   15247       297180 :   pass_combine (gcc::context *ctxt)
   15248       594360 :     : rtl_opt_pass (pass_data_combine, ctxt)
   15249              :   {}
   15250              : 
   15251              :   /* opt_pass methods: */
   15252      1517135 :   bool gate (function *) final override { return (optimize > 0); }
   15253      1064963 :   unsigned int execute (function *) final override
   15254              :     {
   15255      1064963 :       rest_of_handle_combine ();
   15256      1064963 :       return 0;
   15257              :     }
   15258              : 
   15259              : }; // class pass_combine
   15260              : 
   15261              : } // anon namespace
   15262              : 
   15263              : rtl_opt_pass *
   15264       297180 : make_pass_combine (gcc::context *ctxt)
   15265              : {
   15266       297180 :   return new pass_combine (ctxt);
   15267              : }
        

Generated by: LCOV version 2.5.0-beta

LCOV profile is generated on x86_64 machine using following configure options: configure --disable-bootstrap --enable-coverage=opt --enable-languages=c,c++,fortran,go,jit,lto,rust,m2 --enable-host-shared. GCC test suite is run with the built compiler.