LCOV - code coverage report
Current view: top level - gcc - cse.cc (source / functions) Coverage Total Hit
Test: gcc.info Lines: 91.5 % 3143 2875
Test Date: 2026-11-14 16:39:50 Functions: 92.6 % 95 88
Legend: Lines:     hit not hit

            Line data    Source code
       1              : /* Common subexpression elimination for GNU compiler.
       2              :    Copyright (C) 1987-2026 Free Software Foundation, Inc.
       3              : 
       4              : This file is part of GCC.
       5              : 
       6              : GCC is free software; you can redistribute it and/or modify it under
       7              : the terms of the GNU General Public License as published by the Free
       8              : Software Foundation; either version 3, or (at your option) any later
       9              : version.
      10              : 
      11              : GCC is distributed in the hope that it will be useful, but WITHOUT ANY
      12              : WARRANTY; without even the implied warranty of MERCHANTABILITY or
      13              : FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
      14              : for more details.
      15              : 
      16              : You should have received a copy of the GNU General Public License
      17              : along with GCC; see the file COPYING3.  If not see
      18              : <http://www.gnu.org/licenses/>.  */
      19              : 
      20              : #include "config.h"
      21              : #include "system.h"
      22              : #include "coretypes.h"
      23              : #include "backend.h"
      24              : #include "target.h"
      25              : #include "rtl.h"
      26              : #include "stmt.h"
      27              : #include "tree.h"
      28              : #include "cfghooks.h"
      29              : #include "df.h"
      30              : #include "memmodel.h"
      31              : #include "tm_p.h"
      32              : #include "insn-config.h"
      33              : #include "regs.h"
      34              : #include "emit-rtl.h"
      35              : #include "recog.h"
      36              : #include "cfgrtl.h"
      37              : #include "cfganal.h"
      38              : #include "cfgcleanup.h"
      39              : #include "alias.h"
      40              : #include "toplev.h"
      41              : #include "rtlhooks-def.h"
      42              : #include "tree-pass.h"
      43              : #include "dbgcnt.h"
      44              : #include "rtl-iter.h"
      45              : #include "function-abi.h"
      46              : #include "rtlanal.h"
      47              : #include "expr.h"
      48              : 
      49              : /* The basic idea of common subexpression elimination is to go
      50              :    through the code, keeping a record of expressions that would
      51              :    have the same value at the current scan point, and replacing
      52              :    expressions encountered with the cheapest equivalent expression.
      53              : 
      54              :    It is too complicated to keep track of the different possibilities
      55              :    when control paths merge in this code; so, at each label, we forget all
      56              :    that is known and start fresh.  This can be described as processing each
      57              :    extended basic block separately.  We have a separate pass to perform
      58              :    global CSE.
      59              : 
      60              :    Note CSE can turn a conditional or computed jump into a nop or
      61              :    an unconditional jump.  When this occurs we arrange to run the jump
      62              :    optimizer after CSE to delete the unreachable code.
      63              : 
      64              :    We use two data structures to record the equivalent expressions:
      65              :    a hash table for most expressions, and a vector of "quantity
      66              :    numbers" to record equivalent (pseudo) registers.
      67              : 
      68              :    The use of the special data structure for registers is desirable
      69              :    because it is faster.  It is possible because registers references
      70              :    contain a fairly small number, the register number, taken from
      71              :    a contiguously allocated series, and two register references are
      72              :    identical if they have the same number.  General expressions
      73              :    do not have any such thing, so the only way to retrieve the
      74              :    information recorded on an expression other than a register
      75              :    is to keep it in a hash table.
      76              : 
      77              : Registers and "quantity numbers":
      78              : 
      79              :    At the start of each basic block, all of the (hardware and pseudo)
      80              :    registers used in the function are given distinct quantity
      81              :    numbers to indicate their contents.  During scan, when the code
      82              :    copies one register into another, we copy the quantity number.
      83              :    When a register is loaded in any other way, we allocate a new
      84              :    quantity number to describe the value generated by this operation.
      85              :    `REG_QTY (N)' records what quantity register N is currently thought
      86              :    of as containing.
      87              : 
      88              :    All real quantity numbers are greater than or equal to zero.
      89              :    If register N has not been assigned a quantity, `REG_QTY (N)' will
      90              :    equal -N - 1, which is always negative.
      91              : 
      92              :    Quantity numbers below zero do not exist and none of the `qty_table'
      93              :    entries should be referenced with a negative index.
      94              : 
      95              :    We also maintain a bidirectional chain of registers for each
      96              :    quantity number.  The `qty_table` members `first_reg' and `last_reg',
      97              :    and `reg_eqv_table' members `next' and `prev' hold these chains.
      98              : 
      99              :    The first register in a chain is the one whose lifespan is least local.
     100              :    Among equals, it is the one that was seen first.
     101              :    We replace any equivalent register with that one.
     102              : 
     103              :    If two registers have the same quantity number, it must be true that
     104              :    REG expressions with qty_table `mode' must be in the hash table for both
     105              :    registers and must be in the same class.
     106              : 
     107              :    The converse is not true.  Since hard registers may be referenced in
     108              :    any mode, two REG expressions might be equivalent in the hash table
     109              :    but not have the same quantity number if the quantity number of one
     110              :    of the registers is not the same mode as those expressions.
     111              : 
     112              : Constants and quantity numbers
     113              : 
     114              :    When a quantity has a known constant value, that value is stored
     115              :    in the appropriate qty_table `const_rtx'.  This is in addition to
     116              :    putting the constant in the hash table as is usual for non-regs.
     117              : 
     118              :    Whether a reg or a constant is preferred is determined by the configuration
     119              :    macro CONST_COSTS and will often depend on the constant value.  In any
     120              :    event, expressions containing constants can be simplified, by fold_rtx.
     121              : 
     122              :    When a quantity has a known nearly constant value (such as an address
     123              :    of a stack slot), that value is stored in the appropriate qty_table
     124              :    `const_rtx'.
     125              : 
     126              :    Integer constants don't have a machine mode.  However, cse
     127              :    determines the intended machine mode from the destination
     128              :    of the instruction that moves the constant.  The machine mode
     129              :    is recorded in the hash table along with the actual RTL
     130              :    constant expression so that different modes are kept separate.
     131              : 
     132              : Other expressions:
     133              : 
     134              :    To record known equivalences among expressions in general
     135              :    we use a hash table called `table'.  It has a fixed number of buckets
     136              :    that contain chains of `struct table_elt' elements for expressions.
     137              :    These chains connect the elements whose expressions have the same
     138              :    hash codes.
     139              : 
     140              :    Other chains through the same elements connect the elements which
     141              :    currently have equivalent values.
     142              : 
     143              :    Register references in an expression are canonicalized before hashing
     144              :    the expression.  This is done using `reg_qty' and qty_table `first_reg'.
     145              :    The hash code of a register reference is computed using the quantity
     146              :    number, not the register number.
     147              : 
     148              :    When the value of an expression changes, it is necessary to remove from the
     149              :    hash table not just that expression but all expressions whose values
     150              :    could be different as a result.
     151              : 
     152              :      1. If the value changing is in memory, except in special cases
     153              :      ANYTHING referring to memory could be changed.  That is because
     154              :      nobody knows where a pointer does not point.
     155              :      The function `invalidate_memory' removes what is necessary.
     156              : 
     157              :      The special cases are when the address is constant or is
     158              :      a constant plus a fixed register such as the frame pointer
     159              :      or a static chain pointer.  When such addresses are stored in,
     160              :      we can tell exactly which other such addresses must be invalidated
     161              :      due to overlap.  `invalidate' does this.
     162              :      All expressions that refer to non-constant
     163              :      memory addresses are also invalidated.  `invalidate_memory' does this.
     164              : 
     165              :      2. If the value changing is a register, all expressions
     166              :      containing references to that register, and only those,
     167              :      must be removed.
     168              : 
     169              :    Because searching the entire hash table for expressions that contain
     170              :    a register is very slow, we try to figure out when it isn't necessary.
     171              :    Precisely, this is necessary only when expressions have been
     172              :    entered in the hash table using this register, and then the value has
     173              :    changed, and then another expression wants to be added to refer to
     174              :    the register's new value.  This sequence of circumstances is rare
     175              :    within any one basic block.
     176              : 
     177              :    `REG_TICK' and `REG_IN_TABLE', accessors for members of
     178              :    cse_reg_info, are used to detect this case.  REG_TICK (i) is
     179              :    incremented whenever a value is stored in register i.
     180              :    REG_IN_TABLE (i) holds -1 if no references to register i have been
     181              :    entered in the table; otherwise, it contains the value REG_TICK (i)
     182              :    had when the references were entered.  If we want to enter a
     183              :    reference and REG_IN_TABLE (i) != REG_TICK (i), we must scan and
     184              :    remove old references.  Until we want to enter a new entry, the
     185              :    mere fact that the two vectors don't match makes the entries be
     186              :    ignored if anyone tries to match them.
     187              : 
     188              :    Registers themselves are entered in the hash table as well as in
     189              :    the equivalent-register chains.  However, `REG_TICK' and
     190              :    `REG_IN_TABLE' do not apply to expressions which are simple
     191              :    register references.  These expressions are removed from the table
     192              :    immediately when they become invalid, and this can be done even if
     193              :    we do not immediately search for all the expressions that refer to
     194              :    the register.
     195              : 
     196              :    A CLOBBER rtx in an instruction invalidates its operand for further
     197              :    reuse.  A CLOBBER or SET rtx whose operand is a MEM:BLK
     198              :    invalidates everything that resides in memory.
     199              : 
     200              : Related expressions:
     201              : 
     202              :    Constant expressions that differ only by an additive integer
     203              :    are called related.  When a constant expression is put in
     204              :    the table, the related expression with no constant term
     205              :    is also entered.  These are made to point at each other
     206              :    so that it is possible to find out if there exists any
     207              :    register equivalent to an expression related to a given expression.  */
     208              : 
     209              : /* Length of qty_table vector.  We know in advance we will not need
     210              :    a quantity number this big.  */
     211              : 
     212              : static int max_qty;
     213              : 
     214              : /* Next quantity number to be allocated.
     215              :    This is 1 + the largest number needed so far.  */
     216              : 
     217              : static int next_qty;
     218              : 
     219              : /* Per-qty information tracking.
     220              : 
     221              :    `first_reg' and `last_reg' track the head and tail of the
     222              :    chain of registers which currently contain this quantity.
     223              : 
     224              :    `mode' contains the machine mode of this quantity.
     225              : 
     226              :    `const_rtx' holds the rtx of the constant value of this
     227              :    quantity, if known.  A summations of the frame/arg pointer
     228              :    and a constant can also be entered here.  When this holds
     229              :    a known value, `const_insn' is the insn which stored the
     230              :    constant value.
     231              : 
     232              :    `comparison_{code,const,qty}' are used to track when a
     233              :    comparison between a quantity and some constant or register has
     234              :    been passed.  In such a case, we know the results of the comparison
     235              :    in case we see it again.  These members record a comparison that
     236              :    is known to be true.  `comparison_code' holds the rtx code of such
     237              :    a comparison, else it is set to UNKNOWN and the other two
     238              :    comparison members are undefined.  `comparison_const' holds
     239              :    the constant being compared against, or zero if the comparison
     240              :    is not against a constant.  `comparison_qty' holds the quantity
     241              :    being compared against when the result is known.  If the comparison
     242              :    is not with a register, `comparison_qty' is INT_MIN.  */
     243              : 
     244              : struct qty_table_elem
     245              : {
     246              :   rtx const_rtx;
     247              :   rtx_insn *const_insn;
     248              :   rtx comparison_const;
     249              :   int comparison_qty;
     250              :   unsigned int first_reg, last_reg;
     251              :   machine_mode mode : MACHINE_MODE_BITSIZE;
     252              :   enum rtx_code comparison_code : RTX_CODE_BITSIZE;
     253              : };
     254              : 
     255              : /* The table of all qtys, indexed by qty number.  */
     256              : static struct qty_table_elem *qty_table;
     257              : 
     258              : /* Insn being scanned.  */
     259              : 
     260              : static rtx_insn *this_insn;
     261              : static bool optimize_this_for_speed_p;
     262              : 
     263              : /* Index by register number, gives the number of the next (or
     264              :    previous) register in the chain of registers sharing the same
     265              :    value.
     266              : 
     267              :    Or -1 if this register is at the end of the chain.
     268              : 
     269              :    If REG_QTY (N) == -N - 1, reg_eqv_table[N].next is undefined.  */
     270              : 
     271              : /* Per-register equivalence chain.  */
     272              : struct reg_eqv_elem
     273              : {
     274              :   int next, prev;
     275              : };
     276              : 
     277              : /* The table of all register equivalence chains.  */
     278              : static struct reg_eqv_elem *reg_eqv_table;
     279              : 
     280              : struct cse_reg_info
     281              : {
     282              :   /* The timestamp at which this register is initialized.  */
     283              :   unsigned int timestamp;
     284              : 
     285              :   /* The quantity number of the register's current contents.  */
     286              :   int reg_qty;
     287              : 
     288              :   /* The number of times the register has been altered in the current
     289              :      basic block.  */
     290              :   int reg_tick;
     291              : 
     292              :   /* The REG_TICK value at which rtx's containing this register are
     293              :      valid in the hash table.  If this does not equal the current
     294              :      reg_tick value, such expressions existing in the hash table are
     295              :      invalid.  */
     296              :   int reg_in_table;
     297              : 
     298              :   /* The SUBREG that was set when REG_TICK was last incremented.  Set
     299              :      to -1 if the last store was to the whole register, not a subreg.  */
     300              :   unsigned int subreg_ticked;
     301              : };
     302              : 
     303              : /* A table of cse_reg_info indexed by register numbers.  */
     304              : static struct cse_reg_info *cse_reg_info_table;
     305              : 
     306              : /* The size of the above table.  */
     307              : static unsigned int cse_reg_info_table_size;
     308              : 
     309              : /* The index of the first entry that has not been initialized.  */
     310              : static unsigned int cse_reg_info_table_first_uninitialized;
     311              : 
     312              : /* The timestamp at the beginning of the current run of
     313              :    cse_extended_basic_block.  We increment this variable at the beginning of
     314              :    the current run of cse_extended_basic_block.  The timestamp field of a
     315              :    cse_reg_info entry matches the value of this variable if and only
     316              :    if the entry has been initialized during the current run of
     317              :    cse_extended_basic_block.  */
     318              : static unsigned int cse_reg_info_timestamp;
     319              : 
     320              : /* A HARD_REG_SET containing all the hard registers for which there is
     321              :    currently a REG expression in the hash table.  Note the difference
     322              :    from the above variables, which indicate if the REG is mentioned in some
     323              :    expression in the table.  */
     324              : 
     325              : static HARD_REG_SET hard_regs_in_table;
     326              : 
     327              : /* True if CSE has altered the CFG.  */
     328              : static bool cse_cfg_altered;
     329              : 
     330              : /* True if CSE has altered conditional jump insns in such a way
     331              :    that jump optimization should be redone.  */
     332              : static bool cse_jumps_altered;
     333              : 
     334              : /* True if we put a LABEL_REF into the hash table for an INSN
     335              :    without a REG_LABEL_OPERAND, we have to rerun jump after CSE
     336              :    to put in the note.  */
     337              : static bool recorded_label_ref;
     338              : 
     339              : /* canon_hash stores 1 in do_not_record if it notices a reference to PC or
     340              :    some other volatile subexpression.  */
     341              : 
     342              : static int do_not_record;
     343              : 
     344              : /* canon_hash stores 1 in hash_arg_in_memory
     345              :    if it notices a reference to memory within the expression being hashed.  */
     346              : 
     347              : static int hash_arg_in_memory;
     348              : 
     349              : /* The hash table contains buckets which are chains of `struct table_elt's,
     350              :    each recording one expression's information.
     351              :    That expression is in the `exp' field.
     352              : 
     353              :    The canon_exp field contains a canonical (from the point of view of
     354              :    alias analysis) version of the `exp' field.
     355              : 
     356              :    Those elements with the same hash code are chained in both directions
     357              :    through the `next_same_hash' and `prev_same_hash' fields.
     358              : 
     359              :    Each set of expressions with equivalent values
     360              :    are on a two-way chain through the `next_same_value'
     361              :    and `prev_same_value' fields, and all point with
     362              :    the `first_same_value' field at the first element in
     363              :    that chain.  The chain is in order of increasing cost.
     364              :    Each element's cost value is in its `cost' field.
     365              : 
     366              :    The `in_memory' field is nonzero for elements that
     367              :    involve any reference to memory.  These elements are removed
     368              :    whenever a write is done to an unidentified location in memory.
     369              :    To be safe, we assume that a memory address is unidentified unless
     370              :    the address is either a symbol constant or a constant plus
     371              :    the frame pointer or argument pointer.
     372              : 
     373              :    The `related_value' field is used to connect related expressions
     374              :    (that differ by adding an integer).
     375              :    The related expressions are chained in a circular fashion.
     376              :    `related_value' is zero for expressions for which this
     377              :    chain is not useful.
     378              : 
     379              :    The `cost' field stores the cost of this element's expression.
     380              :    The `regcost' field stores the value returned by approx_reg_cost for
     381              :    this element's expression.
     382              : 
     383              :    The `is_const' flag is set if the element is a constant (including
     384              :    a fixed address).
     385              : 
     386              :    The `flag' field is used as a temporary during some search routines.
     387              : 
     388              :    The `mode' field is usually the same as GET_MODE (`exp'), but
     389              :    if `exp' is a CONST_INT and has no machine mode then the `mode'
     390              :    field is the mode it was being used as.  Each constant is
     391              :    recorded separately for each mode it is used with.  */
     392              : 
     393              : struct table_elt
     394              : {
     395              :   rtx exp;
     396              :   rtx canon_exp;
     397              :   struct table_elt *next_same_hash;
     398              :   struct table_elt *prev_same_hash;
     399              :   struct table_elt *next_same_value;
     400              :   struct table_elt *prev_same_value;
     401              :   struct table_elt *first_same_value;
     402              :   struct table_elt *related_value;
     403              :   int cost;
     404              :   int regcost;
     405              :   machine_mode mode : MACHINE_MODE_BITSIZE;
     406              :   char in_memory;
     407              :   char is_const;
     408              :   char flag;
     409              : };
     410              : 
     411              : /* We don't want a lot of buckets, because we rarely have very many
     412              :    things stored in the hash table, and a lot of buckets slows
     413              :    down a lot of loops that happen frequently.  */
     414              : #define HASH_SHIFT      5
     415              : #define HASH_SIZE       (1 << HASH_SHIFT)
     416              : #define HASH_MASK       (HASH_SIZE - 1)
     417              : 
     418              : /* Determine whether register number N is considered a fixed register for the
     419              :    purpose of approximating register costs.
     420              :    It is desirable to replace other regs with fixed regs, to reduce need for
     421              :    non-fixed hard regs.
     422              :    A reg wins if it is either the frame pointer or designated as fixed.  */
     423              : #define FIXED_REGNO_P(N)  \
     424              :   ((N) == FRAME_POINTER_REGNUM || (N) == HARD_FRAME_POINTER_REGNUM \
     425              :    || fixed_regs[N] || global_regs[N])
     426              : 
     427              : /* Compute cost of X, as stored in the `cost' field of a table_elt.  Fixed
     428              :    hard registers and pointers into the frame are the cheapest with a cost
     429              :    of 0.  Next come pseudos with a cost of one and other hard registers with
     430              :    a cost of 2.  Aside from these special cases, call `rtx_cost'.  */
     431              : 
     432              : #define CHEAP_REGNO(N)                                                  \
     433              :   (REGNO_PTR_FRAME_P (N)                                                \
     434              :    || (HARD_REGISTER_NUM_P (N)                                          \
     435              :        && FIXED_REGNO_P (N) && REGNO_REG_CLASS (N) != NO_REGS))
     436              : 
     437              : #define COST(X, MODE)                                                   \
     438              :   (REG_P (X) ? 0 : notreg_cost (X, MODE, SET, 1))
     439              : #define COST_IN(X, MODE, OUTER, OPNO)                                   \
     440              :   (REG_P (X) ? 0 : notreg_cost (X, MODE, OUTER, OPNO))
     441              : 
     442              : /* Get the number of times this register has been updated in this
     443              :    basic block.  */
     444              : 
     445              : #define REG_TICK(N) (get_cse_reg_info (N)->reg_tick)
     446              : 
     447              : /* Get the point at which REG was recorded in the table.  */
     448              : 
     449              : #define REG_IN_TABLE(N) (get_cse_reg_info (N)->reg_in_table)
     450              : 
     451              : /* Get the SUBREG set at the last increment to REG_TICK (-1 if not a
     452              :    SUBREG).  */
     453              : 
     454              : #define SUBREG_TICKED(N) (get_cse_reg_info (N)->subreg_ticked)
     455              : 
     456              : /* Get the quantity number for REG.  */
     457              : 
     458              : #define REG_QTY(N) (get_cse_reg_info (N)->reg_qty)
     459              : 
     460              : /* Determine if the quantity number for register X represents a valid index
     461              :    into the qty_table.  */
     462              : 
     463              : #define REGNO_QTY_VALID_P(N) (REG_QTY (N) >= 0)
     464              : 
     465              : /* Compare table_elt X and Y and return true iff X is cheaper than Y.  */
     466              : 
     467              : #define CHEAPER(X, Y) \
     468              :  (preferable ((X)->cost, (X)->regcost, (Y)->cost, (Y)->regcost) < 0)
     469              : 
     470              : static struct table_elt *table[HASH_SIZE];
     471              : 
     472              : /* Chain of `struct table_elt's made so far for this function
     473              :    but currently removed from the table.  */
     474              : 
     475              : static struct table_elt *free_element_chain;
     476              : 
     477              : /* Trace a patch through the CFG.  */
     478              : 
     479              : struct branch_path
     480              : {
     481              :   /* The basic block for this path entry.  */
     482              :   basic_block bb;
     483              : };
     484              : 
     485              : /* This data describes a pair of vec_duplicates in the same BB, which duplicate
     486              :    the same pseudo to different vector lengths.  The same structure is also
     487              :    used while prescanning a basic block as a temporary cache entry.  */
     488              : 
     489       340471 : struct cse_vec_duplicate_match
     490              : {
     491              :   basic_block bb;
     492              :   machine_mode widest_mode;
     493              :   rtx scalar;
     494              :   rtx_insn *first_insn;
     495              :   rtx_insn *widest_insn;
     496              :   rtx_insn *last_checked_insn;
     497              :   auto_vec<rtx_insn *> related_dups;
     498              : 
     499       340409 :   cse_vec_duplicate_match (basic_block bb_, machine_mode widest_mode_,
     500              :                            rtx scalar_, rtx_insn *first_insn_,
     501              :                            rtx_insn *widest_insn_,
     502              :                            rtx_insn *last_checked_insn_)
     503       340409 :     : bb (bb_), widest_mode (widest_mode_), scalar (scalar_),
     504       340409 :       first_insn (first_insn_), widest_insn (widest_insn_),
     505       340409 :       last_checked_insn (last_checked_insn_)
     506              :   {}
     507              : 
     508       344351 :   cse_vec_duplicate_match (const cse_vec_duplicate_match &other)
     509       344351 :     : bb (other.bb), widest_mode (other.widest_mode), scalar (other.scalar),
     510       344351 :       first_insn (other.first_insn), widest_insn (other.widest_insn),
     511       344351 :       last_checked_insn (other.last_checked_insn)
     512              :   {
     513       688702 :     related_dups.reserve (other.related_dups.length ());
     514       344351 :     related_dups.splice (other.related_dups);
     515       344351 :   }
     516              : };
     517              : 
     518              : /* This data describes a block that will be processed by
     519              :    cse_extended_basic_block.  */
     520              : 
     521      2345959 : struct cse_basic_block_data
     522              : {
     523              :   /* Total number of SETs in block.  */
     524              :   int nsets;
     525              :   /* Size of current branch path, if any.  */
     526              :   int path_size;
     527              :   /* Current path, indicating which basic_blocks will be processed.  */
     528              :   struct branch_path *path;
     529              :   /* vec_duplicate sources seen in the current BB while prescanning.  */
     530              :   auto_vec<cse_vec_duplicate_match, 8> vec_duplicate_cache;
     531              :   /* Syntactic vec_duplicate matches found in the same BB while prescanning.  */
     532              :   auto_vec<cse_vec_duplicate_match, 8> vec_duplicate_matches;
     533              : };
     534              : 
     535              : 
     536              : /* Pointers to the live in/live out bitmaps for the boundaries of the
     537              :    current EBB.  */
     538              : static bitmap cse_ebb_live_in, cse_ebb_live_out;
     539              : 
     540              : /* A simple bitmap to track which basic blocks have been visited
     541              :    already as part of an already processed extended basic block.  */
     542              : static sbitmap cse_visited_basic_blocks;
     543              : 
     544              : static bool fixed_base_plus_p (rtx x);
     545              : static int notreg_cost (rtx, machine_mode, enum rtx_code, int);
     546              : static int preferable (int, int, int, int);
     547              : static void new_basic_block (void);
     548              : static void make_new_qty (unsigned int, machine_mode);
     549              : static void make_regs_eqv (unsigned int, unsigned int);
     550              : static void delete_reg_equiv (unsigned int);
     551              : static bool mention_regs (rtx);
     552              : static bool insert_regs (rtx, struct table_elt *, bool);
     553              : static void remove_from_table (struct table_elt *, unsigned);
     554              : static void remove_pseudo_from_table (rtx, unsigned);
     555              : static struct table_elt *lookup (rtx, unsigned, machine_mode);
     556              : static struct table_elt *lookup_for_remove (rtx, unsigned, machine_mode);
     557              : static rtx lookup_as_function (rtx, enum rtx_code);
     558              : static struct table_elt *insert_with_costs (rtx, struct table_elt *, unsigned,
     559              :                                             machine_mode, int, int);
     560              : static struct table_elt *insert (rtx, struct table_elt *, unsigned,
     561              :                                  machine_mode);
     562              : static void merge_equiv_classes (struct table_elt *, struct table_elt *);
     563              : static void invalidate (rtx, machine_mode);
     564              : static void remove_invalid_refs (unsigned int);
     565              : static void remove_invalid_subreg_refs (unsigned int, poly_uint64,
     566              :                                         machine_mode);
     567              : static void rehash_using_reg (rtx);
     568              : static void invalidate_memory (void);
     569              : static rtx use_related_value (rtx, struct table_elt *);
     570              : 
     571              : static inline unsigned canon_hash (rtx, machine_mode);
     572              : static inline unsigned safe_hash (rtx, machine_mode);
     573              : static inline unsigned hash_rtx_string (const char *);
     574              : 
     575              : static rtx canon_reg (rtx, rtx_insn *);
     576              : static enum rtx_code find_comparison_args (enum rtx_code, rtx *, rtx *,
     577              :                                            machine_mode *,
     578              :                                            machine_mode *);
     579              : static rtx fold_rtx (rtx, rtx_insn *);
     580              : static rtx equiv_constant (rtx);
     581              : static void record_jump_equiv (rtx_insn *, bool);
     582              : static void record_jump_cond (enum rtx_code, machine_mode, rtx, rtx);
     583              : static void cse_insn (rtx_insn *);
     584              : static void cse_prescan_cache_vec_dup (struct cse_basic_block_data *,
     585              :                                        basic_block, rtx_insn *, rtx);
     586              : static void cse_prescan_path (struct cse_basic_block_data *);
     587              : static void invalidate_from_clobbers (rtx_insn *);
     588              : static void invalidate_from_sets_and_clobbers (rtx_insn *);
     589              : static void cse_extended_basic_block (struct cse_basic_block_data *);
     590              : extern void dump_class (struct table_elt*);
     591              : static void get_cse_reg_info_1 (unsigned int regno);
     592              : static struct cse_reg_info * get_cse_reg_info (unsigned int regno);
     593              : 
     594              : static void flush_hash_table (void);
     595              : static bool insn_live_p (rtx_insn *, int *);
     596              : static bool set_live_p (rtx, int *);
     597              : static void cse_change_cc_mode_insn (rtx_insn *, rtx);
     598              : static void cse_change_cc_mode_insns (rtx_insn *, rtx_insn *, rtx);
     599              : static machine_mode cse_cc_succs (basic_block, basic_block, rtx, rtx,
     600              :                                        bool);
     601              : 
     602              : 
     603              : #undef RTL_HOOKS_GEN_LOWPART
     604              : #define RTL_HOOKS_GEN_LOWPART           gen_lowpart_if_possible
     605              : 
     606              : static const struct rtl_hooks cse_rtl_hooks = RTL_HOOKS_INITIALIZER;
     607              : 
     608              : /* Compute hash code of X in mode M.  Special-case case where X is a pseudo
     609              :    register (hard registers may require `do_not_record' to be set).  */
     610              : 
     611              : static inline unsigned
     612    859159334 : HASH (rtx x, machine_mode mode)
     613              : {
     614    557767449 :   unsigned h = (REG_P (x) && REGNO (x) >= FIRST_PSEUDO_REGISTER
     615   1191617007 :                 ? (((unsigned) REG << 7) + (unsigned) REG_QTY (REGNO (x)))
     616    859159334 :                 : canon_hash (x, mode));
     617    859159334 :   return (h ^ (h >> HASH_SHIFT)) & HASH_MASK;
     618              : }
     619              : 
     620              : /* Like HASH, but without side-effects.  */
     621              : 
     622              : static inline unsigned
     623    237083770 : SAFE_HASH (rtx x, machine_mode mode)
     624              : {
     625    118887070 :   unsigned h = (REG_P (x) && REGNO (x) >= FIRST_PSEUDO_REGISTER
     626    304175580 :                 ? (((unsigned) REG << 7) + (unsigned) REG_QTY (REGNO (x)))
     627    237083770 :                 : safe_hash (x, mode));
     628    237083770 :   return (h ^ (h >> HASH_SHIFT)) & HASH_MASK;
     629              : }
     630              : 
     631              : /* Nonzero if X has the form (PLUS frame-pointer integer).  */
     632              : 
     633              : static bool
     634    243465777 : fixed_base_plus_p (rtx x)
     635              : {
     636    276222495 :   switch (GET_CODE (x))
     637              :     {
     638    143980252 :     case REG:
     639    143980252 :       if (x == frame_pointer_rtx || x == hard_frame_pointer_rtx)
     640              :         return true;
     641    129356638 :       if (x == arg_pointer_rtx && fixed_regs[ARG_POINTER_REGNUM])
     642       119167 :         return true;
     643              :       return false;
     644              : 
     645     39180854 :     case PLUS:
     646     39180854 :       if (!CONST_INT_P (XEXP (x, 1)))
     647              :         return false;
     648     32756718 :       return fixed_base_plus_p (XEXP (x, 0));
     649              : 
     650              :     default:
     651              :       return false;
     652              :     }
     653              : }
     654              : 
     655              : /* Dump the expressions in the equivalence class indicated by CLASSP.
     656              :    This function is used only for debugging.  */
     657              : DEBUG_FUNCTION void
     658            0 : dump_class (struct table_elt *classp)
     659              : {
     660            0 :   struct table_elt *elt;
     661              : 
     662            0 :   fprintf (stderr, "Equivalence chain for ");
     663            0 :   print_rtl (stderr, classp->exp);
     664            0 :   fprintf (stderr, ": \n");
     665              : 
     666            0 :   for (elt = classp->first_same_value; elt; elt = elt->next_same_value)
     667              :     {
     668            0 :       print_rtl (stderr, elt->exp);
     669            0 :       fprintf (stderr, "\n");
     670              :     }
     671            0 : }
     672              : 
     673              : /* Return an estimate of the cost of the registers used in an rtx.
     674              :    This is mostly the number of different REG expressions in the rtx;
     675              :    however for some exceptions like fixed registers we use a cost of
     676              :    0.  If any other hard register reference occurs, return MAX_COST.  */
     677              : 
     678              : static int
     679    445017668 : approx_reg_cost (const_rtx x)
     680              : {
     681    445017668 :   int cost = 0;
     682    445017668 :   subrtx_iterator::array_type array;
     683   1403840971 :   FOR_EACH_SUBRTX (iter, array, x, NONCONST)
     684              :     {
     685   1016308083 :       const_rtx x = *iter;
     686   1016308083 :       if (REG_P (x))
     687              :         {
     688    421013676 :           unsigned int regno = REGNO (x);
     689    421013676 :           if (!CHEAP_REGNO (regno))
     690              :             {
     691     57484780 :               if (regno < FIRST_PSEUDO_REGISTER)
     692              :                 {
     693     57484780 :                   if (targetm.small_register_classes_for_mode_p (GET_MODE (x)))
     694     57484780 :                     return MAX_COST;
     695            0 :                   cost += 2;
     696              :                 }
     697              :               else
     698    299327286 :                 cost += 1;
     699              :             }
     700              :         }
     701              :     }
     702    387532888 :   return cost;
     703    445017668 : }
     704              : 
     705              : /* Return a negative value if an rtx A, whose costs are given by COST_A
     706              :    and REGCOST_A, is more desirable than an rtx B.
     707              :    Return a positive value if A is less desirable, or 0 if the two are
     708              :    equally good.  */
     709              : static int
     710    676300643 : preferable (int cost_a, int regcost_a, int cost_b, int regcost_b)
     711              : {
     712              :   /* First, get rid of cases involving expressions that are entirely
     713              :      unwanted.  */
     714    676300643 :   if (cost_a != cost_b)
     715              :     {
     716    632384590 :       if (cost_a == MAX_COST)
     717              :         return 1;
     718    630989037 :       if (cost_b == MAX_COST)
     719              :         return -1;
     720              :     }
     721              : 
     722              :   /* Avoid extending lifetimes of hardregs.  */
     723    177758553 :   if (regcost_a != regcost_b)
     724              :     {
     725     97084298 :       if (regcost_a == MAX_COST)
     726              :         return 1;
     727     75365490 :       if (regcost_b == MAX_COST)
     728              :         return -1;
     729              :     }
     730              : 
     731              :   /* Normal operation costs take precedence.  */
     732    153962095 :   if (cost_a != cost_b)
     733    110169699 :     return cost_a - cost_b;
     734              :   /* Only if these are identical consider effects on register pressure.  */
     735     43792396 :   if (regcost_a != regcost_b)
     736     43792396 :     return regcost_a - regcost_b;
     737              :   return 0;
     738              : }
     739              : 
     740              : /* Internal function, to compute cost when X is not a register; called
     741              :    from COST macro to keep it simple.  */
     742              : 
     743              : static int
     744    313619060 : notreg_cost (rtx x, machine_mode mode, enum rtx_code outer, int opno)
     745              : {
     746    313619060 :   scalar_int_mode int_mode, inner_mode;
     747    313619060 :   return ((GET_CODE (x) == SUBREG
     748      5562419 :            && REG_P (SUBREG_REG (x))
     749    315801745 :            && is_int_mode (mode, &int_mode)
     750    314919401 :            && is_int_mode (GET_MODE (SUBREG_REG (x)), &inner_mode)
     751      7842312 :            && GET_MODE_SIZE (int_mode) < GET_MODE_SIZE (inner_mode)
     752      3869945 :            && subreg_lowpart_p (x)
     753      2620815 :            && TRULY_NOOP_TRUNCATION_MODES_P (int_mode, inner_mode))
     754    313619060 :           ? 0
     755    310998245 :           : rtx_cost (x, mode, outer, opno, optimize_this_for_speed_p) * 2);
     756              : }
     757              : 
     758              : 
     759              : /* Initialize CSE_REG_INFO_TABLE.  */
     760              : 
     761              : static void
     762      2345959 : init_cse_reg_info (unsigned int nregs)
     763              : {
     764              :   /* Do we need to grow the table?  */
     765      2345959 :   if (nregs > cse_reg_info_table_size)
     766              :     {
     767       182434 :       unsigned int new_size;
     768              : 
     769       182434 :       if (cse_reg_info_table_size < 2048)
     770              :         {
     771              :           /* Compute a new size that is a power of 2 and no smaller
     772              :              than the large of NREGS and 64.  */
     773       182091 :           new_size = (cse_reg_info_table_size
     774       182091 :                       ? cse_reg_info_table_size : 64);
     775              : 
     776       405954 :           while (new_size < nregs)
     777       223863 :             new_size *= 2;
     778              :         }
     779              :       else
     780              :         {
     781              :           /* If we need a big table, allocate just enough to hold
     782              :              NREGS registers.  */
     783              :           new_size = nregs;
     784              :         }
     785              : 
     786              :       /* Reallocate the table with NEW_SIZE entries.  */
     787       182434 :       free (cse_reg_info_table);
     788       182434 :       cse_reg_info_table = XNEWVEC (struct cse_reg_info, new_size);
     789       182434 :       cse_reg_info_table_size = new_size;
     790       182434 :       cse_reg_info_table_first_uninitialized = 0;
     791              :     }
     792              : 
     793              :   /* Do we have all of the first NREGS entries initialized?  */
     794      2345959 :   if (cse_reg_info_table_first_uninitialized < nregs)
     795              :     {
     796       333224 :       unsigned int old_timestamp = cse_reg_info_timestamp - 1;
     797       333224 :       unsigned int i;
     798              : 
     799              :       /* Put the old timestamp on newly allocated entries so that they
     800              :          will all be considered out of date.  We do not touch those
     801              :          entries beyond the first NREGS entries to be nice to the
     802              :          virtual memory.  */
     803     34610354 :       for (i = cse_reg_info_table_first_uninitialized; i < nregs; i++)
     804     34277130 :         cse_reg_info_table[i].timestamp = old_timestamp;
     805              : 
     806       333224 :       cse_reg_info_table_first_uninitialized = nregs;
     807              :     }
     808      2345959 : }
     809              : 
     810              : /* Given REGNO, initialize the cse_reg_info entry for REGNO.  */
     811              : 
     812              : static void
     813    896889335 : get_cse_reg_info_1 (unsigned int regno)
     814              : {
     815              :   /* Set TIMESTAMP field to CSE_REG_INFO_TIMESTAMP so that this
     816              :      entry will be considered to have been initialized.  */
     817    896889335 :   cse_reg_info_table[regno].timestamp = cse_reg_info_timestamp;
     818              : 
     819              :   /* Initialize the rest of the entry.  */
     820    896889335 :   cse_reg_info_table[regno].reg_tick = 1;
     821    896889335 :   cse_reg_info_table[regno].reg_in_table = -1;
     822    896889335 :   cse_reg_info_table[regno].subreg_ticked = -1;
     823    896889335 :   cse_reg_info_table[regno].reg_qty = -regno - 1;
     824    896889335 : }
     825              : 
     826              : /* Find a cse_reg_info entry for REGNO.  */
     827              : 
     828              : static inline struct cse_reg_info *
     829  11608316966 : get_cse_reg_info (unsigned int regno)
     830              : {
     831  11608316966 :   struct cse_reg_info *p = &cse_reg_info_table[regno];
     832              : 
     833              :   /* If this entry has not been initialized, go ahead and initialize
     834              :      it.  */
     835  11608316966 :   if (p->timestamp != cse_reg_info_timestamp)
     836    896889335 :     get_cse_reg_info_1 (regno);
     837              : 
     838  11608316966 :   return p;
     839              : }
     840              : 
     841              : /* Clear the hash table and initialize each register with its own quantity,
     842              :    for a new basic block.  */
     843              : 
     844              : static void
     845     21190623 : new_basic_block (void)
     846              : {
     847     21190623 :   int i;
     848              : 
     849     21190623 :   next_qty = 0;
     850              : 
     851              :   /* Invalidate cse_reg_info_table.  */
     852     21190623 :   cse_reg_info_timestamp++;
     853              : 
     854              :   /* Clear out hash table state for this pass.  */
     855     21190623 :   CLEAR_HARD_REG_SET (hard_regs_in_table);
     856              : 
     857              :   /* The per-quantity values used to be initialized here, but it is
     858              :      much faster to initialize each as it is made in `make_new_qty'.  */
     859              : 
     860    699290559 :   for (i = 0; i < HASH_SIZE; i++)
     861              :     {
     862    678099936 :       struct table_elt *first;
     863              : 
     864    678099936 :       first = table[i];
     865    678099936 :       if (first != NULL)
     866              :         {
     867    139506565 :           struct table_elt *last = first;
     868              : 
     869    139506565 :           table[i] = NULL;
     870              : 
     871    198243915 :           while (last->next_same_hash != NULL)
     872              :             last = last->next_same_hash;
     873              : 
     874              :           /* Now relink this hash entire chain into
     875              :              the free element list.  */
     876              : 
     877    139506565 :           last->next_same_hash = free_element_chain;
     878    139506565 :           free_element_chain = first;
     879              :         }
     880              :     }
     881     21190623 : }
     882              : 
     883              : /* Say that register REG contains a quantity in mode MODE not in any
     884              :    register before and initialize that quantity.  */
     885              : 
     886              : static void
     887    106785130 : make_new_qty (unsigned int reg, machine_mode mode)
     888              : {
     889    106785130 :   int q;
     890    106785130 :   struct qty_table_elem *ent;
     891    106785130 :   struct reg_eqv_elem *eqv;
     892              : 
     893    106785130 :   gcc_assert (next_qty < max_qty);
     894              : 
     895    106785130 :   q = REG_QTY (reg) = next_qty++;
     896    106785130 :   ent = &qty_table[q];
     897    106785130 :   ent->first_reg = reg;
     898    106785130 :   ent->last_reg = reg;
     899    106785130 :   ent->mode = mode;
     900    106785130 :   ent->const_rtx = ent->const_insn = NULL;
     901    106785130 :   ent->comparison_code = UNKNOWN;
     902              : 
     903    106785130 :   eqv = &reg_eqv_table[reg];
     904    106785130 :   eqv->next = eqv->prev = -1;
     905    106785130 : }
     906              : 
     907              : /* Make reg NEW equivalent to reg OLD.
     908              :    OLD is not changing; NEW is.  */
     909              : 
     910              : static void
     911     11918961 : make_regs_eqv (unsigned int new_reg, unsigned int old_reg)
     912              : {
     913     11918961 :   unsigned int lastr, firstr;
     914     11918961 :   int q = REG_QTY (old_reg);
     915     11918961 :   struct qty_table_elem *ent;
     916              : 
     917     11918961 :   ent = &qty_table[q];
     918              : 
     919              :   /* Nothing should become eqv until it has a "non-invalid" qty number.  */
     920     11918961 :   gcc_assert (REGNO_QTY_VALID_P (old_reg));
     921              : 
     922     11918961 :   REG_QTY (new_reg) = q;
     923     11918961 :   firstr = ent->first_reg;
     924     11918961 :   lastr = ent->last_reg;
     925              : 
     926              :   /* Prefer fixed hard registers to anything.  Prefer pseudo regs to other
     927              :      hard regs.  Among pseudos, if NEW will live longer than any other reg
     928              :      of the same qty, and that is beyond the current basic block,
     929              :      make it the new canonical replacement for this qty.  */
     930       317565 :   if (! (firstr < FIRST_PSEUDO_REGISTER && FIXED_REGNO_P (firstr))
     931              :       /* Certain fixed registers might be of the class NO_REGS.  This means
     932              :          that not only can they not be allocated by the compiler, but
     933              :          they cannot be used in substitutions or canonicalizations
     934              :          either.  */
     935     11601396 :       && (new_reg >= FIRST_PSEUDO_REGISTER || REGNO_REG_CLASS (new_reg) != NO_REGS)
     936     11924061 :       && ((new_reg < FIRST_PSEUDO_REGISTER && FIXED_REGNO_P (new_reg))
     937     11596296 :           || (new_reg >= FIRST_PSEUDO_REGISTER
     938     11596296 :               && (firstr < FIRST_PSEUDO_REGISTER
     939     11596296 :                   || (bitmap_bit_p (cse_ebb_live_out, new_reg)
     940      3738570 :                       && !bitmap_bit_p (cse_ebb_live_out, firstr))
     941      9515929 :                   || (bitmap_bit_p (cse_ebb_live_in, new_reg)
     942       514176 :                       && !bitmap_bit_p (cse_ebb_live_in, firstr))))))
     943              :     {
     944      2186718 :       reg_eqv_table[firstr].prev = new_reg;
     945      2186718 :       reg_eqv_table[new_reg].next = firstr;
     946      2186718 :       reg_eqv_table[new_reg].prev = -1;
     947      2186718 :       ent->first_reg = new_reg;
     948              :     }
     949              :   else
     950              :     {
     951              :       /* If NEW is a hard reg (known to be non-fixed), insert at end.
     952              :          Otherwise, insert before any non-fixed hard regs that are at the
     953              :          end.  Registers of class NO_REGS cannot be used as an
     954              :          equivalent for anything.  */
     955       298715 :       while (lastr < FIRST_PSEUDO_REGISTER && reg_eqv_table[lastr].prev >= 0
     956            0 :              && (REGNO_REG_CLASS (lastr) == NO_REGS || ! FIXED_REGNO_P (lastr))
     957      9732243 :              && new_reg >= FIRST_PSEUDO_REGISTER)
     958            0 :         lastr = reg_eqv_table[lastr].prev;
     959      9732243 :       reg_eqv_table[new_reg].next = reg_eqv_table[lastr].next;
     960      9732243 :       if (reg_eqv_table[lastr].next >= 0)
     961            0 :         reg_eqv_table[reg_eqv_table[lastr].next].prev = new_reg;
     962              :       else
     963      9732243 :         qty_table[q].last_reg = new_reg;
     964      9732243 :       reg_eqv_table[lastr].next = new_reg;
     965      9732243 :       reg_eqv_table[new_reg].prev = lastr;
     966              :     }
     967     11918961 : }
     968              : 
     969              : /* Remove REG from its equivalence class.  */
     970              : 
     971              : static void
     972   1556080260 : delete_reg_equiv (unsigned int reg)
     973              : {
     974   1556080260 :   struct qty_table_elem *ent;
     975   1556080260 :   int q = REG_QTY (reg);
     976   1556080260 :   int p, n;
     977              : 
     978              :   /* If invalid, do nothing.  */
     979   1556080260 :   if (! REGNO_QTY_VALID_P (reg))
     980              :     return;
     981              : 
     982     19163466 :   ent = &qty_table[q];
     983              : 
     984     19163466 :   p = reg_eqv_table[reg].prev;
     985     19163466 :   n = reg_eqv_table[reg].next;
     986              : 
     987     19163466 :   if (n != -1)
     988       680842 :     reg_eqv_table[n].prev = p;
     989              :   else
     990     18482624 :     ent->last_reg = p;
     991     19163466 :   if (p != -1)
     992       694751 :     reg_eqv_table[p].next = n;
     993              :   else
     994     18468715 :     ent->first_reg = n;
     995              : 
     996     19163466 :   REG_QTY (reg) = -reg - 1;
     997              : }
     998              : 
     999              : /* Remove any invalid expressions from the hash table
    1000              :    that refer to any of the registers contained in expression X.
    1001              : 
    1002              :    Make sure that newly inserted references to those registers
    1003              :    as subexpressions will be considered valid.
    1004              : 
    1005              :    mention_regs is not called when a register itself
    1006              :    is being stored in the table.
    1007              : 
    1008              :    Return true if we have done something that may have changed
    1009              :    the hash code of X.  */
    1010              : 
    1011              : static bool
    1012    478103214 : mention_regs (rtx x)
    1013              : {
    1014    478103214 :   enum rtx_code code;
    1015    478103214 :   int i, j;
    1016    478103214 :   const char *fmt;
    1017    478103214 :   bool changed = false;
    1018              : 
    1019    478103214 :   if (x == 0)
    1020              :     return false;
    1021              : 
    1022    478103214 :   code = GET_CODE (x);
    1023    478103214 :   if (code == REG)
    1024              :     {
    1025    143855556 :       unsigned int regno = REGNO (x);
    1026    143855556 :       unsigned int endregno = END_REGNO (x);
    1027    143855556 :       unsigned int i;
    1028              : 
    1029    287711112 :       for (i = regno; i < endregno; i++)
    1030              :         {
    1031    143855556 :           if (REG_IN_TABLE (i) >= 0 && REG_IN_TABLE (i) != REG_TICK (i))
    1032       169048 :             remove_invalid_refs (i);
    1033              : 
    1034    143855556 :           REG_IN_TABLE (i) = REG_TICK (i);
    1035    143855556 :           SUBREG_TICKED (i) = -1;
    1036              :         }
    1037              : 
    1038              :       return false;
    1039              :     }
    1040              : 
    1041              :   /* If this is a SUBREG, we don't want to discard other SUBREGs of the same
    1042              :      pseudo if they don't use overlapping words.  We handle only pseudos
    1043              :      here for simplicity.  */
    1044      8050034 :   if (code == SUBREG && REG_P (SUBREG_REG (x))
    1045    342279404 :       && REGNO (SUBREG_REG (x)) >= FIRST_PSEUDO_REGISTER)
    1046              :     {
    1047      8031633 :       unsigned int i = REGNO (SUBREG_REG (x));
    1048              : 
    1049      8031633 :       if (REG_IN_TABLE (i) >= 0 && REG_IN_TABLE (i) != REG_TICK (i))
    1050              :         {
    1051              :           /* If REG_IN_TABLE (i) differs from REG_TICK (i) by one, and
    1052              :              the last store to this register really stored into this
    1053              :              subreg, then remove the memory of this subreg.
    1054              :              Otherwise, remove any memory of the entire register and
    1055              :              all its subregs from the table.  */
    1056       343978 :           if (REG_TICK (i) - REG_IN_TABLE (i) > 1
    1057       343978 :               || SUBREG_TICKED (i) != REGNO (SUBREG_REG (x)))
    1058       343978 :             remove_invalid_refs (i);
    1059              :           else
    1060            0 :             remove_invalid_subreg_refs (i, SUBREG_BYTE (x), GET_MODE (x));
    1061              :         }
    1062              : 
    1063      8031633 :       REG_IN_TABLE (i) = REG_TICK (i);
    1064      8031633 :       SUBREG_TICKED (i) = REGNO (SUBREG_REG (x));
    1065      8031633 :       return false;
    1066              :     }
    1067              : 
    1068              :   /* If X is a comparison or a COMPARE and either operand is a register
    1069              :      that does not have a quantity, give it one.  This is so that a later
    1070              :      call to record_jump_equiv won't cause X to be assigned a different
    1071              :      hash code and not found in the table after that call.
    1072              : 
    1073              :      It is not necessary to do this here, since rehash_using_reg can
    1074              :      fix up the table later, but doing this here eliminates the need to
    1075              :      call that expensive function in the most common case where the only
    1076              :      use of the register is in the comparison.  */
    1077              : 
    1078    326216025 :   if (code == COMPARE || COMPARISON_P (x))
    1079              :     {
    1080     24029328 :       if (REG_P (XEXP (x, 0))
    1081     24029328 :           && ! REGNO_QTY_VALID_P (REGNO (XEXP (x, 0))))
    1082      9021754 :         if (insert_regs (XEXP (x, 0), NULL, false))
    1083              :           {
    1084      9021754 :             rehash_using_reg (XEXP (x, 0));
    1085      9021754 :             changed = true;
    1086              :           }
    1087              : 
    1088     24029328 :       if (REG_P (XEXP (x, 1))
    1089     24029328 :           && ! REGNO_QTY_VALID_P (REGNO (XEXP (x, 1))))
    1090      2465739 :         if (insert_regs (XEXP (x, 1), NULL, false))
    1091              :           {
    1092      2465739 :             rehash_using_reg (XEXP (x, 1));
    1093      2465739 :             changed = true;
    1094              :           }
    1095              :     }
    1096              : 
    1097    326216025 :   fmt = GET_RTX_FORMAT (code);
    1098    847416986 :   for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
    1099    521200961 :     if (fmt[i] == 'e')
    1100              :       {
    1101    307284594 :         if (mention_regs (XEXP (x, i)))
    1102    521200961 :           changed = true;
    1103              :       }
    1104    213916367 :     else if (fmt[i] == 'E')
    1105     19001611 :       for (j = 0; j < XVECLEN (x, i); j++)
    1106     14422857 :         if (mention_regs (XVECEXP (x, i, j)))
    1107       359174 :           changed = true;
    1108              : 
    1109              :   return changed;
    1110              : }
    1111              : 
    1112              : /* Update the register quantities for inserting X into the hash table
    1113              :    with a value equivalent to CLASSP.
    1114              :    (If the class does not contain a REG, it is irrelevant.)
    1115              :    If MODIFIED is true, X is a destination; it is being modified.
    1116              :    Note that delete_reg_equiv should be called on a register
    1117              :    before insert_regs is done on that register with MODIFIED != 0.
    1118              : 
    1119              :    True value means that elements of reg_qty have changed
    1120              :    so X's hash code may be different.  */
    1121              : 
    1122              : static bool
    1123    257546490 : insert_regs (rtx x, struct table_elt *classp, bool modified)
    1124              : {
    1125    257546490 :   if (REG_P (x))
    1126              :     {
    1127    125282915 :       unsigned int regno = REGNO (x);
    1128    125282915 :       int qty_valid;
    1129              : 
    1130              :       /* If REGNO is in the equivalence table already but is of the
    1131              :          wrong mode for that equivalence, don't do anything here.  */
    1132              : 
    1133    125282915 :       qty_valid = REGNO_QTY_VALID_P (regno);
    1134    125282915 :       if (qty_valid)
    1135              :         {
    1136      6578824 :           struct qty_table_elem *ent = &qty_table[REG_QTY (regno)];
    1137              : 
    1138      6578824 :           if (ent->mode != GET_MODE (x))
    1139              :             return false;
    1140              :         }
    1141              : 
    1142    125282915 :       if (modified || ! qty_valid)
    1143              :         {
    1144    118704091 :           if (classp)
    1145     96008816 :             for (classp = classp->first_same_value;
    1146    189398670 :                  classp != 0;
    1147     93389854 :                  classp = classp->next_same_value)
    1148    105308815 :               if (REG_P (classp->exp)
    1149     11918961 :                   && GET_MODE (classp->exp) == GET_MODE (x))
    1150              :                 {
    1151     11918961 :                   unsigned c_regno = REGNO (classp->exp);
    1152              : 
    1153     11918961 :                   gcc_assert (REGNO_QTY_VALID_P (c_regno));
    1154              : 
    1155              :                   /* Suppose that 5 is hard reg and 100 and 101 are
    1156              :                      pseudos.  Consider
    1157              : 
    1158              :                      (set (reg:si 100) (reg:si 5))
    1159              :                      (set (reg:si 5) (reg:si 100))
    1160              :                      (set (reg:di 101) (reg:di 5))
    1161              : 
    1162              :                      We would now set REG_QTY (101) = REG_QTY (5), but the
    1163              :                      entry for 5 is in SImode.  When we use this later in
    1164              :                      copy propagation, we get the register in wrong mode.  */
    1165     11918961 :                   if (qty_table[REG_QTY (c_regno)].mode != GET_MODE (x))
    1166            0 :                     continue;
    1167              : 
    1168     11918961 :                   make_regs_eqv (regno, c_regno);
    1169     11918961 :                   return true;
    1170              :                 }
    1171              : 
    1172              :           /* Mention_regs for a SUBREG checks if REG_TICK is exactly one larger
    1173              :              than REG_IN_TABLE to find out if there was only a single preceding
    1174              :              invalidation - for the SUBREG - or another one, which would be
    1175              :              for the full register.  However, if we find here that REG_TICK
    1176              :              indicates that the register is invalid, it means that it has
    1177              :              been invalidated in a separate operation.  The SUBREG might be used
    1178              :              now (then this is a recursive call), or we might use the full REG
    1179              :              now and a SUBREG of it later.  So bump up REG_TICK so that
    1180              :              mention_regs will do the right thing.  */
    1181    106785130 :           if (! modified
    1182     22912122 :               && REG_IN_TABLE (regno) >= 0
    1183    108935666 :               && REG_TICK (regno) == REG_IN_TABLE (regno) + 1)
    1184          498 :             REG_TICK (regno)++;
    1185    106785130 :           make_new_qty (regno, GET_MODE (x));
    1186    106785130 :           return true;
    1187              :         }
    1188              : 
    1189              :       return false;
    1190              :     }
    1191              : 
    1192              :   /* If X is a SUBREG, we will likely be inserting the inner register in the
    1193              :      table.  If that register doesn't have an assigned quantity number at
    1194              :      this point but does later, the insertion that we will be doing now will
    1195              :      not be accessible because its hash code will have changed.  So assign
    1196              :      a quantity number now.  */
    1197              : 
    1198      3513763 :   else if (GET_CODE (x) == SUBREG && REG_P (SUBREG_REG (x))
    1199    135761899 :            && ! REGNO_QTY_VALID_P (REGNO (SUBREG_REG (x))))
    1200              :     {
    1201      1690615 :       insert_regs (SUBREG_REG (x), NULL, false);
    1202      1690615 :       mention_regs (x);
    1203      1690615 :       return true;
    1204              :     }
    1205              :   else
    1206    130572960 :     return mention_regs (x);
    1207              : }
    1208              : 
    1209              : 
    1210              : /* Compute upper and lower anchors for CST.  Also compute the offset of CST
    1211              :    from these anchors/bases such that *_BASE + *_OFFS = CST.  Return false iff
    1212              :    CST is equal to an anchor.  */
    1213              : 
    1214              : static bool
    1215            0 : compute_const_anchors (rtx cst,
    1216              :                        HOST_WIDE_INT *lower_base, HOST_WIDE_INT *lower_offs,
    1217              :                        HOST_WIDE_INT *upper_base, HOST_WIDE_INT *upper_offs)
    1218              : {
    1219            0 :   unsigned HOST_WIDE_INT n = UINTVAL (cst);
    1220              : 
    1221            0 :   *lower_base = n & ~(targetm.const_anchor - 1);
    1222            0 :   if ((unsigned HOST_WIDE_INT) *lower_base == n)
    1223              :     return false;
    1224              : 
    1225            0 :   *upper_base = ((n + (targetm.const_anchor - 1))
    1226            0 :                  & ~(targetm.const_anchor - 1));
    1227            0 :   *upper_offs = n - *upper_base;
    1228            0 :   *lower_offs = n - *lower_base;
    1229            0 :   return true;
    1230              : }
    1231              : 
    1232              : /* Insert the equivalence between ANCHOR and (REG + OFF) in mode MODE.  */
    1233              : 
    1234              : static void
    1235            0 : insert_const_anchor (HOST_WIDE_INT anchor, rtx reg, HOST_WIDE_INT offs,
    1236              :                      machine_mode mode)
    1237              : {
    1238            0 :   struct table_elt *elt;
    1239            0 :   unsigned hash;
    1240            0 :   rtx anchor_exp;
    1241            0 :   rtx exp;
    1242              : 
    1243            0 :   anchor_exp = gen_int_mode (anchor, mode);
    1244            0 :   hash = HASH (anchor_exp, mode);
    1245            0 :   elt = lookup (anchor_exp, hash, mode);
    1246            0 :   if (!elt)
    1247            0 :     elt = insert (anchor_exp, NULL, hash, mode);
    1248              : 
    1249            0 :   exp = plus_constant (mode, reg, offs);
    1250              :   /* REG has just been inserted and the hash codes recomputed.  */
    1251            0 :   mention_regs (exp);
    1252            0 :   hash = HASH (exp, mode);
    1253              : 
    1254              :   /* Use the cost of the register rather than the whole expression.  When
    1255              :      looking up constant anchors we will further offset the corresponding
    1256              :      expression therefore it does not make sense to prefer REGs over
    1257              :      reg-immediate additions.  Prefer instead the oldest expression.  Also
    1258              :      don't prefer pseudos over hard regs so that we derive constants in
    1259              :      argument registers from other argument registers rather than from the
    1260              :      original pseudo that was used to synthesize the constant.  */
    1261            0 :   insert_with_costs (exp, elt, hash, mode, COST (reg, mode), 1);
    1262            0 : }
    1263              : 
    1264              : /* The constant CST is equivalent to the register REG.  Create
    1265              :    equivalences between the two anchors of CST and the corresponding
    1266              :    register-offset expressions using REG.  */
    1267              : 
    1268              : static void
    1269            0 : insert_const_anchors (rtx reg, rtx cst, machine_mode mode)
    1270              : {
    1271            0 :   HOST_WIDE_INT lower_base, lower_offs, upper_base, upper_offs;
    1272              : 
    1273            0 :   if (!compute_const_anchors (cst, &lower_base, &lower_offs,
    1274              :                               &upper_base, &upper_offs))
    1275            0 :       return;
    1276              : 
    1277              :   /* Ignore anchors of value 0.  Constants accessible from zero are
    1278              :      simple.  */
    1279            0 :   if (lower_base != 0)
    1280            0 :     insert_const_anchor (lower_base, reg, -lower_offs, mode);
    1281              : 
    1282            0 :   if (upper_base != 0)
    1283            0 :     insert_const_anchor (upper_base, reg, -upper_offs, mode);
    1284              : }
    1285              : 
    1286              : /* We need to express ANCHOR_ELT->exp + OFFS.  Walk the equivalence list of
    1287              :    ANCHOR_ELT and see if offsetting any of the entries by OFFS would create a
    1288              :    valid expression.  Return the cheapest and oldest of such expressions.  In
    1289              :    *OLD, return how old the resulting expression is compared to the other
    1290              :    equivalent expressions.  */
    1291              : 
    1292              : static rtx
    1293            0 : find_reg_offset_for_const (struct table_elt *anchor_elt, HOST_WIDE_INT offs,
    1294              :                            unsigned *old)
    1295              : {
    1296            0 :   struct table_elt *elt;
    1297            0 :   unsigned idx;
    1298            0 :   struct table_elt *match_elt;
    1299            0 :   rtx match;
    1300              : 
    1301              :   /* Find the cheapest and *oldest* expression to maximize the chance of
    1302              :      reusing the same pseudo.  */
    1303              : 
    1304            0 :   match_elt = NULL;
    1305            0 :   match = NULL_RTX;
    1306            0 :   for (elt = anchor_elt->first_same_value, idx = 0;
    1307            0 :        elt;
    1308            0 :        elt = elt->next_same_value, idx++)
    1309              :     {
    1310            0 :       if (match_elt && CHEAPER (match_elt, elt))
    1311              :         return match;
    1312              : 
    1313            0 :       if (REG_P (elt->exp)
    1314            0 :           || (GET_CODE (elt->exp) == PLUS
    1315            0 :               && REG_P (XEXP (elt->exp, 0))
    1316            0 :               && GET_CODE (XEXP (elt->exp, 1)) == CONST_INT))
    1317              :         {
    1318            0 :           rtx x;
    1319              : 
    1320              :           /* Ignore expressions that are no longer valid.  */
    1321            0 :           if (!REG_P (elt->exp) && !exp_equiv_p (elt->exp, elt->exp, 1, false))
    1322            0 :             continue;
    1323              : 
    1324            0 :           x = plus_constant (GET_MODE (elt->exp), elt->exp, offs);
    1325            0 :           if (REG_P (x)
    1326            0 :               || (GET_CODE (x) == PLUS
    1327            0 :                   && IN_RANGE (INTVAL (XEXP (x, 1)),
    1328              :                                -targetm.const_anchor,
    1329              :                                targetm.const_anchor - 1)))
    1330              :             {
    1331            0 :               match = x;
    1332            0 :               match_elt = elt;
    1333            0 :               *old = idx;
    1334              :             }
    1335              :         }
    1336              :     }
    1337              : 
    1338              :   return match;
    1339              : }
    1340              : 
    1341              : /* Try to express the constant SRC_CONST using a register+offset expression
    1342              :    derived from a constant anchor.  Return it if successful or NULL_RTX,
    1343              :    otherwise.  */
    1344              : 
    1345              : static rtx
    1346            0 : try_const_anchors (rtx src_const, machine_mode mode)
    1347              : {
    1348            0 :   struct table_elt *lower_elt, *upper_elt;
    1349            0 :   HOST_WIDE_INT lower_base, lower_offs, upper_base, upper_offs;
    1350            0 :   rtx lower_anchor_rtx, upper_anchor_rtx;
    1351            0 :   rtx lower_exp = NULL_RTX, upper_exp = NULL_RTX;
    1352            0 :   unsigned lower_old, upper_old;
    1353              : 
    1354              :   /* CONST_INT may be in various modes, avoid non-scalar-int mode. */
    1355            0 :   if (!SCALAR_INT_MODE_P (mode))
    1356              :     return NULL_RTX;
    1357              : 
    1358            0 :   if (!compute_const_anchors (src_const, &lower_base, &lower_offs,
    1359              :                               &upper_base, &upper_offs))
    1360              :     return NULL_RTX;
    1361              : 
    1362            0 :   lower_anchor_rtx = GEN_INT (lower_base);
    1363            0 :   upper_anchor_rtx = GEN_INT (upper_base);
    1364            0 :   lower_elt = lookup (lower_anchor_rtx, HASH (lower_anchor_rtx, mode), mode);
    1365            0 :   upper_elt = lookup (upper_anchor_rtx, HASH (upper_anchor_rtx, mode), mode);
    1366              : 
    1367            0 :   if (lower_elt)
    1368            0 :     lower_exp = find_reg_offset_for_const (lower_elt, lower_offs, &lower_old);
    1369            0 :   if (upper_elt)
    1370            0 :     upper_exp = find_reg_offset_for_const (upper_elt, upper_offs, &upper_old);
    1371              : 
    1372            0 :   if (!lower_exp)
    1373              :     return upper_exp;
    1374            0 :   if (!upper_exp)
    1375              :     return lower_exp;
    1376              : 
    1377              :   /* Return the older expression.  */
    1378            0 :   return (upper_old > lower_old ? upper_exp : lower_exp);
    1379              : }
    1380              : 
    1381              : /* Look in or update the hash table.  */
    1382              : 
    1383              : /* Remove table element ELT from use in the table.
    1384              :    HASH is its hash code, made using the HASH macro.
    1385              :    It's an argument because often that is known in advance
    1386              :    and we save much time not recomputing it.  */
    1387              : 
    1388              : static void
    1389     70202847 : remove_from_table (struct table_elt *elt, unsigned int hash)
    1390              : {
    1391     70202847 :   if (elt == 0)
    1392              :     return;
    1393              : 
    1394              :   /* Mark this element as removed.  See cse_insn.  */
    1395     70202847 :   elt->first_same_value = 0;
    1396              : 
    1397              :   /* Remove the table element from its equivalence class.  */
    1398              : 
    1399     70202847 :   {
    1400     70202847 :     struct table_elt *prev = elt->prev_same_value;
    1401     70202847 :     struct table_elt *next = elt->next_same_value;
    1402              : 
    1403     70202847 :     if (next)
    1404      7970230 :       next->prev_same_value = prev;
    1405              : 
    1406     70202847 :     if (prev)
    1407     45174361 :       prev->next_same_value = next;
    1408              :     else
    1409              :       {
    1410              :         struct table_elt *newfirst = next;
    1411     40546287 :         while (next)
    1412              :           {
    1413     15517801 :             next->first_same_value = newfirst;
    1414     15517801 :             next = next->next_same_value;
    1415              :           }
    1416              :       }
    1417              :   }
    1418              : 
    1419              :   /* Remove the table element from its hash bucket.  */
    1420              : 
    1421     70202847 :   {
    1422     70202847 :     struct table_elt *prev = elt->prev_same_hash;
    1423     70202847 :     struct table_elt *next = elt->next_same_hash;
    1424              : 
    1425     70202847 :     if (next)
    1426     20719407 :       next->prev_same_hash = prev;
    1427              : 
    1428     70202847 :     if (prev)
    1429      8927304 :       prev->next_same_hash = next;
    1430     61275543 :     else if (table[hash] == elt)
    1431     61275537 :       table[hash] = next;
    1432              :     else
    1433              :       {
    1434              :         /* This entry is not in the proper hash bucket.  This can happen
    1435              :            when two classes were merged by `merge_equiv_classes'.  Search
    1436              :            for the hash bucket that it heads.  This happens only very
    1437              :            rarely, so the cost is acceptable.  */
    1438          198 :         for (hash = 0; hash < HASH_SIZE; hash++)
    1439          192 :           if (table[hash] == elt)
    1440            6 :             table[hash] = next;
    1441              :       }
    1442              :   }
    1443              : 
    1444              :   /* Remove the table element from its related-value circular chain.  */
    1445              : 
    1446     70202847 :   if (elt->related_value != 0 && elt->related_value != elt)
    1447              :     {
    1448              :       struct table_elt *p = elt->related_value;
    1449              : 
    1450       264209 :       while (p->related_value != elt)
    1451              :         p = p->related_value;
    1452        33170 :       p->related_value = elt->related_value;
    1453        33170 :       if (p->related_value == p)
    1454        25795 :         p->related_value = 0;
    1455              :     }
    1456              : 
    1457              :   /* Now add it to the free element chain.  */
    1458     70202847 :   elt->next_same_hash = free_element_chain;
    1459     70202847 :   free_element_chain = elt;
    1460              : }
    1461              : 
    1462              : /* Same as above, but X is a pseudo-register.  */
    1463              : 
    1464              : static void
    1465     94233327 : remove_pseudo_from_table (rtx x, unsigned int hash)
    1466              : {
    1467     94233327 :   struct table_elt *elt;
    1468              : 
    1469              :   /* Because a pseudo-register can be referenced in more than one
    1470              :      mode, we might have to remove more than one table entry.  */
    1471     98770556 :   while ((elt = lookup_for_remove (x, hash, VOIDmode)))
    1472      4537229 :     remove_from_table (elt, hash);
    1473     94233327 : }
    1474              : 
    1475              : /* Look up X in the hash table and return its table element,
    1476              :    or 0 if X is not in the table.
    1477              : 
    1478              :    MODE is the machine-mode of X, or if X is an integer constant
    1479              :    with VOIDmode then MODE is the mode with which X will be used.
    1480              : 
    1481              :    Here we are satisfied to find an expression whose tree structure
    1482              :    looks like X.  */
    1483              : 
    1484              : static struct table_elt *
    1485    506451831 : lookup (rtx x, unsigned int hash, machine_mode mode)
    1486              : {
    1487    506451831 :   struct table_elt *p;
    1488              : 
    1489    757170281 :   for (p = table[hash]; p; p = p->next_same_hash)
    1490    383862610 :     if (mode == p->mode && ((x == p->exp && REG_P (x))
    1491    150768965 :                             || exp_equiv_p (x, p->exp, !REG_P (x), false)))
    1492              :       return p;
    1493              : 
    1494              :   return 0;
    1495              : }
    1496              : 
    1497              : /* Like `lookup' but don't care whether the table element uses invalid regs.
    1498              :    Also ignore discrepancies in the machine mode of a register.  */
    1499              : 
    1500              : static struct table_elt *
    1501     98770556 : lookup_for_remove (rtx x, unsigned int hash, machine_mode mode)
    1502              : {
    1503     98770556 :   struct table_elt *p;
    1504              : 
    1505     98770556 :   if (REG_P (x))
    1506              :     {
    1507     98770556 :       unsigned int regno = REGNO (x);
    1508              : 
    1509              :       /* Don't check the machine mode when comparing registers;
    1510              :          invalidating (REG:SI 0) also invalidates (REG:DF 0).  */
    1511    194847102 :       for (p = table[hash]; p; p = p->next_same_hash)
    1512    100613775 :         if (REG_P (p->exp)
    1513    100613775 :             && REGNO (p->exp) == regno)
    1514              :           return p;
    1515              :     }
    1516              :   else
    1517              :     {
    1518            0 :       for (p = table[hash]; p; p = p->next_same_hash)
    1519            0 :         if (mode == p->mode
    1520            0 :             && (x == p->exp || exp_equiv_p (x, p->exp, 0, false)))
    1521              :           return p;
    1522              :     }
    1523              : 
    1524              :   return 0;
    1525              : }
    1526              : 
    1527              : /* Look for an expression equivalent to X and with code CODE.
    1528              :    If one is found, return that expression.  */
    1529              : 
    1530              : static rtx
    1531     62818400 : lookup_as_function (rtx x, enum rtx_code code)
    1532              : {
    1533     62818400 :   struct table_elt *p
    1534     62818400 :     = lookup (x, SAFE_HASH (x, VOIDmode), GET_MODE (x));
    1535              : 
    1536     62818400 :   if (p == 0)
    1537              :     return 0;
    1538              : 
    1539     39541036 :   for (p = p->first_same_value; p; p = p->next_same_value)
    1540     27475753 :     if (GET_CODE (p->exp) == code
    1541              :         /* Make sure this is a valid entry in the table.  */
    1542     27475753 :         && exp_equiv_p (p->exp, p->exp, 1, false))
    1543       829957 :       return p->exp;
    1544              : 
    1545              :   return 0;
    1546              : }
    1547              : 
    1548              : /* Insert X in the hash table, assuming HASH is its hash code and
    1549              :    CLASSP is an element of the class it should go in (or 0 if a new
    1550              :    class should be made).  COST is the code of X and reg_cost is the
    1551              :    cost of registers in X.  It is inserted at the proper position to
    1552              :    keep the class in the order cheapest first.
    1553              : 
    1554              :    MODE is the machine-mode of X, or if X is an integer constant
    1555              :    with VOIDmode then MODE is the mode with which X will be used.
    1556              : 
    1557              :    For elements of equal cheapness, the most recent one
    1558              :    goes in front, except that the first element in the list
    1559              :    remains first unless a cheaper element is added.  The order of
    1560              :    pseudo-registers does not matter, as canon_reg will be called to
    1561              :    find the cheapest when a register is retrieved from the table.
    1562              : 
    1563              :    The in_memory field in the hash table element is set to 0.
    1564              :    The caller must set it nonzero if appropriate.
    1565              : 
    1566              :    You should call insert_regs (X, CLASSP, MODIFY) before calling here,
    1567              :    and if insert_regs returns a nonzero value
    1568              :    you must then recompute its hash code before calling here.
    1569              : 
    1570              :    If necessary, update table showing constant values of quantities.  */
    1571              : 
    1572              : static struct table_elt *
    1573    269030042 : insert_with_costs (rtx x, struct table_elt *classp, unsigned int hash,
    1574              :                    machine_mode mode, int cost, int reg_cost)
    1575              : {
    1576    269030042 :   struct table_elt *elt;
    1577              : 
    1578              :   /* If X is a register and we haven't made a quantity for it,
    1579              :      something is wrong.  */
    1580    269030042 :   gcc_assert (!REG_P (x) || REGNO_QTY_VALID_P (REGNO (x)));
    1581              : 
    1582              :   /* If X is a hard register, show it is being put in the table.  */
    1583    269030042 :   if (REG_P (x) && REGNO (x) < FIRST_PSEUDO_REGISTER)
    1584     22305117 :     add_to_hard_reg_set (&hard_regs_in_table, GET_MODE (x), REGNO (x));
    1585              : 
    1586              :   /* Put an element for X into the right hash bucket.  */
    1587              : 
    1588    269030042 :   elt = free_element_chain;
    1589    269030042 :   if (elt)
    1590    263733197 :     free_element_chain = elt->next_same_hash;
    1591              :   else
    1592      5296845 :     elt = XNEW (struct table_elt);
    1593              : 
    1594    269030042 :   elt->exp = x;
    1595    269030042 :   elt->canon_exp = NULL_RTX;
    1596    269030042 :   elt->cost = cost;
    1597    269030042 :   elt->regcost = reg_cost;
    1598    269030042 :   elt->next_same_value = 0;
    1599    269030042 :   elt->prev_same_value = 0;
    1600    269030042 :   elt->next_same_hash = table[hash];
    1601    269030042 :   elt->prev_same_hash = 0;
    1602    269030042 :   elt->related_value = 0;
    1603    269030042 :   elt->in_memory = 0;
    1604    269030042 :   elt->mode = mode;
    1605    269030042 :   elt->is_const = (CONSTANT_P (x) || fixed_base_plus_p (x));
    1606              : 
    1607    269030042 :   if (table[hash])
    1608     85232813 :     table[hash]->prev_same_hash = elt;
    1609    269030042 :   table[hash] = elt;
    1610              : 
    1611              :   /* Put it into the proper value-class.  */
    1612    269030042 :   if (classp)
    1613              :     {
    1614    131554944 :       classp = classp->first_same_value;
    1615    131554944 :       if (CHEAPER (elt, classp))
    1616              :         /* Insert at the head of the class.  */
    1617              :         {
    1618     61670047 :           struct table_elt *p;
    1619     61670047 :           elt->next_same_value = classp;
    1620     61670047 :           classp->prev_same_value = elt;
    1621     61670047 :           elt->first_same_value = elt;
    1622              : 
    1623    130312236 :           for (p = classp; p; p = p->next_same_value)
    1624     68642189 :             p->first_same_value = elt;
    1625              :         }
    1626              :       else
    1627              :         {
    1628              :           /* Insert not at head of the class.  */
    1629              :           /* Put it after the last element cheaper than X.  */
    1630              :           struct table_elt *p, *next;
    1631              : 
    1632              :           for (p = classp;
    1633    152749970 :                (next = p->next_same_value) && CHEAPER (next, elt);
    1634              :                p = next)
    1635              :             ;
    1636              : 
    1637              :           /* Put it after P and before NEXT.  */
    1638     69884897 :           elt->next_same_value = next;
    1639     69884897 :           if (next)
    1640     17137724 :             next->prev_same_value = elt;
    1641              : 
    1642     69884897 :           elt->prev_same_value = p;
    1643     69884897 :           p->next_same_value = elt;
    1644     69884897 :           elt->first_same_value = classp;
    1645              :         }
    1646              :     }
    1647              :   else
    1648    137475098 :     elt->first_same_value = elt;
    1649              : 
    1650              :   /* If this is a constant being set equivalent to a register or a register
    1651              :      being set equivalent to a constant, note the constant equivalence.
    1652              : 
    1653              :      If this is a constant, it cannot be equivalent to a different constant,
    1654              :      and a constant is the only thing that can be cheaper than a register.  So
    1655              :      we know the register is the head of the class (before the constant was
    1656              :      inserted).
    1657              : 
    1658              :      If this is a register that is not already known equivalent to a
    1659              :      constant, we must check the entire class.
    1660              : 
    1661              :      If this is a register that is already known equivalent to an insn,
    1662              :      update the qtys `const_insn' to show that `this_insn' is the latest
    1663              :      insn making that quantity equivalent to the constant.  */
    1664              : 
    1665    269030042 :   if (elt->is_const && classp && REG_P (classp->exp)
    1666      3373416 :       && !REG_P (x))
    1667              :     {
    1668      3370595 :       int exp_q = REG_QTY (REGNO (classp->exp));
    1669      3370595 :       struct qty_table_elem *exp_ent = &qty_table[exp_q];
    1670              : 
    1671      3370595 :       exp_ent->const_rtx = gen_lowpart (exp_ent->mode, x);
    1672      3370595 :       exp_ent->const_insn = this_insn;
    1673      3370595 :     }
    1674              : 
    1675    265659447 :   else if (REG_P (x)
    1676    112104807 :            && classp
    1677     96018186 :            && ! qty_table[REG_QTY (REGNO (x))].const_rtx
    1678    356980643 :            && ! elt->is_const)
    1679              :     {
    1680              :       struct table_elt *p;
    1681              : 
    1682    212123355 :       for (p = classp; p != 0; p = p->next_same_value)
    1683              :         {
    1684    135421064 :           if (p->is_const && !REG_P (p->exp))
    1685              :             {
    1686     14616023 :               int x_q = REG_QTY (REGNO (x));
    1687     14616023 :               struct qty_table_elem *x_ent = &qty_table[x_q];
    1688              : 
    1689     14616023 :               x_ent->const_rtx
    1690     14616023 :                 = gen_lowpart (GET_MODE (x), p->exp);
    1691     14616023 :               x_ent->const_insn = this_insn;
    1692     14616023 :               break;
    1693              :             }
    1694              :         }
    1695              :     }
    1696              : 
    1697    174341133 :   else if (REG_P (x)
    1698     20786493 :            && qty_table[REG_QTY (REGNO (x))].const_rtx
    1699    179038124 :            && GET_MODE (x) == qty_table[REG_QTY (REGNO (x))].mode)
    1700      4696991 :     qty_table[REG_QTY (REGNO (x))].const_insn = this_insn;
    1701              : 
    1702              :   /* If this is a constant with symbolic value,
    1703              :      and it has a term with an explicit integer value,
    1704              :      link it up with related expressions.  */
    1705    269030042 :   if (GET_CODE (x) == CONST)
    1706              :     {
    1707       875097 :       rtx subexp = get_related_value (x);
    1708       875097 :       unsigned subhash;
    1709       875097 :       struct table_elt *subelt, *subelt_prev;
    1710              : 
    1711       875097 :       if (subexp != 0)
    1712              :         {
    1713              :           /* Get the integer-free subexpression in the hash table.  */
    1714       861262 :           subhash = SAFE_HASH (subexp, mode);
    1715       861262 :           subelt = lookup (subexp, subhash, mode);
    1716       861262 :           if (subelt == 0)
    1717       388088 :             subelt = insert (subexp, NULL, subhash, mode);
    1718              :           /* Initialize SUBELT's circular chain if it has none.  */
    1719       861262 :           if (subelt->related_value == 0)
    1720       563253 :             subelt->related_value = subelt;
    1721              :           /* Find the element in the circular chain that precedes SUBELT.  */
    1722       861262 :           subelt_prev = subelt;
    1723      3563004 :           while (subelt_prev->related_value != subelt)
    1724              :             subelt_prev = subelt_prev->related_value;
    1725              :           /* Put new ELT into SUBELT's circular chain just before SUBELT.
    1726              :              This way the element that follows SUBELT is the oldest one.  */
    1727       861262 :           elt->related_value = subelt_prev->related_value;
    1728       861262 :           subelt_prev->related_value = elt;
    1729              :         }
    1730              :     }
    1731              : 
    1732    269030042 :   return elt;
    1733              : }
    1734              : 
    1735              : /* Wrap insert_with_costs by passing the default costs.  */
    1736              : 
    1737              : static struct table_elt *
    1738    269030042 : insert (rtx x, struct table_elt *classp, unsigned int hash,
    1739              :         machine_mode mode)
    1740              : {
    1741    538060084 :   return insert_with_costs (x, classp, hash, mode,
    1742    269030042 :                             COST (x, mode), approx_reg_cost (x));
    1743              : }
    1744              : 
    1745              : 
    1746              : /* Given two equivalence classes, CLASS1 and CLASS2, put all the entries from
    1747              :    CLASS2 into CLASS1.  This is done when we have reached an insn which makes
    1748              :    the two classes equivalent.
    1749              : 
    1750              :    CLASS1 will be the surviving class; CLASS2 should not be used after this
    1751              :    call.
    1752              : 
    1753              :    Any invalid entries in CLASS2 will not be copied.  */
    1754              : 
    1755              : static void
    1756      5356739 : merge_equiv_classes (struct table_elt *class1, struct table_elt *class2)
    1757              : {
    1758      5356739 :   struct table_elt *elt, *next, *new_elt;
    1759              : 
    1760              :   /* Ensure we start with the head of the classes.  */
    1761      5356739 :   class1 = class1->first_same_value;
    1762      5356739 :   class2 = class2->first_same_value;
    1763              : 
    1764              :   /* If they were already equal, forget it.  */
    1765      5356739 :   if (class1 == class2)
    1766              :     return;
    1767              : 
    1768     12763969 :   for (elt = class2; elt; elt = next)
    1769              :     {
    1770      7407230 :       unsigned int hash;
    1771      7407230 :       rtx exp = elt->exp;
    1772      7407230 :       machine_mode mode = elt->mode;
    1773              : 
    1774      7407230 :       next = elt->next_same_value;
    1775              : 
    1776              :       /* Remove old entry, make a new one in CLASS1's class.
    1777              :          Don't do this for invalid entries as we cannot find their
    1778              :          hash code (it also isn't necessary).  */
    1779      7407230 :       if (REG_P (exp) || exp_equiv_p (exp, exp, 1, false))
    1780              :         {
    1781      7407206 :           bool need_rehash = false;
    1782              : 
    1783      7407206 :           hash_arg_in_memory = 0;
    1784      7407206 :           hash = HASH (exp, mode);
    1785              : 
    1786      7407206 :           if (REG_P (exp))
    1787              :             {
    1788      2013456 :               need_rehash = REGNO_QTY_VALID_P (REGNO (exp));
    1789      2013456 :               delete_reg_equiv (REGNO (exp));
    1790              :             }
    1791              : 
    1792      7407206 :           if (REG_P (exp) && REGNO (exp) >= FIRST_PSEUDO_REGISTER)
    1793      2012284 :             remove_pseudo_from_table (exp, hash);
    1794              :           else
    1795      5394922 :             remove_from_table (elt, hash);
    1796              : 
    1797      7407206 :           if (insert_regs (exp, class1, false) || need_rehash)
    1798              :             {
    1799      2013456 :               rehash_using_reg (exp);
    1800      2013456 :               hash = HASH (exp, mode);
    1801              :             }
    1802      7407206 :           new_elt = insert (exp, class1, hash, mode);
    1803      7407206 :           new_elt->in_memory = hash_arg_in_memory;
    1804      7407206 :           if (GET_CODE (exp) == ASM_OPERANDS && elt->cost == MAX_COST)
    1805            0 :             new_elt->cost = MAX_COST;
    1806              :         }
    1807              :     }
    1808              : }
    1809              : 
    1810              : /* Flush the entire hash table.  */
    1811              : 
    1812              : static void
    1813         8314 : flush_hash_table (void)
    1814              : {
    1815         8314 :   int i;
    1816         8314 :   struct table_elt *p;
    1817              : 
    1818       274362 :   for (i = 0; i < HASH_SIZE; i++)
    1819      1131611 :     for (p = table[i]; p; p = table[i])
    1820              :       {
    1821              :         /* Note that invalidate can remove elements
    1822              :            after P in the current hash chain.  */
    1823       865563 :         if (REG_P (p->exp))
    1824       411606 :           invalidate (p->exp, VOIDmode);
    1825              :         else
    1826       453957 :           remove_from_table (p, i);
    1827              :       }
    1828         8314 : }
    1829              : 
    1830              : /* Check whether an anti dependence exists between X and EXP.  MODE and
    1831              :    ADDR are as for canon_anti_dependence.  */
    1832              : 
    1833              : static bool
    1834    191548978 : check_dependence (const_rtx x, rtx exp, machine_mode mode, rtx addr)
    1835              : {
    1836    191548978 :   subrtx_iterator::array_type array;
    1837    900755276 :   FOR_EACH_SUBRTX (iter, array, x, NONCONST)
    1838              :     {
    1839    717644820 :       const_rtx x = *iter;
    1840    717644820 :       if (MEM_P (x) && canon_anti_dependence (x, true, exp, mode, addr))
    1841      8438522 :         return true;
    1842              :     }
    1843    183110456 :   return false;
    1844    191548978 : }
    1845              : 
    1846              : /* Remove from the hash table, or mark as invalid, all expressions whose
    1847              :    values could be altered by storing in register X.  */
    1848              : 
    1849              : static void
    1850    224784674 : invalidate_reg (rtx x)
    1851              : {
    1852    224784674 :   gcc_assert (GET_CODE (x) == REG);
    1853              : 
    1854              :   /* If X is a register, dependencies on its contents are recorded
    1855              :      through the qty number mechanism.  Just change the qty number of
    1856              :      the register, mark it as invalid for expressions that refer to it,
    1857              :      and remove it itself.  */
    1858    224784674 :   unsigned int regno = REGNO (x);
    1859    224784674 :   unsigned int hash = HASH (x, GET_MODE (x));
    1860              : 
    1861              :   /* Remove REGNO from any quantity list it might be on and indicate
    1862              :      that its value might have changed.  If it is a pseudo, remove its
    1863              :      entry from the hash table.
    1864              : 
    1865              :      For a hard register, we do the first two actions above for any
    1866              :      additional hard registers corresponding to X.  Then, if any of these
    1867              :      registers are in the table, we must remove any REG entries that
    1868              :      overlap these registers.  */
    1869              : 
    1870    224784674 :   delete_reg_equiv (regno);
    1871    224784674 :   REG_TICK (regno)++;
    1872    224784674 :   SUBREG_TICKED (regno) = -1;
    1873              : 
    1874    224784674 :   if (regno >= FIRST_PSEUDO_REGISTER)
    1875     92221043 :     remove_pseudo_from_table (x, hash);
    1876              :   else
    1877              :     {
    1878    132563631 :       HOST_WIDE_INT in_table = TEST_HARD_REG_BIT (hard_regs_in_table, regno);
    1879    132563631 :       unsigned int endregno = END_REGNO (x);
    1880    132563631 :       unsigned int rn;
    1881    132563631 :       struct table_elt *p, *next;
    1882              : 
    1883    132563631 :       CLEAR_HARD_REG_BIT (hard_regs_in_table, regno);
    1884              : 
    1885    133166383 :       for (rn = regno + 1; rn < endregno; rn++)
    1886              :         {
    1887       602752 :           in_table |= TEST_HARD_REG_BIT (hard_regs_in_table, rn);
    1888       602752 :           CLEAR_HARD_REG_BIT (hard_regs_in_table, rn);
    1889       602752 :           delete_reg_equiv (rn);
    1890       602752 :           REG_TICK (rn)++;
    1891       602752 :           SUBREG_TICKED (rn) = -1;
    1892              :         }
    1893              : 
    1894    132563631 :       if (in_table)
    1895    414200127 :         for (hash = 0; hash < HASH_SIZE; hash++)
    1896    645566010 :           for (p = table[hash]; p; p = next)
    1897              :             {
    1898    243917402 :               next = p->next_same_hash;
    1899              : 
    1900    243917402 :               if (!REG_P (p->exp) || REGNO (p->exp) >= FIRST_PSEUDO_REGISTER)
    1901    232999869 :                 continue;
    1902              : 
    1903     10917533 :               unsigned int tregno = REGNO (p->exp);
    1904     10917533 :               unsigned int tendregno = END_REGNO (p->exp);
    1905     10917533 :               if (tendregno > regno && tregno < endregno)
    1906     10843600 :                 remove_from_table (p, hash);
    1907              :             }
    1908              :     }
    1909    224784674 : }
    1910              : 
    1911              : /* Remove from the hash table, or mark as invalid, all expressions whose
    1912              :    values could be altered by storing in X.  X is a register, a subreg, or
    1913              :    a memory reference with nonvarying address (because, when a memory
    1914              :    reference with a varying address is stored in, all memory references are
    1915              :    removed by invalidate_memory so specific invalidation is superfluous).
    1916              :    FULL_MODE, if not VOIDmode, indicates that this much should be
    1917              :    invalidated instead of just the amount indicated by the mode of X.  This
    1918              :    is only used for bitfield stores into memory.
    1919              : 
    1920              :    A nonvarying address may be just a register or just a symbol reference,
    1921              :    or it may be either of those plus a numeric offset.  */
    1922              : 
    1923              : static void
    1924    254230458 : invalidate (rtx x, machine_mode full_mode)
    1925              : {
    1926    255930472 :   int i;
    1927    255930472 :   struct table_elt *p;
    1928    255930472 :   rtx addr;
    1929              : 
    1930    255930472 :   switch (GET_CODE (x))
    1931              :     {
    1932    224784484 :     case REG:
    1933    224784484 :       invalidate_reg (x);
    1934    224784484 :       return;
    1935              : 
    1936      1654303 :     case SUBREG:
    1937      1654303 :       invalidate (SUBREG_REG (x), VOIDmode);
    1938      1654303 :       return;
    1939              : 
    1940        26392 :     case PARALLEL:
    1941        72103 :       for (i = XVECLEN (x, 0) - 1; i >= 0; --i)
    1942        45711 :         invalidate (XVECEXP (x, 0, i), VOIDmode);
    1943              :       return;
    1944              : 
    1945        45711 :     case EXPR_LIST:
    1946              :       /* This is part of a disjoint return value; extract the location in
    1947              :          question ignoring the offset.  */
    1948        45711 :       invalidate (XEXP (x, 0), VOIDmode);
    1949        45711 :       return;
    1950              : 
    1951     29419582 :     case MEM:
    1952     29419582 :       addr = canon_rtx (get_addr (XEXP (x, 0)));
    1953              :       /* Calculate the canonical version of X here so that
    1954              :          true_dependence doesn't generate new RTL for X on each call.  */
    1955     29419582 :       x = canon_rtx (x);
    1956              : 
    1957              :       /* Remove all hash table elements that refer to overlapping pieces of
    1958              :          memory.  */
    1959     29419582 :       if (full_mode == VOIDmode)
    1960     29418614 :         full_mode = GET_MODE (x);
    1961              : 
    1962    970846206 :       for (i = 0; i < HASH_SIZE; i++)
    1963              :         {
    1964    941426624 :           struct table_elt *next;
    1965              : 
    1966   1929770578 :           for (p = table[i]; p; p = next)
    1967              :             {
    1968    988343954 :               next = p->next_same_hash;
    1969    988343954 :               if (p->in_memory)
    1970              :                 {
    1971              :                   /* Just canonicalize the expression once;
    1972              :                      otherwise each time we call invalidate
    1973              :                      true_dependence will canonicalize the
    1974              :                      expression again.  */
    1975    191548978 :                   if (!p->canon_exp)
    1976     27607049 :                     p->canon_exp = canon_rtx (p->exp);
    1977    191548978 :                   if (check_dependence (p->canon_exp, x, full_mode, addr))
    1978      8438522 :                     remove_from_table (p, i);
    1979              :                 }
    1980              :             }
    1981              :         }
    1982              :       return;
    1983              : 
    1984            0 :     default:
    1985            0 :       gcc_unreachable ();
    1986              :     }
    1987              : }
    1988              : 
    1989              : /* Invalidate DEST.  Used when DEST is not going to be added
    1990              :    into the hash table for some reason, e.g. do_not_record
    1991              :    flagged on it.  */
    1992              : 
    1993              : static void
    1994     56448428 : invalidate_dest (rtx dest)
    1995              : {
    1996     56448428 :   if (REG_P (dest)
    1997     27166985 :       || GET_CODE (dest) == SUBREG
    1998     27166985 :       || MEM_P (dest))
    1999     36075419 :     invalidate (dest, VOIDmode);
    2000     20373009 :   else if (GET_CODE (dest) == STRICT_LOW_PART
    2001     20373009 :            || GET_CODE (dest) == ZERO_EXTRACT)
    2002          976 :     invalidate (XEXP (dest, 0), GET_MODE (dest));
    2003     56448428 : }
    2004              : 
    2005              : /* Remove all expressions that refer to register REGNO,
    2006              :    since they are already invalid, and we are about to
    2007              :    mark that register valid again and don't want the old
    2008              :    expressions to reappear as valid.  */
    2009              : 
    2010              : static void
    2011     13530073 : remove_invalid_refs (unsigned int regno)
    2012              : {
    2013     13530073 :   unsigned int i;
    2014     13530073 :   struct table_elt *p, *next;
    2015              : 
    2016    446492409 :   for (i = 0; i < HASH_SIZE; i++)
    2017    687235195 :     for (p = table[i]; p; p = next)
    2018              :       {
    2019    254272859 :         next = p->next_same_hash;
    2020    254272859 :         if (!REG_P (p->exp) && refers_to_regno_p (regno, p->exp))
    2021     17575222 :           remove_from_table (p, i);
    2022              :       }
    2023     13530073 : }
    2024              : 
    2025              : /* Likewise for a subreg with subreg_reg REGNO, subreg_byte OFFSET,
    2026              :    and mode MODE.  */
    2027              : static void
    2028            0 : remove_invalid_subreg_refs (unsigned int regno, poly_uint64 offset,
    2029              :                             machine_mode mode)
    2030              : {
    2031            0 :   unsigned int i;
    2032            0 :   struct table_elt *p, *next;
    2033              : 
    2034            0 :   for (i = 0; i < HASH_SIZE; i++)
    2035            0 :     for (p = table[i]; p; p = next)
    2036              :       {
    2037            0 :         rtx exp = p->exp;
    2038            0 :         next = p->next_same_hash;
    2039              : 
    2040            0 :         if (!REG_P (exp)
    2041            0 :             && (GET_CODE (exp) != SUBREG
    2042            0 :                 || !REG_P (SUBREG_REG (exp))
    2043            0 :                 || REGNO (SUBREG_REG (exp)) != regno
    2044            0 :                 || ranges_maybe_overlap_p (SUBREG_BYTE (exp),
    2045            0 :                                            GET_MODE_SIZE (GET_MODE (exp)),
    2046            0 :                                            offset, GET_MODE_SIZE (mode)))
    2047            0 :             && refers_to_regno_p (regno, p->exp))
    2048            0 :           remove_from_table (p, i);
    2049              :       }
    2050            0 : }
    2051              : 
    2052              : /* Recompute the hash codes of any valid entries in the hash table that
    2053              :    reference X, if X is a register, or SUBREG_REG (X) if X is a SUBREG.
    2054              : 
    2055              :    This is called when we make a jump equivalence.  */
    2056              : 
    2057              : static void
    2058    128505747 : rehash_using_reg (rtx x)
    2059              : {
    2060    128505747 :   unsigned int i;
    2061    128505747 :   struct table_elt *p, *next;
    2062    128505747 :   unsigned hash;
    2063              : 
    2064    128505747 :   if (GET_CODE (x) == SUBREG)
    2065      1690615 :     x = SUBREG_REG (x);
    2066              : 
    2067              :   /* If X is not a register or if the register is known not to be in any
    2068              :      valid entries in the table, we have no work to do.  */
    2069              : 
    2070    128505747 :   if (!REG_P (x)
    2071    118704091 :       || REG_IN_TABLE (REGNO (x)) < 0
    2072    133594720 :       || REG_IN_TABLE (REGNO (x)) != REG_TICK (REGNO (x)))
    2073              :     return;
    2074              : 
    2075              :   /* Scan all hash chains looking for valid entries that mention X.
    2076              :      If we find one and it is in the wrong hash chain, move it.  */
    2077              : 
    2078    167917332 :   for (i = 0; i < HASH_SIZE; i++)
    2079    277215784 :     for (p = table[i]; p; p = next)
    2080              :       {
    2081    114386856 :         next = p->next_same_hash;
    2082    114386856 :         if (reg_mentioned_p (x, p->exp)
    2083      4900862 :             && exp_equiv_p (p->exp, p->exp, 1, false)
    2084    119287533 :             && i != (hash = SAFE_HASH (p->exp, p->mode)))
    2085              :           {
    2086      3484480 :             if (p->next_same_hash)
    2087      1117104 :               p->next_same_hash->prev_same_hash = p->prev_same_hash;
    2088              : 
    2089      3484480 :             if (p->prev_same_hash)
    2090       640443 :               p->prev_same_hash->next_same_hash = p->next_same_hash;
    2091              :             else
    2092      2844037 :               table[i] = p->next_same_hash;
    2093              : 
    2094      3484480 :             p->next_same_hash = table[hash];
    2095      3484480 :             p->prev_same_hash = 0;
    2096      3484480 :             if (table[hash])
    2097      1682205 :               table[hash]->prev_same_hash = p;
    2098      3484480 :             table[hash] = p;
    2099              :           }
    2100              :       }
    2101              : }
    2102              : 
    2103              : /* Remove from the hash table any expression that is a call-clobbered
    2104              :    register in INSN.  Also update their TICK values.  */
    2105              : 
    2106              : static void
    2107     15857452 : invalidate_for_call (rtx_call_insn *insn)
    2108              : {
    2109     15857452 :   unsigned int regno;
    2110     15857452 :   unsigned hash;
    2111     15857452 :   struct table_elt *p, *next;
    2112     15857452 :   int in_table = 0;
    2113     15857452 :   hard_reg_set_iterator hrsi;
    2114              : 
    2115              :   /* Go through all the hard registers.  For each that might be clobbered
    2116              :      in call insn INSN, remove the register from quantity chains and update
    2117              :      reg_tick if defined.  Also see if any of these registers is currently
    2118              :      in the table.
    2119              : 
    2120              :      ??? We could be more precise for partially-clobbered registers,
    2121              :      and only invalidate values that actually occupy the clobbered part
    2122              :      of the registers.  It doesn't seem worth the effort though, since
    2123              :      we shouldn't see this situation much before RA.  Whatever choice
    2124              :      we make here has to be consistent with the table walk below,
    2125              :      so any change to this test will require a change there too.  */
    2126     15857452 :   HARD_REG_SET callee_clobbers
    2127     15857452 :     = insn_callee_abi (insn).full_and_partial_reg_clobbers ();
    2128   1344536830 :   EXECUTE_IF_SET_IN_HARD_REG_SET (callee_clobbers, 0, regno, hrsi)
    2129              :     {
    2130   1328679378 :       delete_reg_equiv (regno);
    2131   1328679378 :       if (REG_TICK (regno) >= 0)
    2132              :         {
    2133   1328679378 :           REG_TICK (regno)++;
    2134   1328679378 :           SUBREG_TICKED (regno) = -1;
    2135              :         }
    2136   1328679378 :       in_table |= (TEST_HARD_REG_BIT (hard_regs_in_table, regno) != 0);
    2137              :     }
    2138              : 
    2139              :   /* In the case where we have no call-clobbered hard registers in the
    2140              :      table, we are done.  Otherwise, scan the table and remove any
    2141              :      entry that overlaps a call-clobbered register.  */
    2142              : 
    2143     15857452 :   if (in_table)
    2144    120706014 :     for (hash = 0; hash < HASH_SIZE; hash++)
    2145    175134271 :       for (p = table[hash]; p; p = next)
    2146              :         {
    2147     58086015 :           next = p->next_same_hash;
    2148              : 
    2149    113049883 :           if (!REG_P (p->exp)
    2150     58086015 :               || REGNO (p->exp) >= FIRST_PSEUDO_REGISTER)
    2151     54963868 :             continue;
    2152              : 
    2153              :           /* This must use the same test as above rather than the
    2154              :              more accurate clobbers_reg_p.  */
    2155      3122147 :           if (overlaps_hard_reg_set_p (callee_clobbers, GET_MODE (p->exp),
    2156      3122147 :                                        REGNO (p->exp)))
    2157      3101170 :             remove_from_table (p, hash);
    2158              :         }
    2159     15857452 : }
    2160              : 
    2161              : /* Given an expression X of type CONST,
    2162              :    and ELT which is its table entry (or 0 if it
    2163              :    is not in the hash table),
    2164              :    return an alternate expression for X as a register plus integer.
    2165              :    If none can be found, return 0.  */
    2166              : 
    2167              : static rtx
    2168       755133 : use_related_value (rtx x, struct table_elt *elt)
    2169              : {
    2170       755133 :   struct table_elt *relt = 0;
    2171       755133 :   struct table_elt *p, *q;
    2172       755133 :   HOST_WIDE_INT offset;
    2173              : 
    2174              :   /* First, is there anything related known?
    2175              :      If we have a table element, we can tell from that.
    2176              :      Otherwise, must look it up.  */
    2177              : 
    2178       755133 :   if (elt != 0 && elt->related_value != 0)
    2179              :     relt = elt;
    2180       578142 :   else if (elt == 0 && GET_CODE (x) == CONST)
    2181              :     {
    2182       578142 :       rtx subexp = get_related_value (x);
    2183       578142 :       if (subexp != 0)
    2184       564307 :         relt = lookup (subexp,
    2185              :                        SAFE_HASH (subexp, GET_MODE (subexp)),
    2186       564307 :                        GET_MODE (subexp));
    2187              :     }
    2188              : 
    2189       695972 :   if (relt == 0)
    2190              :     return 0;
    2191              : 
    2192              :   /* Search all related table entries for one that has an
    2193              :      equivalent register.  */
    2194              : 
    2195              :   p = relt;
    2196       897599 :   while (1)
    2197              :     {
    2198              :       /* This loop is strange in that it is executed in two different cases.
    2199              :          The first is when X is already in the table.  Then it is searching
    2200              :          the RELATED_VALUE list of X's class (RELT).  The second case is when
    2201              :          X is not in the table.  Then RELT points to a class for the related
    2202              :          value.
    2203              : 
    2204              :          Ensure that, whatever case we are in, that we ignore classes that have
    2205              :          the same value as X.  */
    2206              : 
    2207       897599 :       if (rtx_equal_p (x, p->exp))
    2208              :         q = 0;
    2209              :       else
    2210      1757649 :         for (q = p->first_same_value; q; q = q->next_same_value)
    2211      1219156 :           if (REG_P (q->exp))
    2212              :             break;
    2213              : 
    2214       765934 :       if (q)
    2215              :         break;
    2216              : 
    2217       670158 :       p = p->related_value;
    2218              : 
    2219              :       /* We went all the way around, so there is nothing to be found.
    2220              :          Alternatively, perhaps RELT was in the table for some other reason
    2221              :          and it has no related values recorded.  */
    2222       670158 :       if (p == relt || p == 0)
    2223              :         break;
    2224              :     }
    2225              : 
    2226       337722 :   if (q == 0)
    2227              :     return 0;
    2228              : 
    2229       227441 :   offset = (get_integer_term (x) - get_integer_term (p->exp));
    2230              :   /* Note: OFFSET may be 0 if P->xexp and X are related by commutativity.  */
    2231       227441 :   return plus_constant (q->mode, q->exp, offset);
    2232              : }
    2233              : 
    2234              : 
    2235              : /* Hash a string.  Just add its bytes up.  */
    2236              : static inline unsigned
    2237       131531 : hash_rtx_string (const char *ps)
    2238              : {
    2239       131531 :   unsigned hash = 0;
    2240       131531 :   const unsigned char *p = (const unsigned char *) ps;
    2241              : 
    2242       131531 :   if (p)
    2243       746698 :     while (*p)
    2244       615167 :       hash += *p++;
    2245              : 
    2246       131531 :   return hash;
    2247              : }
    2248              : 
    2249              : /* Hash an rtx.  We are careful to make sure the value is never negative.
    2250              :    Equivalent registers hash identically.
    2251              :    MODE is used in hashing for CONST_INTs only;
    2252              :    otherwise the mode of X is used.
    2253              : 
    2254              :    Store 1 in DO_NOT_RECORD_P if any subexpression is volatile.
    2255              : 
    2256              :    If HASH_ARG_IN_MEMORY_P is not NULL, store 1 in it if X contains
    2257              :    a MEM rtx which does not have the MEM_READONLY_P flag set.
    2258              : 
    2259              :    Note that cse_insn knows that the hash code of a MEM expression
    2260              :    is just (int) MEM plus the hash code of the address.
    2261              : 
    2262              :    Call CB on each rtx if CB is not NULL.
    2263              :    When the callback returns true, we continue with the new rtx.  */
    2264              : 
    2265              : unsigned
    2266   1311081789 : hash_rtx (const_rtx x, machine_mode mode,
    2267              :           int *do_not_record_p, int *hash_arg_in_memory_p,
    2268              :           bool have_reg_qty, hash_rtx_callback_function cb)
    2269              : {
    2270   1311081789 :   int i, j;
    2271   1311081789 :   unsigned hash = 0;
    2272   1934269271 :   enum rtx_code code;
    2273   1934269271 :   const char *fmt;
    2274   1934269271 :   machine_mode newmode;
    2275   1934269271 :   rtx newx;
    2276              : 
    2277              :   /* Used to turn recursion into iteration.  We can't rely on GCC's
    2278              :      tail-recursion elimination since we need to keep accumulating values
    2279              :      in HASH.  */
    2280    623187482 :  repeat:
    2281   1934269271 :   if (x == 0)
    2282              :     return hash;
    2283              : 
    2284              :   /* Invoke the callback first.  */
    2285   1934269271 :   if (cb != NULL
    2286   1934269271 :       && ((*cb) (x, mode, &newx, &newmode)))
    2287              :     {
    2288            0 :       hash += hash_rtx (newx, newmode, do_not_record_p,
    2289              :                         hash_arg_in_memory_p, have_reg_qty, cb);
    2290            0 :       return hash;
    2291              :     }
    2292              : 
    2293   1934269271 :   code = GET_CODE (x);
    2294   1934269271 :   switch (code)
    2295              :     {
    2296    671880031 :     case REG:
    2297    671880031 :       {
    2298    671880031 :         unsigned int regno = REGNO (x);
    2299              : 
    2300    671880031 :         if (do_not_record_p && !reload_completed)
    2301              :           {
    2302              :             /* On some machines, we can't record any non-fixed hard register,
    2303              :                because extending its life will cause reload problems.  We
    2304              :                consider ap, fp, sp, gp to be fixed for this purpose.
    2305              : 
    2306              :                We also consider CCmode registers to be fixed for this purpose;
    2307              :                failure to do so leads to failure to simplify 0<100 type of
    2308              :                conditionals.
    2309              : 
    2310              :                On all machines, we can't record any global registers.
    2311              :                Nor should we record any register that is in a small
    2312              :                class, as defined by TARGET_CLASS_LIKELY_SPILLED_P.  */
    2313    668290101 :             bool record;
    2314              : 
    2315    668290101 :             if (regno >= FIRST_PSEUDO_REGISTER)
    2316              :               record = true;
    2317    444309740 :             else if (x == frame_pointer_rtx
    2318    309052551 :                      || x == hard_frame_pointer_rtx
    2319    308935391 :                      || x == arg_pointer_rtx
    2320    301181774 :                      || x == stack_pointer_rtx
    2321    265710968 :                      || x == pic_offset_table_rtx)
    2322              :               record = true;
    2323    265710968 :             else if (global_regs[regno])
    2324              :               record = false;
    2325    265710591 :             else if (fixed_regs[regno])
    2326              :               record = true;
    2327     78948631 :             else if (GET_MODE_CLASS (GET_MODE (x)) == MODE_CC)
    2328              :               record = true;
    2329     78948631 :             else if (targetm.small_register_classes_for_mode_p (GET_MODE (x)))
    2330              :               record = false;
    2331            0 :             else if (targetm.class_likely_spilled_p (REGNO_REG_CLASS (regno)))
    2332              :               record = false;
    2333              :             else
    2334              :               record = true;
    2335              : 
    2336              :             if (!record)
    2337              :               {
    2338     78949008 :                 *do_not_record_p = 1;
    2339     78949008 :                 return 0;
    2340              :               }
    2341              :           }
    2342              : 
    2343    592931023 :         hash += ((unsigned int) REG << 7);
    2344    592931023 :         hash += (have_reg_qty ? (unsigned) REG_QTY (regno) : regno);
    2345    592931023 :         return hash;
    2346              :       }
    2347              : 
    2348              :     /* We handle SUBREG of a REG specially because the underlying
    2349              :        reg changes its hash value with every value change; we don't
    2350              :        want to have to forget unrelated subregs when one subreg changes.  */
    2351     37849626 :     case SUBREG:
    2352     37849626 :       {
    2353     37849626 :         if (REG_P (SUBREG_REG (x)))
    2354              :           {
    2355     75568958 :             hash += (((unsigned int) SUBREG << 7)
    2356     37784479 :                      + REGNO (SUBREG_REG (x))
    2357     37784479 :                      + (constant_lower_bound (SUBREG_BYTE (x))
    2358     37784479 :                         / UNITS_PER_WORD));
    2359     37784479 :             return hash;
    2360              :           }
    2361              :         break;
    2362              :       }
    2363              : 
    2364    349841088 :     case CONST_INT:
    2365    349841088 :       hash += (((unsigned int) CONST_INT << 7) + (unsigned int) mode
    2366    349841088 :                + (unsigned int) INTVAL (x));
    2367    349841088 :       return hash;
    2368              : 
    2369              :     case CONST_WIDE_INT:
    2370      3084753 :       for (i = 0; i < CONST_WIDE_INT_NUNITS (x); i++)
    2371      2056666 :         hash += CONST_WIDE_INT_ELT (x, i);
    2372              :       return hash;
    2373              : 
    2374            0 :     case CONST_POLY_INT:
    2375            0 :       {
    2376            0 :         inchash::hash h;
    2377            0 :         h.add_int (hash);
    2378            0 :         for (unsigned int i = 0; i < NUM_POLY_INT_COEFFS; ++i)
    2379            0 :           h.add_wide_int (CONST_POLY_INT_COEFFS (x)[i]);
    2380            0 :         return h.end ();
    2381              :       }
    2382              : 
    2383      4438658 :     case CONST_DOUBLE:
    2384              :       /* This is like the general case, except that it only counts
    2385              :          the integers representing the constant.  */
    2386      4438658 :       hash += (unsigned int) code + (unsigned int) GET_MODE (x);
    2387      4438658 :       if (TARGET_SUPPORTS_WIDE_INT == 0 && GET_MODE (x) == VOIDmode)
    2388              :         hash += ((unsigned int) CONST_DOUBLE_LOW (x)
    2389              :                  + (unsigned int) CONST_DOUBLE_HIGH (x));
    2390              :       else
    2391      4438658 :         hash += real_hash (CONST_DOUBLE_REAL_VALUE (x));
    2392      4438658 :       return hash;
    2393              : 
    2394            0 :     case CONST_FIXED:
    2395            0 :       hash += (unsigned int) code + (unsigned int) GET_MODE (x);
    2396            0 :       hash += fixed_hash (CONST_FIXED_VALUE (x));
    2397            0 :       return hash;
    2398              : 
    2399      3917191 :     case CONST_VECTOR:
    2400      3917191 :       {
    2401      3917191 :         int units;
    2402      3917191 :         rtx elt;
    2403              : 
    2404      3917191 :         units = const_vector_encoded_nelts (x);
    2405              : 
    2406     10607256 :         for (i = 0; i < units; ++i)
    2407              :           {
    2408      6690065 :             elt = CONST_VECTOR_ENCODED_ELT (x, i);
    2409      6690065 :             hash += hash_rtx (elt, GET_MODE (elt),
    2410              :                               do_not_record_p, hash_arg_in_memory_p,
    2411              :                               have_reg_qty, cb);
    2412              :           }
    2413              : 
    2414              :         return hash;
    2415              :       }
    2416              : 
    2417              :       /* Assume there is only one rtx object for any given label.  */
    2418     20611726 :     case LABEL_REF:
    2419              :       /* We don't hash on the address of the CODE_LABEL to avoid bootstrap
    2420              :          differences and differences between each stage's debugging dumps.  */
    2421     20611726 :          hash += (((unsigned int) LABEL_REF << 7)
    2422     20611726 :                   + CODE_LABEL_NUMBER (label_ref_label (x)));
    2423     20611726 :       return hash;
    2424              : 
    2425    156857790 :     case SYMBOL_REF:
    2426    156857790 :       {
    2427              :         /* Don't hash on the symbol's address to avoid bootstrap differences.
    2428              :            Different hash values may cause expressions to be recorded in
    2429              :            different orders and thus different registers to be used in the
    2430              :            final assembler.  This also avoids differences in the dump files
    2431              :            between various stages.  */
    2432    156857790 :         unsigned int h = 0;
    2433    156857790 :         const unsigned char *p = (const unsigned char *) XSTR (x, 0);
    2434              : 
    2435   3546992768 :         while (*p)
    2436   3390134978 :           h += (h << 7) + *p++; /* ??? revisit */
    2437              : 
    2438    156857790 :         hash += ((unsigned int) SYMBOL_REF << 7) + h;
    2439    156857790 :         return hash;
    2440              :       }
    2441              : 
    2442    272282390 :     case MEM:
    2443              :       /* We don't record if marked volatile or if BLKmode since we don't
    2444              :          know the size of the move.  */
    2445    272282390 :       if (do_not_record_p && (MEM_VOLATILE_P (x) || GET_MODE (x) == BLKmode))
    2446              :         {
    2447      5573460 :           *do_not_record_p = 1;
    2448      5573460 :           return 0;
    2449              :         }
    2450    266708930 :       if (hash_arg_in_memory_p && !MEM_READONLY_P (x))
    2451     60768975 :         *hash_arg_in_memory_p = 1;
    2452              : 
    2453              :       /* Now that we have already found this special case,
    2454              :          might as well speed it up as much as possible.  */
    2455    266708930 :       hash += (unsigned) MEM;
    2456    266708930 :       x = XEXP (x, 0);
    2457    266708930 :       goto repeat;
    2458              : 
    2459           73 :     case USE:
    2460              :       /* A USE that mentions non-volatile memory needs special
    2461              :          handling since the MEM may be BLKmode which normally
    2462              :          prevents an entry from being made.  Pure calls are
    2463              :          marked by a USE which mentions BLKmode memory.
    2464              :          See calls.cc:emit_call_1.  */
    2465           73 :       if (MEM_P (XEXP (x, 0))
    2466           73 :           && ! MEM_VOLATILE_P (XEXP (x, 0)))
    2467              :         {
    2468            0 :           hash += (unsigned) USE;
    2469            0 :           x = XEXP (x, 0);
    2470              : 
    2471            0 :           if (hash_arg_in_memory_p && !MEM_READONLY_P (x))
    2472            0 :             *hash_arg_in_memory_p = 1;
    2473              : 
    2474              :           /* Now that we have already found this special case,
    2475              :              might as well speed it up as much as possible.  */
    2476            0 :           hash += (unsigned) MEM;
    2477            0 :           x = XEXP (x, 0);
    2478            0 :           goto repeat;
    2479              :         }
    2480              :       break;
    2481              : 
    2482     52447159 :     case PRE_DEC:
    2483     52447159 :     case PRE_INC:
    2484     52447159 :     case POST_DEC:
    2485     52447159 :     case POST_INC:
    2486     52447159 :     case PRE_MODIFY:
    2487     52447159 :     case POST_MODIFY:
    2488     52447159 :     case PC:
    2489     52447159 :     case CALL:
    2490     52447159 :     case UNSPEC_VOLATILE:
    2491     52447159 :       if (do_not_record_p) {
    2492     52445782 :         *do_not_record_p = 1;
    2493     52445782 :         return 0;
    2494              :       }
    2495              :       else
    2496              :         return hash;
    2497       211714 :       break;
    2498              : 
    2499       211714 :     case ASM_OPERANDS:
    2500       211714 :       if (do_not_record_p && MEM_VOLATILE_P (x))
    2501              :         {
    2502       174406 :           *do_not_record_p = 1;
    2503       174406 :           return 0;
    2504              :         }
    2505              :       else
    2506              :         {
    2507              :           /* We don't want to take the filename and line into account.  */
    2508        74616 :           hash += (unsigned) code + (unsigned) GET_MODE (x)
    2509        37308 :             + hash_rtx_string (ASM_OPERANDS_TEMPLATE (x))
    2510        37308 :             + hash_rtx_string (ASM_OPERANDS_OUTPUT_CONSTRAINT (x))
    2511        37308 :             + (unsigned) ASM_OPERANDS_OUTPUT_IDX (x);
    2512              : 
    2513        37308 :           if (ASM_OPERANDS_INPUT_LENGTH (x))
    2514              :             {
    2515        56915 :               for (i = 1; i < ASM_OPERANDS_INPUT_LENGTH (x); i++)
    2516              :                 {
    2517        48840 :                   hash += (hash_rtx (ASM_OPERANDS_INPUT (x, i),
    2518        24420 :                                      GET_MODE (ASM_OPERANDS_INPUT (x, i)),
    2519              :                                      do_not_record_p, hash_arg_in_memory_p,
    2520              :                                      have_reg_qty, cb)
    2521        24420 :                            + hash_rtx_string
    2522        48840 :                            (ASM_OPERANDS_INPUT_CONSTRAINT (x, i)));
    2523              :                 }
    2524              : 
    2525        32495 :               hash += hash_rtx_string (ASM_OPERANDS_INPUT_CONSTRAINT (x, 0));
    2526        32495 :               x = ASM_OPERANDS_INPUT (x, 0);
    2527        32495 :               mode = GET_MODE (x);
    2528        32495 :               goto repeat;
    2529              :             }
    2530              : 
    2531              :           return hash;
    2532              :         }
    2533              :       break;
    2534              : 
    2535              :     default:
    2536              :       break;
    2537              :     }
    2538              : 
    2539    362968958 :   i = GET_RTX_LENGTH (code) - 1;
    2540    362968958 :   hash += (unsigned) code + (unsigned) GET_MODE (x);
    2541    362968958 :   fmt = GET_RTX_FORMAT (code);
    2542    734276960 :   for (; i >= 0; i--)
    2543              :     {
    2544    727754059 :       switch (fmt[i])
    2545              :         {
    2546    715828694 :         case 'e':
    2547              :           /* If we are about to do the last recursive call
    2548              :              needed at this level, change it into iteration.
    2549              :              This function  is called enough to be worth it.  */
    2550    715828694 :           if (i == 0)
    2551              :             {
    2552    356446057 :               x = XEXP (x, i);
    2553    356446057 :               goto repeat;
    2554              :             }
    2555              : 
    2556    359382637 :           hash += hash_rtx (XEXP (x, i), VOIDmode, do_not_record_p,
    2557              :                             hash_arg_in_memory_p,
    2558              :                             have_reg_qty, cb);
    2559    359382637 :           break;
    2560              : 
    2561              :         case 'E':
    2562     16746761 :           for (j = 0; j < XVECLEN (x, i); j++)
    2563     10224108 :             hash += hash_rtx (XVECEXP (x, i, j), VOIDmode, do_not_record_p,
    2564              :                               hash_arg_in_memory_p,
    2565              :                               have_reg_qty, cb);
    2566              :           break;
    2567              : 
    2568            0 :         case 's':
    2569            0 :           hash += hash_rtx_string (XSTR (x, i));
    2570            0 :           break;
    2571              : 
    2572      5337513 :         case 'i':
    2573      5337513 :           hash += (unsigned int) XINT (x, i);
    2574      5337513 :           break;
    2575              : 
    2576            0 :         case 'L':
    2577            0 :           hash += (unsigned int) XLOC (x, i);
    2578            0 :           break;
    2579              : 
    2580        65147 :         case 'p':
    2581        65147 :           hash += constant_lower_bound (SUBREG_BYTE (x));
    2582        65147 :           break;
    2583              : 
    2584              :         case '0': case 't':
    2585              :           /* Unused.  */
    2586              :           break;
    2587              : 
    2588            0 :         default:
    2589            0 :           gcc_unreachable ();
    2590              :         }
    2591              :     }
    2592              : 
    2593              :   return hash;
    2594              : }
    2595              : 
    2596              : /* Hash an rtx X for cse via hash_rtx.
    2597              :    Stores 1 in do_not_record if any subexpression is volatile.
    2598              :    Stores 1 in hash_arg_in_memory if X contains a mem rtx which
    2599              :    does not have the MEM_READONLY_P flag set.  */
    2600              : 
    2601              : static inline unsigned
    2602    526701661 : canon_hash (rtx x, machine_mode mode)
    2603              : {
    2604    526701661 :   return hash_rtx (x, mode, &do_not_record, &hash_arg_in_memory, true);
    2605              : }
    2606              : 
    2607              : /* Like canon_hash but with no side effects, i.e. do_not_record
    2608              :    and hash_arg_in_memory are not changed.  */
    2609              : 
    2610              : static inline unsigned
    2611    169991960 : safe_hash (rtx x, machine_mode mode)
    2612              : {
    2613    169991960 :   int dummy_do_not_record;
    2614    169991960 :   return hash_rtx (x, mode, &dummy_do_not_record, NULL, true);
    2615              : }
    2616              : 
    2617              : /* Return true iff X and Y would canonicalize into the same thing,
    2618              :    without actually constructing the canonicalization of either one.
    2619              :    If VALIDATE is nonzero,
    2620              :    we assume X is an expression being processed from the rtl
    2621              :    and Y was found in the hash table.  We check register refs
    2622              :    in Y for being marked as valid.
    2623              : 
    2624              :    If FOR_GCSE is true, we compare X and Y for equivalence for GCSE.  */
    2625              : 
    2626              : bool
    2627    796455072 : exp_equiv_p (const_rtx x, const_rtx y, int validate, bool for_gcse)
    2628              : {
    2629    796455072 :   int i, j;
    2630    796455072 :   enum rtx_code code;
    2631    796455072 :   const char *fmt;
    2632              : 
    2633              :   /* Note: it is incorrect to assume an expression is equivalent to itself
    2634              :      if VALIDATE is nonzero.  */
    2635    796455072 :   if (x == y && !validate)
    2636              :     return true;
    2637              : 
    2638    773016224 :   if (x == 0 || y == 0)
    2639              :     return x == y;
    2640              : 
    2641    773016224 :   code = GET_CODE (x);
    2642    773016224 :   if (code != GET_CODE (y))
    2643              :     return false;
    2644              : 
    2645              :   /* (MULT:SI x y) and (MULT:HI x y) are NOT equivalent.  */
    2646    661897683 :   if (GET_MODE (x) != GET_MODE (y))
    2647              :     return false;
    2648              : 
    2649              :   /* MEMs referring to different address space are not equivalent.  */
    2650    692522084 :   if (code == MEM && MEM_ADDR_SPACE (x) != MEM_ADDR_SPACE (y))
    2651              :     return false;
    2652              : 
    2653    584098295 :   switch (code)
    2654              :     {
    2655              :     case PC:
    2656              :     CASE_CONST_UNIQUE:
    2657              :       return x == y;
    2658              : 
    2659              :     case CONST_VECTOR:
    2660              :       if (!same_vector_encodings_p (x, y))
    2661              :         return false;
    2662              :       break;
    2663              : 
    2664        24070 :     case LABEL_REF:
    2665        24070 :       return label_ref_label (x) == label_ref_label (y);
    2666              : 
    2667     18669881 :     case SYMBOL_REF:
    2668     18669881 :       return XSTR (x, 0) == XSTR (y, 0);
    2669              : 
    2670    169994026 :     case REG:
    2671    169994026 :       if (for_gcse)
    2672      1321751 :         return REGNO (x) == REGNO (y);
    2673              :       else
    2674              :         {
    2675    168672275 :           unsigned int regno = REGNO (y);
    2676    168672275 :           unsigned int i;
    2677    168672275 :           unsigned int endregno = END_REGNO (y);
    2678              : 
    2679              :           /* If the quantities are not the same, the expressions are not
    2680              :              equivalent.  If there are and we are not to validate, they
    2681              :              are equivalent.  Otherwise, ensure all regs are up-to-date.  */
    2682              : 
    2683    168672275 :           if (REG_QTY (REGNO (x)) != REG_QTY (regno))
    2684              :             return false;
    2685              : 
    2686    154798226 :           if (! validate)
    2687              :             return true;
    2688              : 
    2689    286048465 :           for (i = regno; i < endregno; i++)
    2690    144211982 :             if (REG_IN_TABLE (i) != REG_TICK (i))
    2691              :               return false;
    2692              : 
    2693              :           return true;
    2694              :         }
    2695              : 
    2696    106946296 :     case MEM:
    2697    106946296 :       if (for_gcse)
    2698              :         {
    2699              :           /* A volatile mem should not be considered equivalent to any
    2700              :              other.  */
    2701     59067781 :           if (MEM_VOLATILE_P (x) || MEM_VOLATILE_P (y))
    2702              :             return false;
    2703              : 
    2704              :           /* Can't merge two expressions in different alias sets, since we
    2705              :              can decide that the expression is transparent in a block when
    2706              :              it isn't, due to it being set with the different alias set.
    2707              : 
    2708              :              Also, can't merge two expressions with different MEM_ATTRS.
    2709              :              They could e.g. be two different entities allocated into the
    2710              :              same space on the stack (see e.g. PR25130).  In that case, the
    2711              :              MEM addresses can be the same, even though the two MEMs are
    2712              :              absolutely not equivalent.
    2713              : 
    2714              :              But because really all MEM attributes should be the same for
    2715              :              equivalent MEMs, we just use the invariant that MEMs that have
    2716              :              the same attributes share the same mem_attrs data structure.  */
    2717     58956129 :           if (!mem_attrs_eq_p (MEM_ATTRS (x), MEM_ATTRS (y)))
    2718              :             return false;
    2719              : 
    2720              :           /* If we are handling exceptions, we cannot consider two expressions
    2721              :              with different trapping status as equivalent, because simple_mem
    2722              :              might accept one and reject the other.  */
    2723      9186477 :           if (cfun->can_throw_non_call_exceptions
    2724      9186477 :               && (MEM_NOTRAP_P (x) != MEM_NOTRAP_P (y)))
    2725              :             return false;
    2726              :         }
    2727              :       break;
    2728              : 
    2729              :     /*  For commutative operations, check both orders.  */
    2730     74008774 :     case PLUS:
    2731     74008774 :     case MULT:
    2732     74008774 :     case AND:
    2733     74008774 :     case IOR:
    2734     74008774 :     case XOR:
    2735     74008774 :     case NE:
    2736     74008774 :     case EQ:
    2737     74008774 :       return ((exp_equiv_p (XEXP (x, 0), XEXP (y, 0),
    2738              :                              validate, for_gcse)
    2739     68233375 :                && exp_equiv_p (XEXP (x, 1), XEXP (y, 1),
    2740              :                                 validate, for_gcse))
    2741     81087290 :               || (exp_equiv_p (XEXP (x, 0), XEXP (y, 1),
    2742              :                                 validate, for_gcse)
    2743        18130 :                   && exp_equiv_p (XEXP (x, 1), XEXP (y, 0),
    2744              :                                    validate, for_gcse)));
    2745              : 
    2746        13119 :     case ASM_OPERANDS:
    2747              :       /* We don't use the generic code below because we want to
    2748              :          disregard filename and line numbers.  */
    2749              : 
    2750              :       /* A volatile asm isn't equivalent to any other.  */
    2751        13119 :       if (MEM_VOLATILE_P (x) || MEM_VOLATILE_P (y))
    2752              :         return false;
    2753              : 
    2754        13119 :       if (GET_MODE (x) != GET_MODE (y)
    2755        13119 :           || strcmp (ASM_OPERANDS_TEMPLATE (x), ASM_OPERANDS_TEMPLATE (y))
    2756        13119 :           || strcmp (ASM_OPERANDS_OUTPUT_CONSTRAINT (x),
    2757        13119 :                      ASM_OPERANDS_OUTPUT_CONSTRAINT (y))
    2758        13109 :           || ASM_OPERANDS_OUTPUT_IDX (x) != ASM_OPERANDS_OUTPUT_IDX (y)
    2759        13109 :           || ASM_OPERANDS_INPUT_LENGTH (x) != ASM_OPERANDS_INPUT_LENGTH (y))
    2760              :         return false;
    2761              : 
    2762        13109 :       if (ASM_OPERANDS_INPUT_LENGTH (x))
    2763              :         {
    2764        17809 :           for (i = ASM_OPERANDS_INPUT_LENGTH (x) - 1; i >= 0; i--)
    2765         9020 :             if (! exp_equiv_p (ASM_OPERANDS_INPUT (x, i),
    2766         9020 :                                ASM_OPERANDS_INPUT (y, i),
    2767              :                                validate, for_gcse)
    2768         9020 :                 || strcmp (ASM_OPERANDS_INPUT_CONSTRAINT (x, i),
    2769         8907 :                            ASM_OPERANDS_INPUT_CONSTRAINT (y, i)))
    2770              :               return false;
    2771              :         }
    2772              : 
    2773              :       return true;
    2774              : 
    2775              :     default:
    2776              :       break;
    2777              :     }
    2778              : 
    2779              :   /* Compare the elements.  If any pair of corresponding elements
    2780              :      fail to match, return 0 for the whole thing.  */
    2781              : 
    2782    127609886 :   fmt = GET_RTX_FORMAT (code);
    2783    356191597 :   for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
    2784              :     {
    2785    241262202 :       switch (fmt[i])
    2786              :         {
    2787    162256766 :         case 'e':
    2788    162256766 :           if (! exp_equiv_p (XEXP (x, i), XEXP (y, i),
    2789              :                               validate, for_gcse))
    2790              :             return false;
    2791              :           break;
    2792              : 
    2793      9798128 :         case 'E':
    2794      9798128 :           if (XVECLEN (x, i) != XVECLEN (y, i))
    2795              :             return 0;
    2796     47079858 :           for (j = 0; j < XVECLEN (x, i); j++)
    2797     38186350 :             if (! exp_equiv_p (XVECEXP (x, i, j), XVECEXP (y, i, j),
    2798              :                                 validate, for_gcse))
    2799              :               return false;
    2800              :           break;
    2801              : 
    2802            0 :         case 's':
    2803            0 :           if (strcmp (XSTR (x, i), XSTR (y, i)))
    2804              :             return false;
    2805              :           break;
    2806              : 
    2807      3628790 :         case 'i':
    2808      3628790 :           if (XINT (x, i) != XINT (y, i))
    2809              :             return false;
    2810              :           break;
    2811              : 
    2812            0 :         case 'L':
    2813            0 :           if (XLOC (x, i) != XLOC (y, i))
    2814              :             return false;
    2815              :           break;
    2816              : 
    2817            0 :         case 'w':
    2818            0 :           if (XWINT (x, i) != XWINT (y, i))
    2819              :             return false;
    2820              :           break;
    2821              : 
    2822      8523184 :         case 'p':
    2823      8523184 :           if (maybe_ne (SUBREG_BYTE (x), SUBREG_BYTE (y)))
    2824              :             return false;
    2825              :           break;
    2826              : 
    2827              :         case '0':
    2828              :         case 't':
    2829              :           break;
    2830              : 
    2831            0 :         default:
    2832            0 :           gcc_unreachable ();
    2833              :         }
    2834              :     }
    2835              : 
    2836              :   return true;
    2837              : }
    2838              : 
    2839              : /* Subroutine of canon_reg.  Pass *XLOC through canon_reg, and validate
    2840              :    the result if necessary.  INSN is as for canon_reg.  */
    2841              : 
    2842              : static void
    2843   1058449243 : validate_canon_reg (rtx *xloc, rtx_insn *insn)
    2844              : {
    2845   1058449243 :   if (*xloc)
    2846              :     {
    2847   1058449243 :       rtx new_rtx = canon_reg (*xloc, insn);
    2848              : 
    2849              :       /* If replacing pseudo with hard reg or vice versa, ensure the
    2850              :          insn remains valid.  Likewise if the insn has MATCH_DUPs.  */
    2851   1058449243 :       gcc_assert (insn && new_rtx);
    2852   1058449243 :       validate_change (insn, xloc, new_rtx, 1);
    2853              :     }
    2854   1058449243 : }
    2855              : 
    2856              : /* Canonicalize an expression:
    2857              :    replace each register reference inside it
    2858              :    with the "oldest" equivalent register.
    2859              : 
    2860              :    If INSN is nonzero validate_change is used to ensure that INSN remains valid
    2861              :    after we make our substitution.  The calls are made with IN_GROUP nonzero
    2862              :    so apply_change_group must be called upon the outermost return from this
    2863              :    function (unless INSN is zero).  The result of apply_change_group can
    2864              :    generally be discarded since the changes we are making are optional.  */
    2865              : 
    2866              : static rtx
    2867   1754214475 : canon_reg (rtx x, rtx_insn *insn)
    2868              : {
    2869   1754214475 :   int i;
    2870   1754214475 :   enum rtx_code code;
    2871   1754214475 :   const char *fmt;
    2872              : 
    2873   1754214475 :   if (x == 0)
    2874              :     return x;
    2875              : 
    2876   1754214475 :   code = GET_CODE (x);
    2877   1754214475 :   switch (code)
    2878              :     {
    2879              :     case PC:
    2880              :     case CONST:
    2881              :     CASE_CONST_ANY:
    2882              :     case SYMBOL_REF:
    2883              :     case LABEL_REF:
    2884              :     case ADDR_VEC:
    2885              :     case ADDR_DIFF_VEC:
    2886              :       return x;
    2887              : 
    2888     10336308 :     case SUBREG:
    2889     10336308 :       {
    2890     10336308 :         rtx inner = canon_reg (SUBREG_REG (x), insn);
    2891     10336308 :         if (inner != SUBREG_REG (x))
    2892              :           {
    2893       258632 :             rtx newx = simplify_subreg (GET_MODE (x), inner,
    2894       129316 :                                         GET_MODE (SUBREG_REG (x)),
    2895       129316 :                                         SUBREG_BYTE (x));
    2896       129316 :             if (newx)
    2897              :               return newx;
    2898              : 
    2899       129316 :             if (validate_subreg (GET_MODE (x), GET_MODE (inner),
    2900       129316 :                                  inner, SUBREG_BYTE (x)))
    2901       129316 :               validate_change (insn, &SUBREG_REG (x), inner, 1);
    2902              :           }
    2903              :         return x;
    2904              :       }
    2905              : 
    2906    483585877 :     case REG:
    2907    483585877 :       {
    2908    483585877 :         int first;
    2909    483585877 :         int q;
    2910    483585877 :         struct qty_table_elem *ent;
    2911              : 
    2912              :         /* Never replace a hard reg, because hard regs can appear
    2913              :            in more than one machine mode, and we must preserve the mode
    2914              :            of each occurrence.  Also, some hard regs appear in
    2915              :            MEMs that are shared and mustn't be altered.  Don't try to
    2916              :            replace any reg that maps to a reg of class NO_REGS.  */
    2917    483585877 :         if (REGNO (x) < FIRST_PSEUDO_REGISTER
    2918    483585877 :             || ! REGNO_QTY_VALID_P (REGNO (x)))
    2919              :           return x;
    2920              : 
    2921    170897483 :         q = REG_QTY (REGNO (x));
    2922    170897483 :         ent = &qty_table[q];
    2923    170897483 :         first = ent->first_reg;
    2924    170897483 :         return (first >= FIRST_PSEUDO_REGISTER ? regno_reg_rtx[first]
    2925       406002 :                 : REGNO_REG_CLASS (first) == NO_REGS ? x
    2926    170897483 :                 : gen_rtx_REG (ent->mode, first));
    2927              :       }
    2928              : 
    2929    776818544 :     default:
    2930    776818544 :       break;
    2931              :     }
    2932              : 
    2933    776818544 :   fmt = GET_RTX_FORMAT (code);
    2934   2134359159 :   for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
    2935              :     {
    2936   1357540615 :       int j;
    2937              : 
    2938   1357540615 :       if (fmt[i] == 'e')
    2939   1045024044 :         validate_canon_reg (&XEXP (x, i), insn);
    2940    312516571 :       else if (fmt[i] == 'E')
    2941     20205435 :         for (j = 0; j < XVECLEN (x, i); j++)
    2942     13425199 :           validate_canon_reg (&XVECEXP (x, i, j), insn);
    2943              :     }
    2944              : 
    2945              :   return x;
    2946              : }
    2947              : 
    2948              : /* Given an operation (CODE, *PARG1, *PARG2), where code is a comparison
    2949              :    operation (EQ, NE, GT, etc.), follow it back through the hash table and
    2950              :    what values are being compared.
    2951              : 
    2952              :    *PARG1 and *PARG2 are updated to contain the rtx representing the values
    2953              :    actually being compared.  For example, if *PARG1 was (reg:CC CC_REG) and
    2954              :    *PARG2 was (const_int 0), *PARG1 and *PARG2 will be set to the objects that
    2955              :    were compared to produce (reg:CC CC_REG).
    2956              : 
    2957              :    The return value is the comparison operator and is either the code of
    2958              :    A or the code corresponding to the inverse of the comparison.  */
    2959              : 
    2960              : static enum rtx_code
    2961     37383098 : find_comparison_args (enum rtx_code code, rtx *parg1, rtx *parg2,
    2962              :                       machine_mode *pmode1, machine_mode *pmode2)
    2963              : {
    2964     37383098 :   rtx arg1, arg2;
    2965     37383098 :   hash_set<rtx> *visited = NULL;
    2966              :   /* Set nonzero when we find something of interest.  */
    2967     37383098 :   rtx x = NULL;
    2968              : 
    2969     37383098 :   arg1 = *parg1, arg2 = *parg2;
    2970              : 
    2971              :   /* If ARG2 is const0_rtx, see what ARG1 is equivalent to.  */
    2972              : 
    2973     72842369 :   while (arg2 == CONST0_RTX (GET_MODE (arg1)))
    2974              :     {
    2975     54408935 :       int reverse_code = 0;
    2976     54408935 :       struct table_elt *p = 0;
    2977              : 
    2978              :       /* Remember state from previous iteration.  */
    2979     54408935 :       if (x)
    2980              :         {
    2981     17102577 :           if (!visited)
    2982     17097925 :             visited = new hash_set<rtx>;
    2983     17102577 :           visited->add (x);
    2984     17102577 :           x = 0;
    2985              :         }
    2986              : 
    2987              :       /* If arg1 is a COMPARE, extract the comparison arguments from it.  */
    2988              : 
    2989     54408935 :       if (GET_CODE (arg1) == COMPARE && arg2 == const0_rtx)
    2990            0 :         x = arg1;
    2991              : 
    2992              :       /* If ARG1 is a comparison operator and CODE is testing for
    2993              :          STORE_FLAG_VALUE, get the inner arguments.  */
    2994              : 
    2995     54408935 :       else if (COMPARISON_P (arg1))
    2996              :         {
    2997              : #ifdef FLOAT_STORE_FLAG_VALUE
    2998              :           REAL_VALUE_TYPE fsfv;
    2999              : #endif
    3000              : 
    3001            0 :           if (code == NE
    3002              :               || (GET_MODE_CLASS (GET_MODE (arg1)) == MODE_INT
    3003              :                   && code == LT && STORE_FLAG_VALUE == -1)
    3004              : #ifdef FLOAT_STORE_FLAG_VALUE
    3005              :               || (SCALAR_FLOAT_MODE_P (GET_MODE (arg1))
    3006              :                   && (fsfv = FLOAT_STORE_FLAG_VALUE (GET_MODE (arg1)),
    3007              :                       REAL_VALUE_NEGATIVE (fsfv)))
    3008              : #endif
    3009              :               )
    3010            0 :             x = arg1;
    3011            0 :           else if (code == EQ
    3012              :                    || (GET_MODE_CLASS (GET_MODE (arg1)) == MODE_INT
    3013              :                        && code == GE && STORE_FLAG_VALUE == -1)
    3014              : #ifdef FLOAT_STORE_FLAG_VALUE
    3015              :                    || (SCALAR_FLOAT_MODE_P (GET_MODE (arg1))
    3016              :                        && (fsfv = FLOAT_STORE_FLAG_VALUE (GET_MODE (arg1)),
    3017              :                            REAL_VALUE_NEGATIVE (fsfv)))
    3018              : #endif
    3019              :                    )
    3020            0 :             x = arg1, reverse_code = 1;
    3021              :         }
    3022              : 
    3023              :       /* ??? We could also check for
    3024              : 
    3025              :          (ne (and (eq (...) (const_int 1))) (const_int 0))
    3026              : 
    3027              :          and related forms, but let's wait until we see them occurring.  */
    3028              : 
    3029     54408935 :       if (x == 0)
    3030              :         /* Look up ARG1 in the hash table and see if it has an equivalence
    3031              :            that lets us see what is being compared.  */
    3032     54408935 :         p = lookup (arg1, SAFE_HASH (arg1, GET_MODE (arg1)), GET_MODE (arg1));
    3033     54408935 :       if (p)
    3034              :         {
    3035     43440779 :           p = p->first_same_value;
    3036              : 
    3037              :           /* If what we compare is already known to be constant, that is as
    3038              :              good as it gets.
    3039              :              We need to break the loop in this case, because otherwise we
    3040              :              can have an infinite loop when looking at a reg that is known
    3041              :              to be a constant which is the same as a comparison of a reg
    3042              :              against zero which appears later in the insn stream, which in
    3043              :              turn is constant and the same as the comparison of the first reg
    3044              :              against zero...  */
    3045     43440779 :           if (p->is_const)
    3046              :             break;
    3047              :         }
    3048              : 
    3049     69714854 :       for (; p; p = p->next_same_value)
    3050              :         {
    3051     50770781 :           machine_mode inner_mode = GET_MODE (p->exp);
    3052              : #ifdef FLOAT_STORE_FLAG_VALUE
    3053              :           REAL_VALUE_TYPE fsfv;
    3054              : #endif
    3055              : 
    3056              :           /* If the entry isn't valid, skip it.  */
    3057     50770781 :           if (! exp_equiv_p (p->exp, p->exp, 1, false))
    3058      1847917 :             continue;
    3059              : 
    3060              :           /* If it's a comparison we've used before, skip it.  */
    3061     48922864 :           if (visited && visited->contains (p->exp))
    3062            0 :             continue;
    3063              : 
    3064     48922864 :           if (GET_CODE (p->exp) == COMPARE
    3065              :               /* Another possibility is that this machine has a compare insn
    3066              :                  that includes the comparison code.  In that case, ARG1 would
    3067              :                  be equivalent to a comparison operation that would set ARG1 to
    3068              :                  either STORE_FLAG_VALUE or zero.  If this is an NE operation,
    3069              :                  ORIG_CODE is the actual comparison being done; if it is an EQ,
    3070              :                  we must reverse ORIG_CODE.  On machine with a negative value
    3071              :                  for STORE_FLAG_VALUE, also look at LT and GE operations.  */
    3072     48922864 :               || ((code == NE
    3073      9162068 :                    || (code == LT
    3074       261134 :                        && val_signbit_known_set_p (inner_mode,
    3075              :                                                    STORE_FLAG_VALUE))
    3076              : #ifdef FLOAT_STORE_FLAG_VALUE
    3077              :                    || (code == LT
    3078              :                        && SCALAR_FLOAT_MODE_P (inner_mode)
    3079              :                        && (fsfv = FLOAT_STORE_FLAG_VALUE (GET_MODE (arg1)),
    3080              :                            REAL_VALUE_NEGATIVE (fsfv)))
    3081              : #endif
    3082              :                    )
    3083      4409421 :                   && COMPARISON_P (p->exp)))
    3084              :             {
    3085     35354744 :               x = p->exp;
    3086     35354744 :               break;
    3087              :             }
    3088     13568120 :           else if ((code == EQ
    3089      7599831 :                     || (code == GE
    3090       229245 :                         && val_signbit_known_set_p (inner_mode,
    3091              :                                                     STORE_FLAG_VALUE))
    3092              : #ifdef FLOAT_STORE_FLAG_VALUE
    3093              :                     || (code == GE
    3094              :                         && SCALAR_FLOAT_MODE_P (inner_mode)
    3095              :                         && (fsfv = FLOAT_STORE_FLAG_VALUE (GET_MODE (arg1)),
    3096              :                             REAL_VALUE_NEGATIVE (fsfv)))
    3097              : #endif
    3098              :                     )
    3099     13568120 :                    && COMPARISON_P (p->exp))
    3100              :             {
    3101       104527 :               reverse_code = 1;
    3102       104527 :               x = p->exp;
    3103       104527 :               break;
    3104              :             }
    3105              : 
    3106              :           /* If this non-trapping address, e.g. fp + constant, the
    3107              :              equivalent is a better operand since it may let us predict
    3108              :              the value of the comparison.  */
    3109     13463593 :           else if (!rtx_addr_can_trap_p (p->exp))
    3110              :             {
    3111            0 :               arg1 = p->exp;
    3112            0 :               continue;
    3113              :             }
    3114              :         }
    3115              : 
    3116              :       /* If we didn't find a useful equivalence for ARG1, we are done.
    3117              :          Otherwise, set up for the next iteration.  */
    3118     54403344 :       if (x == 0)
    3119              :         break;
    3120              : 
    3121              :       /* If we need to reverse the comparison, make sure that is
    3122              :          possible -- we can't necessarily infer the value of GE from LT
    3123              :          with floating-point operands.  */
    3124     35459271 :       if (reverse_code)
    3125              :         {
    3126       104527 :           enum rtx_code reversed = reversed_comparison_code (x, NULL);
    3127       104527 :           if (reversed == UNKNOWN)
    3128              :             break;
    3129              :           else
    3130              :             code = reversed;
    3131              :         }
    3132     35354744 :       else if (COMPARISON_P (x))
    3133         3369 :         code = GET_CODE (x);
    3134     35459271 :       arg1 = XEXP (x, 0), arg2 = XEXP (x, 1);
    3135              :     }
    3136              : 
    3137              :   /* Return our results.  Return the modes from before fold_rtx
    3138              :      because fold_rtx might produce const_int, and then it's too late.  */
    3139     37383098 :   *pmode1 = GET_MODE (arg1), *pmode2 = GET_MODE (arg2);
    3140     37383098 :   *parg1 = fold_rtx (arg1, 0), *parg2 = fold_rtx (arg2, 0);
    3141              : 
    3142     37383098 :   if (visited)
    3143     17097925 :     delete visited;
    3144     37383098 :   return code;
    3145              : }
    3146              : 
    3147              : /* If X is a nontrivial arithmetic operation on an argument for which
    3148              :    a constant value can be determined, return the result of operating
    3149              :    on that value, as a constant.  Otherwise, return X, possibly with
    3150              :    one or more operands changed to a forward-propagated constant.
    3151              : 
    3152              :    If X is a register whose contents are known, we do NOT return
    3153              :    those contents here; equiv_constant is called to perform that task.
    3154              :    For SUBREGs and MEMs, we do that both here and in equiv_constant.
    3155              : 
    3156              :    INSN is the insn that we may be modifying.  If it is 0, make a copy
    3157              :    of X before modifying it.  */
    3158              : 
    3159              : static rtx
    3160    405042424 : fold_rtx (rtx x, rtx_insn *insn)
    3161              : {
    3162    405044255 :   enum rtx_code code;
    3163    405044255 :   machine_mode mode;
    3164    405044255 :   const char *fmt;
    3165    405044255 :   int i;
    3166    405044255 :   rtx new_rtx = 0;
    3167    405044255 :   bool changed = false;
    3168    405044255 :   poly_int64 xval;
    3169              : 
    3170              :   /* Operands of X.  */
    3171              :   /* Workaround -Wmaybe-uninitialized false positive during
    3172              :      profiledbootstrap by initializing them.  */
    3173    405044255 :   rtx folded_arg0 = NULL_RTX;
    3174    405044255 :   rtx folded_arg1 = NULL_RTX;
    3175              : 
    3176              :   /* Constant equivalents of first three operands of X;
    3177              :      0 when no such equivalent is known.  */
    3178    405044255 :   rtx const_arg0;
    3179    405044255 :   rtx const_arg1;
    3180    405044255 :   rtx const_arg2;
    3181              : 
    3182              :   /* The mode of the first operand of X.  We need this for sign and zero
    3183              :      extends.  */
    3184    405044255 :   machine_mode mode_arg0;
    3185              : 
    3186    405044255 :   if (x == 0)
    3187              :     return x;
    3188              : 
    3189              :   /* Try to perform some initial simplifications on X.  */
    3190    405044255 :   code = GET_CODE (x);
    3191    405044255 :   switch (code)
    3192              :     {
    3193     63371791 :     case MEM:
    3194     63371791 :     case SUBREG:
    3195              :     /* The first operand of a SIGN/ZERO_EXTRACT has a different meaning
    3196              :        than it would in other contexts.  Basically its mode does not
    3197              :        signify the size of the object read.  That information is carried
    3198              :        by size operand.    If we happen to have a MEM of the appropriate
    3199              :        mode in our tables with a constant value we could simplify the
    3200              :        extraction incorrectly if we allowed substitution of that value
    3201              :        for the MEM.   */
    3202     63371791 :     case ZERO_EXTRACT:
    3203     63371791 :     case SIGN_EXTRACT:
    3204     63371791 :       if ((new_rtx = equiv_constant (x)) != NULL_RTX)
    3205      2352895 :         return new_rtx;
    3206              :       return x;
    3207              : 
    3208              :     case CONST:
    3209              :     CASE_CONST_ANY:
    3210              :     case SYMBOL_REF:
    3211              :     case LABEL_REF:
    3212              :     case REG:
    3213              :     case PC:
    3214              :       /* No use simplifying an EXPR_LIST
    3215              :          since they are used only for lists of args
    3216              :          in a function call's REG_EQUAL note.  */
    3217              :     case EXPR_LIST:
    3218              :       return x;
    3219              : 
    3220       210780 :     case ASM_OPERANDS:
    3221       210780 :       if (insn)
    3222              :         {
    3223            0 :           for (i = ASM_OPERANDS_INPUT_LENGTH (x) - 1; i >= 0; i--)
    3224            0 :             validate_change (insn, &ASM_OPERANDS_INPUT (x, i),
    3225            0 :                              fold_rtx (ASM_OPERANDS_INPUT (x, i), insn), 0);
    3226              :         }
    3227              :       return x;
    3228              : 
    3229     15857452 :     case CALL:
    3230     15857452 :       if (NO_FUNCTION_CSE && CONSTANT_P (XEXP (XEXP (x, 0), 0)))
    3231              :         return x;
    3232              :       break;
    3233       946417 :     case VEC_SELECT:
    3234       946417 :       {
    3235       946417 :         rtx trueop0 = XEXP (x, 0);
    3236       946417 :         mode = GET_MODE (trueop0);
    3237       946417 :         rtx trueop1 = XEXP (x, 1);
    3238              :         /* If we select a low-part subreg, return that.  */
    3239       946417 :         if (vec_series_lowpart_p (GET_MODE (x), mode, trueop1))
    3240              :           {
    3241          219 :             rtx new_rtx = lowpart_subreg (GET_MODE (x), trueop0, mode);
    3242          219 :             if (new_rtx != NULL_RTX)
    3243              :               return new_rtx;
    3244              :           }
    3245              :       }
    3246              : 
    3247              :     /* Anything else goes through the loop below.  */
    3248              :     default:
    3249              :       break;
    3250              :     }
    3251              : 
    3252    118337966 :   mode = GET_MODE (x);
    3253    118337966 :   const_arg0 = 0;
    3254    118337966 :   const_arg1 = 0;
    3255    118337966 :   const_arg2 = 0;
    3256    118337966 :   mode_arg0 = VOIDmode;
    3257              : 
    3258              :   /* Try folding our operands.
    3259              :      Then see which ones have constant values known.  */
    3260              : 
    3261    118337966 :   fmt = GET_RTX_FORMAT (code);
    3262    369101996 :   for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
    3263    250764030 :     if (fmt[i] == 'e')
    3264              :       {
    3265    246165092 :         rtx folded_arg = XEXP (x, i), const_arg;
    3266    246165092 :         machine_mode mode_arg = GET_MODE (folded_arg);
    3267              : 
    3268    246165092 :         switch (GET_CODE (folded_arg))
    3269              :           {
    3270    110251827 :           case MEM:
    3271    110251827 :           case REG:
    3272    110251827 :           case SUBREG:
    3273    110251827 :             const_arg = equiv_constant (folded_arg);
    3274    110251827 :             break;
    3275              : 
    3276              :           case CONST:
    3277              :           CASE_CONST_ANY:
    3278              :           case SYMBOL_REF:
    3279              :           case LABEL_REF:
    3280              :             const_arg = folded_arg;
    3281              :             break;
    3282              : 
    3283     46270484 :           default:
    3284     46270484 :             folded_arg = fold_rtx (folded_arg, insn);
    3285     46270484 :             const_arg = equiv_constant (folded_arg);
    3286     46270484 :             break;
    3287              :           }
    3288              : 
    3289              :         /* For the first three operands, see if the operand
    3290              :            is constant or equivalent to a constant.  */
    3291    246165092 :         switch (i)
    3292              :           {
    3293    115565398 :           case 0:
    3294    115565398 :             folded_arg0 = folded_arg;
    3295    115565398 :             const_arg0 = const_arg;
    3296    115565398 :             mode_arg0 = mode_arg;
    3297    115565398 :             break;
    3298    109384150 :           case 1:
    3299    109384150 :             folded_arg1 = folded_arg;
    3300    109384150 :             const_arg1 = const_arg;
    3301    109384150 :             break;
    3302     21215544 :           case 2:
    3303     21215544 :             const_arg2 = const_arg;
    3304     21215544 :             break;
    3305              :           }
    3306              : 
    3307              :         /* Pick the least expensive of the argument and an equivalent constant
    3308              :            argument.  */
    3309    246165092 :         if (const_arg != 0
    3310    246165092 :             && const_arg != folded_arg
    3311      6083284 :             && (COST_IN (const_arg, mode_arg, code, i)
    3312      3041642 :                 <= COST_IN (folded_arg, mode_arg, code, i))
    3313              : 
    3314              :             /* It's not safe to substitute the operand of a conversion
    3315              :                operator with a constant, as the conversion's identity
    3316              :                depends upon the mode of its operand.  This optimization
    3317              :                is handled by the call to simplify_unary_operation.  */
    3318    247766314 :             && (GET_RTX_CLASS (code) != RTX_UNARY
    3319       401015 :                 || GET_MODE (const_arg) == mode_arg0
    3320       338341 :                 || (code != ZERO_EXTEND
    3321              :                     && code != SIGN_EXTEND
    3322       338341 :                     && code != TRUNCATE
    3323       338341 :                     && code != FLOAT_TRUNCATE
    3324       264606 :                     && code != FLOAT_EXTEND
    3325       264606 :                     && code != FLOAT
    3326              :                     && code != FIX
    3327       264413 :                     && code != UNSIGNED_FLOAT
    3328       264413 :                     && code != UNSIGNED_FIX)))
    3329              :           folded_arg = const_arg;
    3330              : 
    3331    246165092 :         if (folded_arg == XEXP (x, i))
    3332    244100243 :           continue;
    3333              : 
    3334      2064849 :         if (insn == NULL_RTX && !changed)
    3335      1871623 :           x = copy_rtx (x);
    3336      2064849 :         changed = true;
    3337      2064849 :         validate_unshare_change (insn, &XEXP (x, i), folded_arg, 1);
    3338              :       }
    3339              : 
    3340    118337966 :   if (changed)
    3341              :     {
    3342              :       /* Canonicalize X if necessary, and keep const_argN and folded_argN
    3343              :          consistent with the order in X.  */
    3344      1872122 :       if (canonicalize_change_group (insn, x))
    3345              :         {
    3346       104824 :           std::swap (const_arg0, const_arg1);
    3347       104824 :           std::swap (folded_arg0, folded_arg1);
    3348              :         }
    3349              : 
    3350      1872122 :       apply_change_group ();
    3351              :     }
    3352              : 
    3353              :   /* If X is an arithmetic operation, see if we can simplify it.  */
    3354              : 
    3355    118337966 :   switch (GET_RTX_CLASS (code))
    3356              :     {
    3357      6181248 :     case RTX_UNARY:
    3358      6181248 :       {
    3359              :         /* We can't simplify extension ops unless we know the
    3360              :            original mode.  */
    3361      6181248 :         if ((code == ZERO_EXTEND || code == SIGN_EXTEND)
    3362      4346758 :             && mode_arg0 == VOIDmode)
    3363              :           break;
    3364              : 
    3365      6181248 :         new_rtx = simplify_unary_operation (code, mode,
    3366              :                                             const_arg0 ? const_arg0 : folded_arg0,
    3367              :                                             mode_arg0);
    3368              :       }
    3369      6181248 :       break;
    3370              : 
    3371     22609917 :     case RTX_COMPARE:
    3372     22609917 :     case RTX_COMM_COMPARE:
    3373              :       /* See what items are actually being compared and set FOLDED_ARG[01]
    3374              :          to those values and CODE to the actual comparison code.  If any are
    3375              :          constant, set CONST_ARG0 and CONST_ARG1 appropriately.  We needn't
    3376              :          do anything if both operands are already known to be constant.  */
    3377              : 
    3378              :       /* ??? Vector mode comparisons are not supported yet.  */
    3379     22609917 :       if (VECTOR_MODE_P (mode))
    3380              :         break;
    3381              : 
    3382     22471121 :       if (const_arg0 == 0 || const_arg1 == 0)
    3383              :         {
    3384     22469976 :           struct table_elt *p0, *p1;
    3385     22469976 :           rtx true_rtx, false_rtx;
    3386     22469976 :           machine_mode mode_arg1;
    3387              : 
    3388     22469976 :           if (SCALAR_FLOAT_MODE_P (mode))
    3389              :             {
    3390              : #ifdef FLOAT_STORE_FLAG_VALUE
    3391              :               true_rtx = (const_double_from_real_value
    3392              :                           (FLOAT_STORE_FLAG_VALUE (mode), mode));
    3393              : #else
    3394         2557 :               true_rtx = NULL_RTX;
    3395              : #endif
    3396         2557 :               false_rtx = CONST0_RTX (mode);
    3397              :             }
    3398              :           else
    3399              :             {
    3400     22467419 :               true_rtx = const_true_rtx;
    3401     22467419 :               false_rtx = const0_rtx;
    3402              :             }
    3403              : 
    3404     22469976 :           code = find_comparison_args (code, &folded_arg0, &folded_arg1,
    3405              :                                        &mode_arg0, &mode_arg1);
    3406              : 
    3407              :           /* If the mode is VOIDmode or a MODE_CC mode, we don't know
    3408              :              what kinds of things are being compared, so we can't do
    3409              :              anything with this comparison.  */
    3410              : 
    3411     22469976 :           if (mode_arg0 == VOIDmode || GET_MODE_CLASS (mode_arg0) == MODE_CC)
    3412              :             break;
    3413              : 
    3414     21116239 :           const_arg0 = equiv_constant (folded_arg0);
    3415     21116239 :           const_arg1 = equiv_constant (folded_arg1);
    3416              : 
    3417              :           /* If we do not now have two constants being compared, see
    3418              :              if we can nevertheless deduce some things about the
    3419              :              comparison.  */
    3420     21116239 :           if (const_arg0 == 0 || const_arg1 == 0)
    3421              :             {
    3422     20911107 :               if (const_arg1 != NULL)
    3423              :                 {
    3424     15624820 :                   rtx cheapest_simplification;
    3425     15624820 :                   int cheapest_cost;
    3426     15624820 :                   rtx simp_result;
    3427     15624820 :                   struct table_elt *p;
    3428              : 
    3429              :                   /* See if we can find an equivalent of folded_arg0
    3430              :                      that gets us a cheaper expression, possibly a
    3431              :                      constant through simplifications.  */
    3432     15624820 :                   p = lookup (folded_arg0, SAFE_HASH (folded_arg0, mode_arg0),
    3433              :                               mode_arg0);
    3434              : 
    3435     15624820 :                   if (p != NULL)
    3436              :                     {
    3437      6431747 :                       cheapest_simplification = x;
    3438      6431747 :                       cheapest_cost = COST (x, mode);
    3439              : 
    3440     18650324 :                       for (p = p->first_same_value; p != NULL; p = p->next_same_value)
    3441              :                         {
    3442     12218577 :                           int cost;
    3443              : 
    3444              :                           /* If the entry isn't valid, skip it.  */
    3445     12218577 :                           if (! exp_equiv_p (p->exp, p->exp, 1, false))
    3446       521420 :                             continue;
    3447              : 
    3448              :                           /* Try to simplify using this equivalence.  */
    3449     11697157 :                           simp_result
    3450     11697157 :                             = simplify_relational_operation (code, mode,
    3451              :                                                              mode_arg0,
    3452              :                                                              p->exp,
    3453              :                                                              const_arg1);
    3454              : 
    3455     11697157 :                           if (simp_result == NULL)
    3456     11551070 :                             continue;
    3457              : 
    3458       146087 :                           cost = COST (simp_result, mode);
    3459       146087 :                           if (cost < cheapest_cost)
    3460              :                             {
    3461     12218577 :                               cheapest_cost = cost;
    3462     12218577 :                               cheapest_simplification = simp_result;
    3463              :                             }
    3464              :                         }
    3465              : 
    3466              :                       /* If we have a cheaper expression now, use that
    3467              :                          and try folding it further, from the top.  */
    3468      6431747 :                       if (cheapest_simplification != x)
    3469         1796 :                         return fold_rtx (copy_rtx (cheapest_simplification),
    3470        11243 :                                          insn);
    3471              :                     }
    3472              :                 }
    3473              : 
    3474              :               /* See if the two operands are the same.  */
    3475              : 
    3476     21103220 :               if ((REG_P (folded_arg0)
    3477     17646505 :                    && REG_P (folded_arg1)
    3478      4615763 :                    && (REG_QTY (REGNO (folded_arg0))
    3479      4615763 :                        == REG_QTY (REGNO (folded_arg1))))
    3480     38737758 :                   || ((p0 = lookup (folded_arg0,
    3481              :                                     SAFE_HASH (folded_arg0, mode_arg0),
    3482              :                                     mode_arg0))
    3483      8986425 :                       && (p1 = lookup (folded_arg1,
    3484              :                                        SAFE_HASH (folded_arg1, mode_arg0),
    3485              :                                        mode_arg0))
    3486      2507633 :                       && p0->first_same_value == p1->first_same_value))
    3487        12980 :                 folded_arg1 = folded_arg0;
    3488              : 
    3489              :               /* If FOLDED_ARG0 is a register, see if the comparison we are
    3490              :                  doing now is either the same as we did before or the reverse
    3491              :                  (we only check the reverse if not floating-point).  */
    3492     21090240 :               else if (REG_P (folded_arg0))
    3493              :                 {
    3494     17634146 :                   int qty = REG_QTY (REGNO (folded_arg0));
    3495              : 
    3496     17634146 :                   if (REGNO_QTY_VALID_P (REGNO (folded_arg0)))
    3497              :                     {
    3498     17623338 :                       struct qty_table_elem *ent = &qty_table[qty];
    3499              : 
    3500     17623338 :                       if ((comparison_dominates_p (ent->comparison_code, code)
    3501     17144629 :                            || (! FLOAT_MODE_P (mode_arg0)
    3502     16911362 :                                && comparison_dominates_p (ent->comparison_code,
    3503              :                                                           reverse_condition (code))))
    3504     18073554 :                           && (rtx_equal_p (ent->comparison_const, folded_arg1)
    3505       921272 :                               || (const_arg1
    3506       783394 :                                   && rtx_equal_p (ent->comparison_const,
    3507              :                                                   const_arg1))
    3508       921272 :                               || (REG_P (folded_arg1)
    3509       129050 :                                   && (REG_QTY (REGNO (folded_arg1)) == ent->comparison_qty))))
    3510              :                         {
    3511         9447 :                           if (comparison_dominates_p (ent->comparison_code, code))
    3512              :                             {
    3513         6740 :                               if (true_rtx)
    3514              :                                 return true_rtx;
    3515              :                               else
    3516              :                                 break;
    3517              :                             }
    3518              :                           else
    3519              :                             return false_rtx;
    3520              :                         }
    3521              :                     }
    3522              :                 }
    3523              :             }
    3524              :         }
    3525              : 
    3526              :       /* If we are comparing against zero, see if the first operand is
    3527              :          equivalent to an IOR with a constant.  If so, we may be able to
    3528              :          determine the result of this comparison.  */
    3529     21106141 :       if (const_arg1 == const0_rtx && !const_arg0)
    3530              :         {
    3531      9945449 :           rtx y = lookup_as_function (folded_arg0, IOR);
    3532      9945449 :           rtx inner_const;
    3533              : 
    3534      9945449 :           if (y != 0
    3535        78080 :               && (inner_const = equiv_constant (XEXP (y, 1))) != 0
    3536           53 :               && CONST_INT_P (inner_const)
    3537      9945502 :               && INTVAL (inner_const) != 0)
    3538           53 :             folded_arg0 = gen_rtx_IOR (mode_arg0, XEXP (y, 0), inner_const);
    3539              :         }
    3540              : 
    3541     21098578 :       {
    3542     21098578 :         rtx op0 = const_arg0 ? const_arg0 : copy_rtx (folded_arg0);
    3543     21106141 :         rtx op1 = const_arg1 ? const_arg1 : copy_rtx (folded_arg1);
    3544     21106141 :         new_rtx = simplify_relational_operation (code, mode, mode_arg0,
    3545              :                                                  op0, op1);
    3546              :       }
    3547     21106141 :       break;
    3548              : 
    3549     65041611 :     case RTX_BIN_ARITH:
    3550     65041611 :     case RTX_COMM_ARITH:
    3551     65041611 :       switch (code)
    3552              :         {
    3553     27786712 :         case PLUS:
    3554              :           /* If the second operand is a LABEL_REF, see if the first is a MINUS
    3555              :              with that LABEL_REF as its second operand.  If so, the result is
    3556              :              the first operand of that MINUS.  This handles switches with an
    3557              :              ADDR_DIFF_VEC table.  */
    3558     27786712 :           if (const_arg1 && GET_CODE (const_arg1) == LABEL_REF)
    3559              :             {
    3560         3035 :               rtx y
    3561         3035 :                 = GET_CODE (folded_arg0) == MINUS ? folded_arg0
    3562         3035 :                 : lookup_as_function (folded_arg0, MINUS);
    3563              : 
    3564            0 :               if (y != 0 && GET_CODE (XEXP (y, 1)) == LABEL_REF
    3565         3035 :                   && label_ref_label (XEXP (y, 1)) == label_ref_label (const_arg1))
    3566            0 :                 return XEXP (y, 0);
    3567              : 
    3568              :               /* Now try for a CONST of a MINUS like the above.  */
    3569         3035 :               if ((y = (GET_CODE (folded_arg0) == CONST ? folded_arg0
    3570         3035 :                         : lookup_as_function (folded_arg0, CONST))) != 0
    3571            0 :                   && GET_CODE (XEXP (y, 0)) == MINUS
    3572            0 :                   && GET_CODE (XEXP (XEXP (y, 0), 1)) == LABEL_REF
    3573         3035 :                   && label_ref_label (XEXP (XEXP (y, 0), 1)) == label_ref_label (const_arg1))
    3574            0 :                 return XEXP (XEXP (y, 0), 0);
    3575              :             }
    3576              : 
    3577              :           /* Likewise if the operands are in the other order.  */
    3578     27786712 :           if (const_arg0 && GET_CODE (const_arg0) == LABEL_REF)
    3579              :             {
    3580           23 :               rtx y
    3581           23 :                 = GET_CODE (folded_arg1) == MINUS ? folded_arg1
    3582           23 :                 : lookup_as_function (folded_arg1, MINUS);
    3583              : 
    3584            0 :               if (y != 0 && GET_CODE (XEXP (y, 1)) == LABEL_REF
    3585           23 :                   && label_ref_label (XEXP (y, 1)) == label_ref_label (const_arg0))
    3586            0 :                 return XEXP (y, 0);
    3587              : 
    3588              :               /* Now try for a CONST of a MINUS like the above.  */
    3589           23 :               if ((y = (GET_CODE (folded_arg1) == CONST ? folded_arg1
    3590           23 :                         : lookup_as_function (folded_arg1, CONST))) != 0
    3591            0 :                   && GET_CODE (XEXP (y, 0)) == MINUS
    3592            0 :                   && GET_CODE (XEXP (XEXP (y, 0), 1)) == LABEL_REF
    3593           23 :                   && label_ref_label (XEXP (XEXP (y, 0), 1)) == label_ref_label (const_arg0))
    3594            0 :                 return XEXP (XEXP (y, 0), 0);
    3595              :             }
    3596              : 
    3597              :           /* If second operand is a register equivalent to a negative
    3598              :              CONST_INT, see if we can find a register equivalent to the
    3599              :              positive constant.  Make a MINUS if so.  Don't do this for
    3600              :              a non-negative constant since we might then alternate between
    3601              :              choosing positive and negative constants.  Having the positive
    3602              :              constant previously-used is the more common case.  Be sure
    3603              :              the resulting constant is non-negative; if const_arg1 were
    3604              :              the smallest negative number this would overflow: depending
    3605              :              on the mode, this would either just be the same value (and
    3606              :              hence not save anything) or be incorrect.  */
    3607     27786712 :           if (const_arg1 != 0 && CONST_INT_P (const_arg1)
    3608     22166211 :               && INTVAL (const_arg1) < 0
    3609              :               /* This used to test
    3610              : 
    3611              :                  -INTVAL (const_arg1) >= 0
    3612              : 
    3613              :                  But The Sun V5.0 compilers mis-compiled that test.  So
    3614              :                  instead we test for the problematic value in a more direct
    3615              :                  manner and hope the Sun compilers get it correct.  */
    3616     12573621 :               && INTVAL (const_arg1) !=
    3617              :                 (HOST_WIDE_INT_1 << (HOST_BITS_PER_WIDE_INT - 1))
    3618     12554446 :               && REG_P (folded_arg1))
    3619              :             {
    3620        32528 :               rtx new_const = GEN_INT (-INTVAL (const_arg1));
    3621        32528 :               struct table_elt *p
    3622        32528 :                 = lookup (new_const, SAFE_HASH (new_const, mode), mode);
    3623              : 
    3624        32528 :               if (p)
    3625         5137 :                 for (p = p->first_same_value; p; p = p->next_same_value)
    3626         5133 :                   if (REG_P (p->exp))
    3627         2690 :                     return simplify_gen_binary (MINUS, mode, folded_arg0,
    3628         2690 :                                                 canon_reg (p->exp, NULL));
    3629              :             }
    3630     27784022 :           goto from_plus;
    3631              : 
    3632      2314281 :         case MINUS:
    3633              :           /* If we have (MINUS Y C), see if Y is known to be (PLUS Z C2).
    3634              :              If so, produce (PLUS Z C2-C).  */
    3635      2314281 :           if (const_arg1 != 0 && poly_int_rtx_p (const_arg1, &xval))
    3636              :             {
    3637        43412 :               rtx y = lookup_as_function (XEXP (x, 0), PLUS);
    3638        43412 :               if (y && poly_int_rtx_p (XEXP (y, 1)))
    3639           35 :                 return fold_rtx (plus_constant (mode, copy_rtx (y), -xval),
    3640           35 :                                  NULL);
    3641              :             }
    3642              : 
    3643              :           /* Fall through.  */
    3644              : 
    3645     40709884 :         from_plus:
    3646     40709884 :         case SMIN:    case SMAX:      case UMIN:    case UMAX:
    3647     40709884 :         case IOR:     case AND:       case XOR:
    3648     40709884 :         case MULT:
    3649     40709884 :         case ASHIFT:  case LSHIFTRT:  case ASHIFTRT:
    3650              :           /* If we have (<op> <reg> <const_int>) for an associative OP and REG
    3651              :              is known to be of similar form, we may be able to replace the
    3652              :              operation with a combined operation.  This may eliminate the
    3653              :              intermediate operation if every use is simplified in this way.
    3654              :              Note that the similar optimization done by combine.cc only works
    3655              :              if the intermediate operation's result has only one reference.  */
    3656              : 
    3657     40709884 :           if (REG_P (folded_arg0)
    3658     37341254 :               && const_arg1 && CONST_INT_P (const_arg1))
    3659              :             {
    3660     27156254 :               int is_shift
    3661     27156254 :                 = (code == ASHIFT || code == ASHIFTRT || code == LSHIFTRT);
    3662              :               rtx y, inner_const, new_const;
    3663              :               rtx canon_const_arg1 = const_arg1;
    3664              :               enum rtx_code associate_code;
    3665              : 
    3666              :               if (is_shift
    3667      6771500 :                   && (INTVAL (const_arg1) >= GET_MODE_UNIT_PRECISION (mode)
    3668      3385583 :                       || INTVAL (const_arg1) < 0))
    3669              :                 {
    3670              :                   if (SHIFT_COUNT_TRUNCATED)
    3671              :                     canon_const_arg1 = gen_int_shift_amount
    3672              :                       (mode, (INTVAL (const_arg1)
    3673              :                               & (GET_MODE_UNIT_BITSIZE (mode) - 1)));
    3674              :                   else
    3675              :                     break;
    3676              :                 }
    3677              : 
    3678     27156085 :               y = lookup_as_function (folded_arg0, code);
    3679     27156085 :               if (y == 0)
    3680              :                 break;
    3681              : 
    3682              :               /* If we have compiled a statement like
    3683              :                  "if (x == (x & mask1))", and now are looking at
    3684              :                  "x & mask2", we will have a case where the first operand
    3685              :                  of Y is the same as our first operand.  Unless we detect
    3686              :                  this case, an infinite loop will result.  */
    3687       734560 :               if (XEXP (y, 0) == folded_arg0)
    3688              :                 break;
    3689              : 
    3690       734249 :               inner_const = equiv_constant (fold_rtx (XEXP (y, 1), 0));
    3691       734249 :               if (!inner_const || !CONST_INT_P (inner_const))
    3692              :                 break;
    3693              : 
    3694              :               /* Don't associate these operations if they are a PLUS with the
    3695              :                  same constant and it is a power of two.  These might be doable
    3696              :                  with a pre- or post-increment.  Similarly for two subtracts of
    3697              :                  identical powers of two with post decrement.  */
    3698              : 
    3699       534602 :               if (code == PLUS && const_arg1 == inner_const
    3700              :                   && ((HAVE_PRE_INCREMENT
    3701              :                           && pow2p_hwi (INTVAL (const_arg1)))
    3702              :                       || (HAVE_POST_INCREMENT
    3703              :                           && pow2p_hwi (INTVAL (const_arg1)))
    3704              :                       || (HAVE_PRE_DECREMENT
    3705              :                           && pow2p_hwi (- INTVAL (const_arg1)))
    3706              :                       || (HAVE_POST_DECREMENT
    3707              :                           && pow2p_hwi (- INTVAL (const_arg1)))))
    3708              :                 break;
    3709              : 
    3710              :               /* ??? Vector mode shifts by scalar
    3711              :                  shift operand are not supported yet.  */
    3712       534602 :               if (is_shift && VECTOR_MODE_P (mode))
    3713              :                 break;
    3714              : 
    3715         4167 :               if (is_shift
    3716         8334 :                   && (INTVAL (inner_const) >= GET_MODE_UNIT_PRECISION (mode)
    3717         4167 :                       || INTVAL (inner_const) < 0))
    3718              :                 {
    3719              :                   if (SHIFT_COUNT_TRUNCATED)
    3720              :                     inner_const = gen_int_shift_amount
    3721              :                       (mode, (INTVAL (inner_const)
    3722              :                               & (GET_MODE_UNIT_BITSIZE (mode) - 1)));
    3723              :                   else
    3724              :                     break;
    3725              :                 }
    3726              : 
    3727              :               /* Compute the code used to compose the constants.  For example,
    3728              :                  A-C1-C2 is A-(C1 + C2), so if CODE == MINUS, we want PLUS.  */
    3729              : 
    3730       534337 :               associate_code = (is_shift || code == MINUS ? PLUS : code);
    3731              : 
    3732       534337 :               new_const = simplify_binary_operation (associate_code, mode,
    3733              :                                                      canon_const_arg1,
    3734              :                                                      inner_const);
    3735              : 
    3736       534337 :               if (new_const == 0)
    3737              :                 break;
    3738              : 
    3739              :               /* If we are associating shift operations, don't let this
    3740              :                  produce a shift of the size of the object or larger.
    3741              :                  This could occur when we follow a sign-extend by a right
    3742              :                  shift on a machine that does a sign-extend as a pair
    3743              :                  of shifts.  */
    3744              : 
    3745       534337 :               if (is_shift
    3746         4167 :                   && CONST_INT_P (new_const)
    3747       542671 :                   && INTVAL (new_const) >= GET_MODE_UNIT_PRECISION (mode))
    3748              :                 {
    3749              :                   /* As an exception, we can turn an ASHIFTRT of this
    3750              :                      form into a shift of the number of bits - 1.  */
    3751         1557 :                   if (code == ASHIFTRT)
    3752         1534 :                     new_const = gen_int_shift_amount
    3753         1534 :                       (mode, GET_MODE_UNIT_BITSIZE (mode) - 1);
    3754           23 :                   else if (!side_effects_p (XEXP (y, 0)))
    3755           23 :                     return CONST0_RTX (mode);
    3756              :                   else
    3757              :                     break;
    3758              :                 }
    3759              : 
    3760       534314 :               y = copy_rtx (XEXP (y, 0));
    3761              : 
    3762              :               /* If Y contains our first operand (the most common way this
    3763              :                  can happen is if Y is a MEM), we would do into an infinite
    3764              :                  loop if we tried to fold it.  So don't in that case.  */
    3765              : 
    3766       534314 :               if (! reg_mentioned_p (folded_arg0, y))
    3767       534314 :                 y = fold_rtx (y, insn);
    3768              : 
    3769       534314 :               return simplify_gen_binary (code, mode, y, new_const);
    3770              :             }
    3771              :           break;
    3772              : 
    3773              :         case DIV:       case UDIV:
    3774              :           /* ??? The associative optimization performed immediately above is
    3775              :              also possible for DIV and UDIV using associate_code of MULT.
    3776              :              However, we would need extra code to verify that the
    3777              :              multiplication does not overflow, that is, there is no overflow
    3778              :              in the calculation of new_const.  */
    3779              :           break;
    3780              : 
    3781              :         default:
    3782              :           break;
    3783              :         }
    3784              : 
    3785     64504549 :       new_rtx = simplify_binary_operation (code, mode,
    3786              :                                        const_arg0 ? const_arg0 : folded_arg0,
    3787              :                                        const_arg1 ? const_arg1 : folded_arg1);
    3788     64504549 :       break;
    3789              : 
    3790            0 :     case RTX_OBJ:
    3791              :       /* (lo_sum (high X) X) is simply X.  */
    3792            0 :       if (code == LO_SUM && const_arg0 != 0
    3793            0 :           && GET_CODE (const_arg0) == HIGH
    3794            0 :           && rtx_equal_p (XEXP (const_arg0, 0), const_arg1))
    3795              :         return const_arg1;
    3796              :       break;
    3797              : 
    3798     21215544 :     case RTX_TERNARY:
    3799     21215544 :     case RTX_BITFIELD_OPS:
    3800     21215544 :       new_rtx = simplify_ternary_operation (code, mode, mode_arg0,
    3801              :                                         const_arg0 ? const_arg0 : folded_arg0,
    3802              :                                         const_arg1 ? const_arg1 : folded_arg1,
    3803              :                                         const_arg2 ? const_arg2 : XEXP (x, 2));
    3804     21215544 :       break;
    3805              : 
    3806              :     default:
    3807              :       break;
    3808              :     }
    3809              : 
    3810    114361219 :   return new_rtx ? new_rtx : x;
    3811              : }
    3812              : 
    3813              : /* Return a constant value currently equivalent to X.
    3814              :    Return 0 if we don't know one.  */
    3815              : 
    3816              : static rtx
    3817    276739385 : equiv_constant (rtx x)
    3818              : {
    3819    276739385 :   if (REG_P (x)
    3820    276739385 :       && REGNO_QTY_VALID_P (REGNO (x)))
    3821              :     {
    3822     88063118 :       int x_q = REG_QTY (REGNO (x));
    3823     88063118 :       struct qty_table_elem *x_ent = &qty_table[x_q];
    3824              : 
    3825     88063118 :       if (x_ent->const_rtx)
    3826      4517447 :         x = gen_lowpart (GET_MODE (x), x_ent->const_rtx);
    3827              :     }
    3828              : 
    3829    276739385 :   if (x == 0 || CONSTANT_P (x))
    3830              :     return x;
    3831              : 
    3832    249662595 :   if (GET_CODE (x) == SUBREG)
    3833              :     {
    3834      6419187 :       machine_mode mode = GET_MODE (x);
    3835      6419187 :       machine_mode imode = GET_MODE (SUBREG_REG (x));
    3836      6419187 :       rtx new_rtx;
    3837              : 
    3838              :       /* See if we previously assigned a constant value to this SUBREG.  */
    3839      6419187 :       if ((new_rtx = lookup_as_function (x, CONST_INT)) != 0
    3840      6408661 :           || (new_rtx = lookup_as_function (x, CONST_WIDE_INT)) != 0
    3841      6408661 :           || (NUM_POLY_INT_COEFFS > 1
    3842              :               && (new_rtx = lookup_as_function (x, CONST_POLY_INT)) != 0)
    3843      6402745 :           || (new_rtx = lookup_as_function (x, CONST_DOUBLE)) != 0
    3844     12821742 :           || (new_rtx = lookup_as_function (x, CONST_FIXED)) != 0)
    3845              :         return new_rtx;
    3846              : 
    3847              :       /* If we didn't and if doing so makes sense, see if we previously
    3848              :          assigned a constant value to the enclosing word mode SUBREG.  */
    3849     13685700 :       if (known_lt (GET_MODE_SIZE (mode), UNITS_PER_WORD)
    3850     10048424 :           && known_lt (UNITS_PER_WORD, GET_MODE_SIZE (imode)))
    3851              :         {
    3852        34190 :           poly_int64 byte = (SUBREG_BYTE (x)
    3853        34190 :                              - subreg_lowpart_offset (mode, word_mode));
    3854        68380 :           if (known_ge (byte, 0) && multiple_p (byte, UNITS_PER_WORD))
    3855              :             {
    3856        34190 :               rtx y = gen_rtx_SUBREG (word_mode, SUBREG_REG (x), byte);
    3857        34190 :               new_rtx = lookup_as_function (y, CONST_INT);
    3858        34190 :               if (new_rtx)
    3859            0 :                 return gen_lowpart (mode, new_rtx);
    3860              :             }
    3861              :         }
    3862              : 
    3863              :       /* Otherwise see if we already have a constant for the inner REG,
    3864              :          and if that is enough to calculate an equivalent constant for
    3865              :          the subreg.  Note that the upper bits of paradoxical subregs
    3866              :          are undefined, so they cannot be said to equal anything.  */
    3867      6402555 :       if (REG_P (SUBREG_REG (x))
    3868      6393076 :           && !paradoxical_subreg_p (x)
    3869     12605211 :           && (new_rtx = equiv_constant (SUBREG_REG (x))) != 0)
    3870       104655 :         return simplify_subreg (mode, new_rtx, imode, SUBREG_BYTE (x));
    3871              : 
    3872              :       return 0;
    3873              :     }
    3874              : 
    3875              :   /* If X is a MEM, see if it is a constant-pool reference, or look it up in
    3876              :      the hash table in case its value was seen before.  */
    3877              : 
    3878    243243408 :   if (MEM_P (x))
    3879              :     {
    3880     69973340 :       struct table_elt *elt;
    3881              : 
    3882     69973340 :       x = avoid_constant_pool_reference (x);
    3883     69973340 :       if (CONSTANT_P (x))
    3884              :         return x;
    3885              : 
    3886     67795163 :       elt = lookup (x, SAFE_HASH (x, GET_MODE (x)), GET_MODE (x));
    3887     67795163 :       if (elt == 0)
    3888              :         return 0;
    3889              : 
    3890     11936272 :       for (elt = elt->first_same_value; elt; elt = elt->next_same_value)
    3891     10199138 :         if (elt->is_const && CONSTANT_P (elt->exp))
    3892              :           return elt->exp;
    3893              :     }
    3894              : 
    3895              :   return 0;
    3896              : }
    3897              : 
    3898              : /* Given INSN, a jump insn, TAKEN indicates if we are following the
    3899              :    "taken" branch.
    3900              : 
    3901              :    In certain cases, this can cause us to add an equivalence.  For example,
    3902              :    if we are following the taken case of
    3903              :         if (i == 2)
    3904              :    we can add the fact that `i' and '2' are now equivalent.
    3905              : 
    3906              :    In any case, we can record that this comparison was passed.  If the same
    3907              :    comparison is seen later, we will know its value.  */
    3908              : 
    3909              : static void
    3910     14913122 : record_jump_equiv (rtx_insn *insn, bool taken)
    3911              : {
    3912     14913122 :   int cond_known_true;
    3913     14913122 :   rtx op0, op1;
    3914     14913122 :   rtx set;
    3915     14913122 :   machine_mode mode, mode0, mode1;
    3916     14913122 :   enum rtx_code code;
    3917              : 
    3918              :   /* Ensure this is the right kind of insn.  */
    3919     14913122 :   gcc_assert (any_condjump_p (insn));
    3920              : 
    3921     14913122 :   set = pc_set (insn);
    3922              : 
    3923              :   /* See if this jump condition is known true or false.  */
    3924     14913122 :   if (taken)
    3925      5952591 :     cond_known_true = (XEXP (SET_SRC (set), 2) == pc_rtx);
    3926              :   else
    3927      8960531 :     cond_known_true = (XEXP (SET_SRC (set), 1) == pc_rtx);
    3928              : 
    3929              :   /* Get the type of comparison being done and the operands being compared.
    3930              :      If we had to reverse a non-equality condition, record that fact so we
    3931              :      know that it isn't valid for floating-point.  */
    3932     14913122 :   code = GET_CODE (XEXP (SET_SRC (set), 0));
    3933     14913122 :   op0 = fold_rtx (XEXP (XEXP (SET_SRC (set), 0), 0), insn);
    3934     14913122 :   op1 = fold_rtx (XEXP (XEXP (SET_SRC (set), 0), 1), insn);
    3935              : 
    3936              :   /* If fold_rtx returns NULL_RTX, there's nothing to record.  */
    3937     14913122 :   if (op0 == NULL_RTX || op1 == NULL_RTX)
    3938        92285 :     return;
    3939              : 
    3940     14913122 :   code = find_comparison_args (code, &op0, &op1, &mode0, &mode1);
    3941     14913122 :   if (! cond_known_true)
    3942              :     {
    3943      8960531 :       code = reversed_comparison_code_parts (code, op0, op1, insn);
    3944              : 
    3945              :       /* Don't remember if we can't find the inverse.  */
    3946      8960531 :       if (code == UNKNOWN)
    3947              :         return;
    3948              :     }
    3949              : 
    3950              :   /* The mode is the mode of the non-constant.  */
    3951     14820837 :   mode = mode0;
    3952     14820837 :   if (mode1 != VOIDmode)
    3953      3733454 :     mode = mode1;
    3954              : 
    3955     14820837 :   record_jump_cond (code, mode, op0, op1);
    3956              : }
    3957              : 
    3958              : /* Yet another form of subreg creation.  In this case, we want something in
    3959              :    MODE, and we should assume OP has MODE iff it is naturally modeless.  */
    3960              : 
    3961              : static rtx
    3962        58682 : record_jump_cond_subreg (machine_mode mode, rtx op)
    3963              : {
    3964        58682 :   machine_mode op_mode = GET_MODE (op);
    3965        58682 :   if (op_mode == mode || op_mode == VOIDmode)
    3966              :     return op;
    3967         6169 :   return lowpart_subreg (mode, op, op_mode);
    3968              : }
    3969              : 
    3970              : /* We know that comparison CODE applied to OP0 and OP1 in MODE is true.
    3971              :    Make any useful entries we can with that information.  Called from
    3972              :    above function and called recursively.  */
    3973              : 
    3974              : static void
    3975     14879515 : record_jump_cond (enum rtx_code code, machine_mode mode, rtx op0, rtx op1)
    3976              : {
    3977     14879515 :   unsigned op0_hash, op1_hash;
    3978     14879515 :   int op0_in_memory, op1_in_memory;
    3979     14879515 :   struct table_elt *op0_elt, *op1_elt;
    3980              : 
    3981              :   /* If OP0 and OP1 are known equal, and either is a paradoxical SUBREG,
    3982              :      we know that they are also equal in the smaller mode (this is also
    3983              :      true for all smaller modes whether or not there is a SUBREG, but
    3984              :      is not worth testing for with no SUBREG).  */
    3985              : 
    3986              :   /* Note that GET_MODE (op0) may not equal MODE.  */
    3987     14928002 :   if (code == EQ && paradoxical_subreg_p (op0))
    3988              :     {
    3989            0 :       machine_mode inner_mode = GET_MODE (SUBREG_REG (op0));
    3990            0 :       rtx tem = record_jump_cond_subreg (inner_mode, op1);
    3991            0 :       if (tem)
    3992            0 :         record_jump_cond (code, mode, SUBREG_REG (op0), tem);
    3993              :     }
    3994              : 
    3995     14894218 :   if (code == EQ && paradoxical_subreg_p (op1))
    3996              :     {
    3997            0 :       machine_mode inner_mode = GET_MODE (SUBREG_REG (op1));
    3998            0 :       rtx tem = record_jump_cond_subreg (inner_mode, op0);
    3999            0 :       if (tem)
    4000            0 :         record_jump_cond (code, mode, SUBREG_REG (op1), tem);
    4001              :     }
    4002              : 
    4003              :   /* Similarly, if this is an NE comparison, and either is a SUBREG
    4004              :      making a smaller mode, we know the whole thing is also NE.  */
    4005              : 
    4006              :   /* Note that GET_MODE (op0) may not equal MODE;
    4007              :      if we test MODE instead, we can get an infinite recursion
    4008              :      alternating between two modes each wider than MODE.  */
    4009              : 
    4010     14879515 :   if (code == NE
    4011        53739 :       && partial_subreg_p (op0)
    4012     14932998 :       && subreg_lowpart_p (op0))
    4013              :     {
    4014        53119 :       machine_mode inner_mode = GET_MODE (SUBREG_REG (op0));
    4015        53119 :       rtx tem = record_jump_cond_subreg (inner_mode, op1);
    4016        53119 :       if (tem)
    4017        53119 :         record_jump_cond (code, mode, SUBREG_REG (op0), tem);
    4018              :     }
    4019              : 
    4020     14879515 :   if (code == NE
    4021        11836 :       && partial_subreg_p (op1)
    4022     14885134 :       && subreg_lowpart_p (op1))
    4023              :     {
    4024         5563 :       machine_mode inner_mode = GET_MODE (SUBREG_REG (op1));
    4025         5563 :       rtx tem = record_jump_cond_subreg (inner_mode, op0);
    4026         5563 :       if (tem)
    4027         5559 :         record_jump_cond (code, mode, SUBREG_REG (op1), tem);
    4028              :     }
    4029              : 
    4030              :   /* Hash both operands.  */
    4031              : 
    4032     14879515 :   do_not_record = 0;
    4033     14879515 :   hash_arg_in_memory = 0;
    4034     14879515 :   op0_hash = HASH (op0, mode);
    4035     14879515 :   op0_in_memory = hash_arg_in_memory;
    4036              : 
    4037     14879515 :   if (do_not_record)
    4038              :     return;
    4039              : 
    4040     14879515 :   do_not_record = 0;
    4041     14879515 :   hash_arg_in_memory = 0;
    4042     14879515 :   op1_hash = HASH (op1, mode);
    4043     14879515 :   op1_in_memory = hash_arg_in_memory;
    4044              : 
    4045     14879515 :   if (do_not_record)
    4046              :     return;
    4047              : 
    4048              :   /* Look up both operands.  */
    4049     14879515 :   op0_elt = lookup (op0, op0_hash, mode);
    4050     14879515 :   op1_elt = lookup (op1, op1_hash, mode);
    4051              : 
    4052              :   /* If both operands are already equivalent or if they are not in the
    4053              :      table but are identical, do nothing.  */
    4054     14879515 :   if ((op0_elt != 0 && op1_elt != 0
    4055      1868712 :        && op0_elt->first_same_value == op1_elt->first_same_value)
    4056     16748142 :       || op0 == op1 || rtx_equal_p (op0, op1))
    4057              :     return;
    4058              : 
    4059              :   /* If we aren't setting two things equal all we can do is save this
    4060              :      comparison.   Similarly if this is floating-point.  In the latter
    4061              :      case, OP1 might be zero and both -0.0 and 0.0 are equal to it.
    4062              :      If we record the equality, we might inadvertently delete code
    4063              :      whose intent was to change -0 to +0.  */
    4064              : 
    4065     14878666 :   if (code != EQ || FLOAT_MODE_P (GET_MODE (op0)))
    4066              :     {
    4067      9533021 :       struct qty_table_elem *ent;
    4068      9533021 :       int qty;
    4069              : 
    4070              :       /* If OP0 is not a register, or if OP1 is neither a register
    4071              :          or constant, we can't do anything.  */
    4072              : 
    4073      9533021 :       if (!REG_P (op1))
    4074      7597820 :         op1 = equiv_constant (op1);
    4075              : 
    4076      9533021 :       if (!REG_P (op0) || op1 == 0)
    4077              :         return;
    4078              : 
    4079              :       /* Put OP0 in the hash table if it isn't already.  This gives it a
    4080              :          new quantity number.  */
    4081      8119558 :       if (op0_elt == 0)
    4082              :         {
    4083      3333898 :           if (insert_regs (op0, NULL, false))
    4084              :             {
    4085        58498 :               rehash_using_reg (op0);
    4086        58498 :               op0_hash = HASH (op0, mode);
    4087              : 
    4088              :               /* If OP0 is contained in OP1, this changes its hash code
    4089              :                  as well.  Faster to rehash than to check, except
    4090              :                  for the simple case of a constant.  */
    4091        58498 :               if (! CONSTANT_P (op1))
    4092          230 :                 op1_hash = HASH (op1,mode);
    4093              :             }
    4094              : 
    4095      3333898 :           op0_elt = insert (op0, NULL, op0_hash, mode);
    4096      3333898 :           op0_elt->in_memory = op0_in_memory;
    4097              :         }
    4098              : 
    4099      8119558 :       qty = REG_QTY (REGNO (op0));
    4100      8119558 :       ent = &qty_table[qty];
    4101              : 
    4102      8119558 :       ent->comparison_code = code;
    4103      8119558 :       if (REG_P (op1))
    4104              :         {
    4105              :           /* Look it up again--in case op0 and op1 are the same.  */
    4106      1842148 :           op1_elt = lookup (op1, op1_hash, mode);
    4107              : 
    4108              :           /* Put OP1 in the hash table so it gets a new quantity number.  */
    4109      1842148 :           if (op1_elt == 0)
    4110              :             {
    4111       684431 :               if (insert_regs (op1, NULL, false))
    4112              :                 {
    4113          218 :                   rehash_using_reg (op1);
    4114          218 :                   op1_hash = HASH (op1, mode);
    4115              :                 }
    4116              : 
    4117       684431 :               op1_elt = insert (op1, NULL, op1_hash, mode);
    4118       684431 :               op1_elt->in_memory = op1_in_memory;
    4119              :             }
    4120              : 
    4121      1842148 :           ent->comparison_const = NULL_RTX;
    4122      1842148 :           ent->comparison_qty = REG_QTY (REGNO (op1));
    4123              :         }
    4124              :       else
    4125              :         {
    4126      6277410 :           ent->comparison_const = op1;
    4127      6277410 :           ent->comparison_qty = INT_MIN;
    4128              :         }
    4129              : 
    4130              :       return;
    4131              :     }
    4132              : 
    4133              :   /* If either side is still missing an equivalence, make it now,
    4134              :      then merge the equivalences.  */
    4135              : 
    4136      5345645 :   if (op0_elt == 0)
    4137              :     {
    4138      3320601 :       if (insert_regs (op0, NULL, false))
    4139              :         {
    4140        19942 :           rehash_using_reg (op0);
    4141        19942 :           op0_hash = HASH (op0, mode);
    4142              :         }
    4143              : 
    4144      3320601 :       op0_elt = insert (op0, NULL, op0_hash, mode);
    4145      3320601 :       op0_elt->in_memory = op0_in_memory;
    4146              :     }
    4147              : 
    4148      5345645 :   if (op1_elt == 0)
    4149              :     {
    4150      4031830 :       if (insert_regs (op1, NULL, false))
    4151              :         {
    4152         8537 :           rehash_using_reg (op1);
    4153         8537 :           op1_hash = HASH (op1, mode);
    4154              :         }
    4155              : 
    4156      4031830 :       op1_elt = insert (op1, NULL, op1_hash, mode);
    4157      4031830 :       op1_elt->in_memory = op1_in_memory;
    4158              :     }
    4159              : 
    4160      5345645 :   merge_equiv_classes (op0_elt, op1_elt);
    4161              : }
    4162              : 
    4163              : /* CSE processing for one instruction.
    4164              : 
    4165              :    Most "true" common subexpressions are mostly optimized away in GIMPLE,
    4166              :    but the few that "leak through" are cleaned up by cse_insn, and complex
    4167              :    addressing modes are often formed here.
    4168              : 
    4169              :    The main function is cse_insn, and between here and that function
    4170              :    a couple of helper functions is defined to keep the size of cse_insn
    4171              :    within reasonable proportions.
    4172              : 
    4173              :    Data is shared between the main and helper functions via STRUCT SET,
    4174              :    that contains all data related for every set in the instruction that
    4175              :    is being processed.
    4176              : 
    4177              :    Note that cse_main processes all sets in the instruction.  Most
    4178              :    passes in GCC only process simple SET insns or single_set insns, but
    4179              :    CSE processes insns with multiple sets as well.  */
    4180              : 
    4181              : /* Data on one SET contained in the instruction.  */
    4182              : 
    4183              : struct set
    4184              : {
    4185              :   /* The SET rtx itself.  */
    4186              :   rtx rtl;
    4187              :   /* The SET_SRC of the rtx (the original value, if it is changing).  */
    4188              :   rtx src;
    4189              :   /* The hash-table element for the SET_SRC of the SET.  */
    4190              :   struct table_elt *src_elt;
    4191              :   /* Hash value for the SET_SRC.  */
    4192              :   unsigned src_hash;
    4193              :   /* Hash value for the SET_DEST.  */
    4194              :   unsigned dest_hash;
    4195              :   /* The SET_DEST, with SUBREG, etc., stripped.  */
    4196              :   rtx inner_dest;
    4197              :   /* Original machine mode, in case it becomes a CONST_INT.  */
    4198              :   machine_mode mode : MACHINE_MODE_BITSIZE;
    4199              :   /* Nonzero if the SET_SRC is in memory.  */
    4200              :   unsigned int src_in_memory : 1;
    4201              :   /* Nonzero if the SET_SRC contains something
    4202              :      whose value cannot be predicted and understood.  */
    4203              :   unsigned int src_volatile : 1;
    4204              :   /* Nonzero if RTL is an artificial set that has been created to describe
    4205              :      part of an insn's effect.  Zero means that RTL appears directly in
    4206              :      the insn pattern.  */
    4207              :   unsigned int is_fake_set : 1;
    4208              :   /* Hash value of constant equivalent for SET_SRC.  */
    4209              :   unsigned src_const_hash;
    4210              :   /* A constant equivalent for SET_SRC, if any.  */
    4211              :   rtx src_const;
    4212              :   /* Table entry for constant equivalent for SET_SRC, if any.  */
    4213              :   struct table_elt *src_const_elt;
    4214              :   /* Table entry for the destination address.  */
    4215              :   struct table_elt *dest_addr_elt;
    4216              : };
    4217              : 
    4218              : /* Special handling for (set REG0 REG1) where REG0 is the
    4219              :    "cheapest", cheaper than REG1.  After cse, REG1 will probably not
    4220              :    be used in the sequel, so (if easily done) change this insn to
    4221              :    (set REG1 REG0) and replace REG1 with REG0 in the previous insn
    4222              :    that computed their value.  Then REG1 will become a dead store
    4223              :    and won't cloud the situation for later optimizations.
    4224              : 
    4225              :    Do not make this change if REG1 is a hard register, because it will
    4226              :    then be used in the sequel and we may be changing a two-operand insn
    4227              :    into a three-operand insn.
    4228              : 
    4229              :    This is the last transformation that cse_insn will try to do.  */
    4230              : 
    4231              : static void
    4232    139224217 : try_back_substitute_reg (rtx set, rtx_insn *insn)
    4233              : {
    4234    139224217 :   rtx dest = SET_DEST (set);
    4235    139224217 :   rtx src = SET_SRC (set);
    4236              : 
    4237    139224217 :   if (REG_P (dest)
    4238    115852984 :       && REG_P (src) && ! HARD_REGISTER_P (src)
    4239    146762927 :       && REGNO_QTY_VALID_P (REGNO (src)))
    4240              :     {
    4241      7538677 :       int src_q = REG_QTY (REGNO (src));
    4242      7538677 :       struct qty_table_elem *src_ent = &qty_table[src_q];
    4243              : 
    4244      7538677 :       if (src_ent->first_reg == REGNO (dest))
    4245              :         {
    4246              :           /* Scan for the previous nonnote insn, but stop at a basic
    4247              :              block boundary.  */
    4248      1843998 :           rtx_insn *prev = insn;
    4249      1843998 :           rtx_insn *bb_head = BB_HEAD (BLOCK_FOR_INSN (insn));
    4250      4600749 :           do
    4251              :             {
    4252      4600749 :               prev = PREV_INSN (prev);
    4253              :             }
    4254      4600749 :           while (prev != bb_head && (NOTE_P (prev) || DEBUG_INSN_P (prev)));
    4255              : 
    4256              :           /* Do not swap the registers around if the previous instruction
    4257              :              attaches a REG_EQUIV note to REG1.
    4258              : 
    4259              :              ??? It's not entirely clear whether we can transfer a REG_EQUIV
    4260              :              from the pseudo that originally shadowed an incoming argument
    4261              :              to another register.  Some uses of REG_EQUIV might rely on it
    4262              :              being attached to REG1 rather than REG2.
    4263              : 
    4264              :              This section previously turned the REG_EQUIV into a REG_EQUAL
    4265              :              note.  We cannot do that because REG_EQUIV may provide an
    4266              :              uninitialized stack slot when REG_PARM_STACK_SPACE is used.  */
    4267      1843998 :           if (NONJUMP_INSN_P (prev)
    4268      1108176 :               && GET_CODE (PATTERN (prev)) == SET
    4269       805879 :               && SET_DEST (PATTERN (prev)) == src
    4270      2084319 :               && ! find_reg_note (prev, REG_EQUIV, NULL_RTX))
    4271              :             {
    4272       240193 :               rtx note;
    4273              : 
    4274       240193 :               validate_change (prev, &SET_DEST (PATTERN (prev)), dest, 1);
    4275       240193 :               validate_change (insn, &SET_DEST (set), src, 1);
    4276       240193 :               validate_change (insn, &SET_SRC (set), dest, 1);
    4277       240193 :               apply_change_group ();
    4278              : 
    4279              :               /* If INSN has a REG_EQUAL note, and this note mentions
    4280              :                  REG0, then we must delete it, because the value in
    4281              :                  REG0 has changed.  If the note's value is REG1, we must
    4282              :                  also delete it because that is now this insn's dest.  */
    4283       240193 :               note = find_reg_note (insn, REG_EQUAL, NULL_RTX);
    4284       240193 :               if (note != 0
    4285       240193 :                   && (reg_mentioned_p (dest, XEXP (note, 0))
    4286         1465 :                       || rtx_equal_p (src, XEXP (note, 0))))
    4287           10 :                 remove_note (insn, note);
    4288              : 
    4289              :               /* If INSN has a REG_ARGS_SIZE note, move it to PREV.  */
    4290       240193 :               note = find_reg_note (insn, REG_ARGS_SIZE, NULL_RTX);
    4291       240193 :               if (note != 0)
    4292              :                 {
    4293            0 :                   remove_note (insn, note);
    4294            0 :                   gcc_assert (!find_reg_note (prev, REG_ARGS_SIZE, NULL_RTX));
    4295            0 :                   set_unique_reg_note (prev, REG_ARGS_SIZE, XEXP (note, 0));
    4296              :                 }
    4297              :             }
    4298              :         }
    4299              :     }
    4300    139224217 : }
    4301              : 
    4302              : /* Add an entry containing RTL X into SETS.  IS_FAKE_SET is true if X is
    4303              :    an artificial set that has been created to describe part of an insn's
    4304              :    effect.  */
    4305              : static inline void
    4306    198857980 : add_to_set (vec<struct set> *sets, rtx x, bool is_fake_set)
    4307              : {
    4308    198857980 :   struct set entry = {};
    4309    198857980 :   entry.rtl = x;
    4310    198857980 :   entry.is_fake_set = is_fake_set;
    4311    198857980 :   sets->safe_push (entry);
    4312    198857980 : }
    4313              : 
    4314              : /* Record all the SETs in this instruction into SETS_PTR,
    4315              :    and return the number of recorded sets.  */
    4316              : static int
    4317    411686435 : find_sets_in_insn (rtx_insn *insn, vec<struct set> *psets)
    4318              : {
    4319    411686435 :   rtx x = PATTERN (insn);
    4320              : 
    4321    411686435 :   if (GET_CODE (x) == SET)
    4322              :     {
    4323              :       /* Ignore SETs that are unconditional jumps.
    4324              :          They never need cse processing, so this does not hurt.
    4325              :          The reason is not efficiency but rather
    4326              :          so that we can test at the end for instructions
    4327              :          that have been simplified to unconditional jumps
    4328              :          and not be misled by unchanged instructions
    4329              :          that were unconditional jumps to begin with.  */
    4330    172878014 :       if (SET_DEST (x) == pc_rtx
    4331     20372640 :           && GET_CODE (SET_SRC (x)) == LABEL_REF)
    4332              :         ;
    4333              :       /* Don't count call-insns, (set (reg 0) (call ...)), as a set.
    4334              :          The hard function value register is used only once, to copy to
    4335              :          someplace else, so it isn't worth cse'ing.  */
    4336    172877832 :       else if (GET_CODE (SET_SRC (x)) == CALL)
    4337              :         ;
    4338    165562243 :       else if (GET_CODE (SET_SRC (x)) == CONST_VECTOR
    4339       676590 :                && GET_MODE_CLASS (GET_MODE (SET_SRC (x))) != MODE_VECTOR_BOOL
    4340              :                /* Prevent duplicates from being generated if the type is a V1
    4341              :                   type and a subreg.  Folding this will result in the same
    4342              :                   element as folding x itself.  */
    4343    166238833 :                && !(SUBREG_P (SET_DEST (x))
    4344           70 :                     && known_eq (GET_MODE_NUNITS (GET_MODE (SET_SRC (x))), 1)))
    4345              :         {
    4346              :           /* First register the vector itself.  */
    4347       676589 :           add_to_set (psets, x, false);
    4348       676589 :           rtx src = SET_SRC (x);
    4349              :           /* Go over the constants of the CONST_VECTOR in forward order, to
    4350              :              put them in the same order in the SETS array.  */
    4351      1353328 :           for (unsigned i = 0; i < const_vector_encoded_nelts (src) ; i++)
    4352              :             {
    4353              :               /* These are templates and don't actually get emitted but are
    4354              :                  used to tell CSE how to get to a particular constant.  */
    4355       676739 :               rtx y = simplify_gen_vec_select (SET_DEST (x), i);
    4356       676739 :               gcc_assert (y);
    4357       676739 :               if (!REG_P (y))
    4358              :                 {
    4359       675032 :                   rtx set = gen_rtx_SET (y, CONST_VECTOR_ELT (src, i));
    4360       675032 :                   add_to_set (psets, set, true);
    4361              :                 }
    4362              :             }
    4363              :         }
    4364              :       else
    4365    164885654 :         add_to_set (psets, x, false);
    4366              :     }
    4367    238808421 :   else if (GET_CODE (x) == PARALLEL)
    4368              :     {
    4369     31679152 :       int i, lim = XVECLEN (x, 0);
    4370              : 
    4371              :       /* Go over the expressions of the PARALLEL in forward order, to
    4372              :          put them in the same order in the SETS array.  */
    4373     96435054 :       for (i = 0; i < lim; i++)
    4374              :         {
    4375     64755902 :           rtx y = XVECEXP (x, 0, i);
    4376     64755902 :           if (GET_CODE (y) == SET)
    4377              :             {
    4378              :               /* As above, we ignore unconditional jumps and call-insns and
    4379              :                  ignore the result of apply_change_group.  */
    4380     32630912 :               if (SET_DEST (y) == pc_rtx
    4381        21207 :                   && GET_CODE (SET_SRC (y)) == LABEL_REF)
    4382              :                 ;
    4383     32630912 :               else if (GET_CODE (SET_SRC (y)) == CALL)
    4384              :                 ;
    4385              :               else
    4386     32620705 :                 add_to_set (psets, y, false);
    4387              :             }
    4388              :         }
    4389              :     }
    4390              : 
    4391    411686435 :   return psets->length ();
    4392              : }
    4393              : 
    4394              : /* Subroutine of canonicalize_insn.  X is an ASM_OPERANDS in INSN.  */
    4395              : 
    4396              : static void
    4397        85615 : canon_asm_operands (rtx x, rtx_insn *insn)
    4398              : {
    4399       115238 :   for (int i = ASM_OPERANDS_INPUT_LENGTH (x) - 1; i >= 0; i--)
    4400              :     {
    4401        29623 :       rtx input = ASM_OPERANDS_INPUT (x, i);
    4402        29623 :       if (!(REG_P (input) && HARD_REGISTER_P (input)))
    4403              :         {
    4404        29237 :           input = canon_reg (input, insn);
    4405        29237 :           validate_change (insn, &ASM_OPERANDS_INPUT (x, i), input, 1);
    4406              :         }
    4407              :     }
    4408        85615 : }
    4409              : 
    4410              : /* Where possible, substitute every register reference in the N_SETS
    4411              :    number of SETS in INSN with the canonical register.
    4412              : 
    4413              :    Register canonicalization propagatest the earliest register (i.e.
    4414              :    one that is set before INSN) with the same value.  This is a very
    4415              :    useful, simple form of CSE, to clean up warts from expanding GIMPLE
    4416              :    to RTL.  For instance, a CONST for an address is usually expanded
    4417              :    multiple times to loads into different registers, thus creating many
    4418              :    subexpressions of the form:
    4419              : 
    4420              :    (set (reg1) (some_const))
    4421              :    (set (mem (... reg1 ...) (thing)))
    4422              :    (set (reg2) (some_const))
    4423              :    (set (mem (... reg2 ...) (thing)))
    4424              : 
    4425              :    After canonicalizing, the code takes the following form:
    4426              : 
    4427              :    (set (reg1) (some_const))
    4428              :    (set (mem (... reg1 ...) (thing)))
    4429              :    (set (reg2) (some_const))
    4430              :    (set (mem (... reg1 ...) (thing)))
    4431              : 
    4432              :    The set to reg2 is now trivially dead, and the memory reference (or
    4433              :    address, or whatever) may be a candidate for further CSEing.
    4434              : 
    4435              :    In this function, the result of apply_change_group can be ignored;
    4436              :    see canon_reg.  */
    4437              : 
    4438              : static void
    4439    411686435 : canonicalize_insn (rtx_insn *insn, vec<struct set> *psets)
    4440              : {
    4441    411686435 :   vec<struct set> sets = *psets;
    4442    411686435 :   int n_sets = sets.length ();
    4443    411686435 :   rtx tem;
    4444    411686435 :   rtx x = PATTERN (insn);
    4445    411686435 :   int i;
    4446              : 
    4447    411686435 :   if (auto call = dyn_cast<rtx_call_insn *> (insn))
    4448              :     {
    4449     46958816 :       for (tem = CALL_INSN_FUNCTION_USAGE (call); tem; tem = XEXP (tem, 1))
    4450     31101364 :         if (GET_CODE (XEXP (tem, 0)) != SET)
    4451     30889669 :           XEXP (tem, 0) = canon_reg (XEXP (tem, 0), call);
    4452              :     }
    4453              : 
    4454    411686435 :   if (GET_CODE (x) == SET && GET_CODE (SET_SRC (x)) == CALL)
    4455              :     {
    4456      7315589 :       canon_reg (SET_SRC (x), insn);
    4457      7315589 :       apply_change_group ();
    4458      7315589 :       fold_rtx (SET_SRC (x), insn);
    4459              :     }
    4460    404370846 :   else if (GET_CODE (x) == CLOBBER)
    4461              :     {
    4462              :       /* If we clobber memory, canon the address.
    4463              :          This does nothing when a register is clobbered
    4464              :          because we have already invalidated the reg.  */
    4465        69609 :       if (MEM_P (XEXP (x, 0)))
    4466        13142 :         canon_reg (XEXP (x, 0), insn);
    4467              :     }
    4468    404301237 :   else if (GET_CODE (x) == USE
    4469    404301237 :            && ! (REG_P (XEXP (x, 0))
    4470      1294273 :                  && REGNO (XEXP (x, 0)) < FIRST_PSEUDO_REGISTER))
    4471              :     /* Canonicalize a USE of a pseudo register or memory location.  */
    4472            0 :     canon_reg (x, insn);
    4473    404301237 :   else if (GET_CODE (x) == ASM_OPERANDS)
    4474           18 :     canon_asm_operands (x, insn);
    4475    404301219 :   else if (GET_CODE (x) == CALL)
    4476              :     {
    4477      8041642 :       canon_reg (x, insn);
    4478      8041642 :       apply_change_group ();
    4479      8041642 :       fold_rtx (x, insn);
    4480              :     }
    4481    396259577 :   else if (DEBUG_INSN_P (insn))
    4482    197015612 :     canon_reg (PATTERN (insn), insn);
    4483    199243965 :   else if (GET_CODE (x) == PARALLEL)
    4484              :     {
    4485     96435054 :       for (i = XVECLEN (x, 0) - 1; i >= 0; i--)
    4486              :         {
    4487     64755902 :           rtx y = XVECEXP (x, 0, i);
    4488     64755902 :           if (GET_CODE (y) == SET && GET_CODE (SET_SRC (y)) == CALL)
    4489              :             {
    4490        10207 :               canon_reg (SET_SRC (y), insn);
    4491        10207 :               apply_change_group ();
    4492        10207 :               fold_rtx (SET_SRC (y), insn);
    4493              :             }
    4494     64745695 :           else if (GET_CODE (y) == CLOBBER)
    4495              :             {
    4496     31276929 :               if (MEM_P (XEXP (y, 0)))
    4497        63924 :                 canon_reg (XEXP (y, 0), insn);
    4498              :             }
    4499     33468766 :           else if (GET_CODE (y) == USE
    4500     33468766 :                    && ! (REG_P (XEXP (y, 0))
    4501       195610 :                          && REGNO (XEXP (y, 0)) < FIRST_PSEUDO_REGISTER))
    4502       234125 :             canon_reg (y, insn);
    4503     33234641 :           else if (GET_CODE (y) == ASM_OPERANDS)
    4504        85597 :             canon_asm_operands (y, insn);
    4505     33149044 :           else if (GET_CODE (y) == CALL)
    4506              :             {
    4507       490014 :               canon_reg (y, insn);
    4508       490014 :               apply_change_group ();
    4509       490014 :               fold_rtx (y, insn);
    4510              :             }
    4511              :         }
    4512              :     }
    4513              : 
    4514    195659585 :   if (n_sets == 1 && REG_NOTES (insn) != 0
    4515    537785989 :       && (tem = find_reg_note (insn, REG_EQUAL, NULL_RTX)) != 0)
    4516              :     {
    4517              :       /* We potentially will process this insn many times.  Therefore,
    4518              :          drop the REG_EQUAL note if it is equal to the SET_SRC of the
    4519              :          unique set in INSN.
    4520              : 
    4521              :          Do not do so if the REG_EQUAL note is for a STRICT_LOW_PART,
    4522              :          because cse_insn handles those specially.  */
    4523      9092513 :       if (GET_CODE (SET_DEST (sets[0].rtl)) != STRICT_LOW_PART
    4524      9092513 :           && rtx_equal_p (XEXP (tem, 0), SET_SRC (sets[0].rtl)))
    4525       182244 :         remove_note (insn, tem);
    4526              :       else
    4527              :         {
    4528      8910269 :           canon_reg (XEXP (tem, 0), insn);
    4529      8910269 :           apply_change_group ();
    4530      8910269 :           XEXP (tem, 0) = fold_rtx (XEXP (tem, 0), insn);
    4531      8910269 :           df_notes_rescan (insn);
    4532              :         }
    4533              :     }
    4534              : 
    4535              :   /* Canonicalize sources and addresses of destinations.
    4536              :      We do this in a separate pass to avoid problems when a MATCH_DUP is
    4537              :      present in the insn pattern.  In that case, we want to ensure that
    4538              :      we don't break the duplicate nature of the pattern.  So we will replace
    4539              :      both operands at the same time.  Otherwise, we would fail to find an
    4540              :      equivalent substitution in the loop calling validate_change below.
    4541              : 
    4542              :      We used to suppress canonicalization of DEST if it appears in SRC,
    4543              :      but we don't do this any more.  */
    4544              : 
    4545    610544415 :   for (i = 0; i < n_sets; i++)
    4546              :     {
    4547    198857980 :       rtx dest = SET_DEST (sets[i].rtl);
    4548    198857980 :       rtx src = SET_SRC (sets[i].rtl);
    4549    198857980 :       rtx new_rtx = canon_reg (src, insn);
    4550              : 
    4551    198857980 :       validate_change (insn, &SET_SRC (sets[i].rtl), new_rtx, 1);
    4552              : 
    4553    198857980 :       if (GET_CODE (dest) == ZERO_EXTRACT)
    4554              :         {
    4555         4033 :           validate_change (insn, &XEXP (dest, 1),
    4556              :                            canon_reg (XEXP (dest, 1), insn), 1);
    4557         4033 :           validate_change (insn, &XEXP (dest, 2),
    4558              :                            canon_reg (XEXP (dest, 2), insn), 1);
    4559              :         }
    4560              : 
    4561    200530273 :       while (GET_CODE (dest) == SUBREG
    4562    198875876 :              || GET_CODE (dest) == ZERO_EXTRACT
    4563    399402116 :              || GET_CODE (dest) == STRICT_LOW_PART)
    4564      1672293 :         dest = XEXP (dest, 0);
    4565              : 
    4566    198857980 :       if (MEM_P (dest))
    4567     29285228 :         canon_reg (dest, insn);
    4568              :     }
    4569              : 
    4570              :   /* Now that we have done all the replacements, we can apply the change
    4571              :      group and see if they all work.  Note that this will cause some
    4572              :      canonicalizations that would have worked individually not to be applied
    4573              :      because some other canonicalization didn't work, but this should not
    4574              :      occur often.
    4575              : 
    4576              :      The result of apply_change_group can be ignored; see canon_reg.  */
    4577              : 
    4578    411686435 :   apply_change_group ();
    4579    411686435 : }
    4580              : 
    4581              : /* Main function of CSE.
    4582              :    First simplify sources and addresses of all assignments
    4583              :    in the instruction, using previously-computed equivalents values.
    4584              :    Then install the new sources and destinations in the table
    4585              :    of available values.  */
    4586              : 
    4587              : static void
    4588    411686435 : cse_insn (rtx_insn *insn)
    4589              : {
    4590    411686435 :   rtx x = PATTERN (insn);
    4591    411686435 :   int i;
    4592    411686435 :   rtx tem;
    4593    411686435 :   int n_sets = 0;
    4594              : 
    4595    411686435 :   rtx src_eqv = 0;
    4596    411686435 :   struct table_elt *src_eqv_elt = 0;
    4597    411686435 :   int src_eqv_volatile = 0;
    4598    411686435 :   int src_eqv_in_memory = 0;
    4599    411686435 :   unsigned src_eqv_hash = 0;
    4600              : 
    4601    411686435 :   this_insn = insn;
    4602              : 
    4603              :   /* Find all regs explicitly clobbered in this insn,
    4604              :      to ensure they are not replaced with any other regs
    4605              :      elsewhere in this insn.  */
    4606    411686435 :   invalidate_from_sets_and_clobbers (insn);
    4607              : 
    4608              :   /* Record all the SETs in this instruction.  */
    4609    411686435 :   auto_vec<struct set, 8> sets;
    4610    411686435 :   n_sets = find_sets_in_insn (insn, (vec<struct set>*)&sets);
    4611              : 
    4612              :   /* Substitute the canonical register where possible.  */
    4613    411686435 :   canonicalize_insn (insn, (vec<struct set>*)&sets);
    4614              : 
    4615              :   /* If this insn has a REG_EQUAL note, store the equivalent value in SRC_EQV,
    4616              :      if different, or if the DEST is a STRICT_LOW_PART/ZERO_EXTRACT.  The
    4617              :      latter condition is necessary because SRC_EQV is handled specially for
    4618              :      this case, and if it isn't set, then there will be no equivalence
    4619              :      for the destination.  */
    4620    195659585 :   if (n_sets == 1 && REG_NOTES (insn) != 0
    4621    537642681 :       && (tem = find_reg_note (insn, REG_EQUAL, NULL_RTX)) != 0)
    4622              :     {
    4623              : 
    4624      8910269 :       if (GET_CODE (SET_DEST (sets[0].rtl)) != ZERO_EXTRACT
    4625      8910269 :           && (! rtx_equal_p (XEXP (tem, 0), SET_SRC (sets[0].rtl))
    4626        16367 :               || GET_CODE (SET_DEST (sets[0].rtl)) == STRICT_LOW_PART))
    4627      8893902 :         src_eqv = copy_rtx (XEXP (tem, 0));
    4628              :       /* If DEST is of the form ZERO_EXTACT, as in:
    4629              :          (set (zero_extract:SI (reg:SI 119)
    4630              :                   (const_int 16 [0x10])
    4631              :                   (const_int 16 [0x10]))
    4632              :               (const_int 51154 [0xc7d2]))
    4633              :          REG_EQUAL note will specify the value of register (reg:SI 119) at this
    4634              :          point.  Note that this is different from SRC_EQV. We can however
    4635              :          calculate SRC_EQV with the position and width of ZERO_EXTRACT.  */
    4636        16367 :       else if (GET_CODE (SET_DEST (sets[0].rtl)) == ZERO_EXTRACT
    4637            0 :                && CONST_INT_P (XEXP (tem, 0))
    4638            0 :                && CONST_INT_P (XEXP (SET_DEST (sets[0].rtl), 1))
    4639        16367 :                && CONST_INT_P (XEXP (SET_DEST (sets[0].rtl), 2)))
    4640              :         {
    4641            0 :           rtx dest_reg = XEXP (SET_DEST (sets[0].rtl), 0);
    4642              :           /* This is the mode of XEXP (tem, 0) as well.  */
    4643            0 :           scalar_int_mode dest_mode
    4644            0 :             = as_a <scalar_int_mode> (GET_MODE (dest_reg));
    4645            0 :           rtx width = XEXP (SET_DEST (sets[0].rtl), 1);
    4646            0 :           rtx pos = XEXP (SET_DEST (sets[0].rtl), 2);
    4647            0 :           HOST_WIDE_INT val = INTVAL (XEXP (tem, 0));
    4648            0 :           HOST_WIDE_INT mask;
    4649            0 :           unsigned int shift;
    4650            0 :           if (BITS_BIG_ENDIAN)
    4651              :             shift = (GET_MODE_PRECISION (dest_mode)
    4652              :                      - INTVAL (pos) - INTVAL (width));
    4653              :           else
    4654            0 :             shift = INTVAL (pos);
    4655            0 :           if (INTVAL (width) == HOST_BITS_PER_WIDE_INT)
    4656              :             mask = HOST_WIDE_INT_M1;
    4657              :           else
    4658            0 :             mask = (HOST_WIDE_INT_1 << INTVAL (width)) - 1;
    4659            0 :           val = (val >> shift) & mask;
    4660            0 :           src_eqv = GEN_INT (val);
    4661              :         }
    4662              :     }
    4663              : 
    4664              :   /* Set sets[i].src_elt to the class each source belongs to.
    4665              :      Detect assignments from or to volatile things
    4666              :      and set set[i] to zero so they will be ignored
    4667              :      in the rest of this function.
    4668              : 
    4669              :      Nothing in this loop changes the hash table or the register chains.  */
    4670              : 
    4671    610544423 :   for (i = 0; i < n_sets; i++)
    4672              :     {
    4673    198857988 :       bool repeat = false;
    4674    198857988 :       bool noop_insn = false;
    4675    198857988 :       rtx src, dest;
    4676    198857988 :       rtx src_folded;
    4677    198857988 :       struct table_elt *elt = 0, *p;
    4678    198857988 :       machine_mode mode;
    4679    198857988 :       rtx src_eqv_here;
    4680    198857988 :       rtx src_const = 0;
    4681    198857988 :       rtx src_related = 0;
    4682    198857988 :       rtx dest_related = 0;
    4683    198857988 :       bool src_related_is_const_anchor = false;
    4684    198857988 :       struct table_elt *src_const_elt = 0;
    4685    198857988 :       int src_cost = MAX_COST;
    4686    198857988 :       int src_eqv_cost = MAX_COST;
    4687    198857988 :       int src_folded_cost = MAX_COST;
    4688    198857988 :       int src_related_cost = MAX_COST;
    4689    198857988 :       int src_elt_cost = MAX_COST;
    4690    198857988 :       int src_regcost = MAX_COST;
    4691    198857988 :       int src_eqv_regcost = MAX_COST;
    4692    198857988 :       int src_folded_regcost = MAX_COST;
    4693    198857988 :       int src_related_regcost = MAX_COST;
    4694    198857988 :       int src_elt_regcost = MAX_COST;
    4695    198857988 :       scalar_int_mode int_mode;
    4696    198857988 :       bool is_fake_set = sets[i].is_fake_set;
    4697              : 
    4698    198857988 :       dest = SET_DEST (sets[i].rtl);
    4699    198857988 :       src = SET_SRC (sets[i].rtl);
    4700              : 
    4701              :       /* If SRC is a constant that has no machine mode,
    4702              :          hash it with the destination's machine mode.
    4703              :          This way we can keep different modes separate.  */
    4704              : 
    4705    198857988 :       mode = GET_MODE (src) == VOIDmode ? GET_MODE (dest) : GET_MODE (src);
    4706    198857988 :       sets[i].mode = mode;
    4707              : 
    4708    198857988 :       if (!is_fake_set && src_eqv)
    4709              :         {
    4710      8893902 :           machine_mode eqvmode = mode;
    4711      8893902 :           if (GET_CODE (dest) == STRICT_LOW_PART)
    4712            0 :             eqvmode = GET_MODE (SUBREG_REG (XEXP (dest, 0)));
    4713      8893902 :           do_not_record = 0;
    4714      8893902 :           hash_arg_in_memory = 0;
    4715      8893902 :           src_eqv_hash = HASH (src_eqv, eqvmode);
    4716              : 
    4717              :           /* Find the equivalence class for the equivalent expression.  */
    4718              : 
    4719      8893902 :           if (!do_not_record)
    4720      8891678 :             src_eqv_elt = lookup (src_eqv, src_eqv_hash, eqvmode);
    4721              : 
    4722      8893902 :           src_eqv_volatile = do_not_record;
    4723      8893902 :           src_eqv_in_memory = hash_arg_in_memory;
    4724              :         }
    4725              : 
    4726              :       /* If this is a STRICT_LOW_PART assignment, src_eqv corresponds to the
    4727              :          value of the INNER register, not the destination.  So it is not
    4728              :          a valid substitution for the source.  But save it for later.  */
    4729    198857988 :       if (is_fake_set || GET_CODE (dest) == STRICT_LOW_PART)
    4730              :         src_eqv_here = 0;
    4731              :       else
    4732    198857988 :         src_eqv_here = src_eqv;
    4733              : 
    4734              :       /* Simplify and foldable subexpressions in SRC.  Then get the fully-
    4735              :          simplified result, which may not necessarily be valid.  */
    4736    198857988 :       src_folded = fold_rtx (src, NULL);
    4737              : 
    4738              : #if 0
    4739              :       /* ??? This caused bad code to be generated for the m68k port with -O2.
    4740              :          Suppose src is (CONST_INT -1), and that after truncation src_folded
    4741              :          is (CONST_INT 3).  Suppose src_folded is then used for src_const.
    4742              :          At the end we will add src and src_const to the same equivalence
    4743              :          class.  We now have 3 and -1 on the same equivalence class.  This
    4744              :          causes later instructions to be mis-optimized.  */
    4745              :       /* If storing a constant in a bitfield, pre-truncate the constant
    4746              :          so we will be able to record it later.  */
    4747              :       if (GET_CODE (SET_DEST (sets[i].rtl)) == ZERO_EXTRACT)
    4748              :         {
    4749              :           rtx width = XEXP (SET_DEST (sets[i].rtl), 1);
    4750              : 
    4751              :           if (CONST_INT_P (src)
    4752              :               && CONST_INT_P (width)
    4753              :               && INTVAL (width) < HOST_BITS_PER_WIDE_INT
    4754              :               && (INTVAL (src) & ((HOST_WIDE_INT) (-1) << INTVAL (width))))
    4755              :             src_folded
    4756              :               = GEN_INT (INTVAL (src) & ((HOST_WIDE_INT_1
    4757              :                                           << INTVAL (width)) - 1));
    4758              :         }
    4759              : #endif
    4760              : 
    4761              :       /* Compute SRC's hash code, and also notice if it
    4762              :          should not be recorded at all.  In that case,
    4763              :          prevent any further processing of this assignment.
    4764              : 
    4765              :          We set DO_NOT_RECORD if the destination has a REG_UNUSED note.
    4766              :          This avoids getting the source register into the tables, where it
    4767              :          may be invalidated later (via REG_QTY), then trigger an ICE upon
    4768              :          re-insertion.
    4769              : 
    4770              :          This is only a problem in multi-set insns.  If it were a single
    4771              :          set the dead copy would have been removed.  If the RHS were anything
    4772              :          but a simple REG, then we won't call insert_regs and thus there's
    4773              :          no potential for triggering the ICE.  */
    4774    397715976 :       do_not_record = (REG_P (dest)
    4775    147522498 :                        && REG_P (src)
    4776    234066664 :                        && find_reg_note (insn, REG_UNUSED, dest));
    4777    198857988 :       hash_arg_in_memory = 0;
    4778              : 
    4779    198857988 :       sets[i].src = src;
    4780    198857988 :       sets[i].src_hash = HASH (src, mode);
    4781    198857988 :       sets[i].src_volatile = do_not_record;
    4782    198857988 :       sets[i].src_in_memory = hash_arg_in_memory;
    4783              : 
    4784              :       /* If SRC is a MEM, there is a REG_EQUIV note for SRC, and DEST is
    4785              :          a pseudo, do not record SRC.  Using SRC as a replacement for
    4786              :          anything else will be incorrect in that situation.  Note that
    4787              :          this usually occurs only for stack slots, in which case all the
    4788              :          RTL would be referring to SRC, so we don't lose any optimization
    4789              :          opportunities by not having SRC in the hash table.  */
    4790              : 
    4791    198857988 :       if (MEM_P (src)
    4792     25393199 :           && find_reg_note (insn, REG_EQUIV, NULL_RTX) != 0
    4793       926997 :           && REG_P (dest)
    4794    199784985 :           && REGNO (dest) >= FIRST_PSEUDO_REGISTER)
    4795       926997 :         sets[i].src_volatile = 1;
    4796              : 
    4797    197930991 :       else if (GET_CODE (src) == ASM_OPERANDS
    4798       210780 :                && GET_CODE (x) == PARALLEL)
    4799              :         {
    4800              :           /* Do not record result of a non-volatile inline asm with
    4801              :              more than one result.  */
    4802       210756 :           if (n_sets > 1)
    4803       167066 :             sets[i].src_volatile = 1;
    4804              : 
    4805       210756 :           int j, lim = XVECLEN (x, 0);
    4806      1075618 :           for (j = 0; j < lim; j++)
    4807              :             {
    4808       867310 :               rtx y = XVECEXP (x, 0, j);
    4809              :               /* And do not record result of a non-volatile inline asm
    4810              :                  with "memory" clobber.  */
    4811       867310 :               if (GET_CODE (y) == CLOBBER && MEM_P (XEXP (y, 0)))
    4812              :                 {
    4813         2448 :                   sets[i].src_volatile = 1;
    4814         2448 :                   break;
    4815              :                 }
    4816              :             }
    4817              :         }
    4818              : 
    4819              : #if 0
    4820              :       /* It is no longer clear why we used to do this, but it doesn't
    4821              :          appear to still be needed.  So let's try without it since this
    4822              :          code hurts cse'ing widened ops.  */
    4823              :       /* If source is a paradoxical subreg (such as QI treated as an SI),
    4824              :          treat it as volatile.  It may do the work of an SI in one context
    4825              :          where the extra bits are not being used, but cannot replace an SI
    4826              :          in general.  */
    4827              :       if (paradoxical_subreg_p (src))
    4828              :         sets[i].src_volatile = 1;
    4829              : #endif
    4830              : 
    4831              :       /* Locate all possible equivalent forms for SRC.  Try to replace
    4832              :          SRC in the insn with each cheaper equivalent.
    4833              : 
    4834              :          We have the following types of equivalents: SRC itself, a folded
    4835              :          version, a value given in a REG_EQUAL note, or a value related
    4836              :          to a constant.
    4837              : 
    4838              :          Each of these equivalents may be part of an additional class
    4839              :          of equivalents (if more than one is in the table, they must be in
    4840              :          the same class; we check for this).
    4841              : 
    4842              :          If the source is volatile, we don't do any table lookups.
    4843              : 
    4844              :          We note any constant equivalent for possible later use in a
    4845              :          REG_NOTE.  */
    4846              : 
    4847    198857988 :       if (!sets[i].src_volatile)
    4848    164183642 :         elt = lookup (src, sets[i].src_hash, mode);
    4849              : 
    4850    198857988 :       sets[i].src_elt = elt;
    4851              : 
    4852    198857988 :       if (elt && src_eqv_here && src_eqv_elt)
    4853              :         {
    4854      3019978 :           if (elt->first_same_value != src_eqv_elt->first_same_value)
    4855              :             {
    4856              :               /* The REG_EQUAL is indicating that two formerly distinct
    4857              :                  classes are now equivalent.  So merge them.  */
    4858        10014 :               merge_equiv_classes (elt, src_eqv_elt);
    4859        10014 :               src_eqv_hash = HASH (src_eqv, elt->mode);
    4860        10014 :               src_eqv_elt = lookup (src_eqv, src_eqv_hash, elt->mode);
    4861              :             }
    4862              : 
    4863              :           src_eqv_here = 0;
    4864              :         }
    4865              : 
    4866    195638700 :       else if (src_eqv_elt)
    4867              :         elt = src_eqv_elt;
    4868              : 
    4869              :       /* Try to find a constant somewhere and record it in `src_const'.
    4870              :          Record its table element, if any, in `src_const_elt'.  Look in
    4871              :          any known equivalences first.  (If the constant is not in the
    4872              :          table, also set `sets[i].src_const_hash').  */
    4873    195465772 :       if (elt)
    4874    103131013 :         for (p = elt->first_same_value; p; p = p->next_same_value)
    4875     84590116 :           if (p->is_const)
    4876              :             {
    4877     16638613 :               src_const = p->exp;
    4878     16638613 :               src_const_elt = elt;
    4879     16638613 :               break;
    4880              :             }
    4881              : 
    4882     35179510 :       if (src_const == 0
    4883    182219375 :           && (CONSTANT_P (src_folded)
    4884              :               /* Consider (minus (label_ref L1) (label_ref L2)) as
    4885              :                  "constant" here so we will record it. This allows us
    4886              :                  to fold switch statements when an ADDR_DIFF_VEC is used.  */
    4887    155105001 :               || (GET_CODE (src_folded) == MINUS
    4888      1890692 :                   && GET_CODE (XEXP (src_folded, 0)) == LABEL_REF
    4889           95 :                   && GET_CODE (XEXP (src_folded, 1)) == LABEL_REF)))
    4890              :         src_const = src_folded, src_const_elt = elt;
    4891    171743528 :       else if (src_const == 0 && src_eqv_here && CONSTANT_P (src_eqv_here))
    4892       422784 :         src_const = src_eqv_here, src_const_elt = src_eqv_elt;
    4893              : 
    4894              :       /* If we don't know if the constant is in the table, get its
    4895              :          hash code and look it up.  */
    4896    198857988 :       if (src_const && src_const_elt == 0)
    4897              :         {
    4898     27536200 :           sets[i].src_const_hash = HASH (src_const, mode);
    4899     27536200 :           src_const_elt = lookup (src_const, sets[i].src_const_hash, mode);
    4900              :         }
    4901              : 
    4902    198857988 :       sets[i].src_const = src_const;
    4903    198857988 :       sets[i].src_const_elt = src_const_elt;
    4904              : 
    4905              :       /* If the constant and our source are both in the table, mark them as
    4906              :          equivalent.  Otherwise, if a constant is in the table but the source
    4907              :          isn't, set ELT to it.  */
    4908    198857988 :       if (src_const_elt && elt
    4909     16639657 :           && src_const_elt->first_same_value != elt->first_same_value)
    4910            0 :         merge_equiv_classes (elt, src_const_elt);
    4911    198857988 :       else if (src_const_elt && elt == 0)
    4912    198857988 :         elt = src_const_elt;
    4913              : 
    4914              :       /* See if there is a register linearly related to a constant
    4915              :          equivalent of SRC.  */
    4916    198857988 :       if (src_const
    4917     44175857 :           && (GET_CODE (src_const) == CONST
    4918     43513857 :               || (src_const_elt && src_const_elt->related_value != 0)))
    4919              :         {
    4920       755133 :           src_related = use_related_value (src_const, src_const_elt);
    4921       755133 :           if (src_related)
    4922              :             {
    4923       227441 :               struct table_elt *src_related_elt
    4924       227441 :                 = lookup (src_related, HASH (src_related, mode), mode);
    4925       227441 :               if (src_related_elt && elt)
    4926              :                 {
    4927         1925 :                   if (elt->first_same_value
    4928         1925 :                       != src_related_elt->first_same_value)
    4929              :                     /* This can occur when we previously saw a CONST
    4930              :                        involving a SYMBOL_REF and then see the SYMBOL_REF
    4931              :                        twice.  Merge the involved classes.  */
    4932          919 :                     merge_equiv_classes (elt, src_related_elt);
    4933              : 
    4934              :                   src_related = 0;
    4935    198857988 :                   src_related_elt = 0;
    4936              :                 }
    4937       225516 :               else if (src_related_elt && elt == 0)
    4938         6922 :                 elt = src_related_elt;
    4939              :             }
    4940              :         }
    4941              : 
    4942              :       /* See if we have a CONST_INT that is already in a register in a
    4943              :          wider mode.  */
    4944              : 
    4945     43950341 :       if (src_const && src_related == 0 && CONST_INT_P (src_const)
    4946     19106567 :           && is_int_mode (mode, &int_mode)
    4947    219997125 :           && GET_MODE_PRECISION (int_mode) < BITS_PER_WORD)
    4948              :         {
    4949      8298145 :           opt_scalar_int_mode wider_mode_iter;
    4950     21239895 :           FOR_EACH_WIDER_MODE (wider_mode_iter, int_mode)
    4951              :             {
    4952     21239895 :               scalar_int_mode wider_mode = wider_mode_iter.require ();
    4953     21991824 :               if (GET_MODE_PRECISION (wider_mode) > BITS_PER_WORD)
    4954              :                 break;
    4955              : 
    4956     13184151 :               struct table_elt *const_elt
    4957     13184151 :                 = lookup (src_const, HASH (src_const, wider_mode), wider_mode);
    4958              : 
    4959     13184151 :               if (const_elt == 0)
    4960     12505865 :                 continue;
    4961              : 
    4962       678286 :               for (const_elt = const_elt->first_same_value;
    4963      2062228 :                    const_elt; const_elt = const_elt->next_same_value)
    4964      1626343 :                 if (REG_P (const_elt->exp))
    4965              :                   {
    4966       242401 :                     src_related = gen_lowpart (int_mode, const_elt->exp);
    4967       242401 :                     break;
    4968              :                   }
    4969              : 
    4970       678286 :               if (src_related != 0)
    4971              :                 break;
    4972              :             }
    4973              :         }
    4974              : 
    4975              :       /* Another possibility is that we have an AND with a constant in
    4976              :          a mode narrower than a word.  If so, it might have been generated
    4977              :          as part of an "if" which would narrow the AND.  If we already
    4978              :          have done the AND in a wider mode, we can use a SUBREG of that
    4979              :          value.  */
    4980              : 
    4981    194488111 :       if (flag_expensive_optimizations && ! src_related
    4982    332624864 :           && is_a <scalar_int_mode> (mode, &int_mode)
    4983    133766876 :           && GET_CODE (src) == AND && CONST_INT_P (XEXP (src, 1))
    4984    200176408 :           && GET_MODE_SIZE (int_mode) < UNITS_PER_WORD)
    4985              :         {
    4986       783913 :           opt_scalar_int_mode tmode_iter;
    4987       783913 :           rtx new_and = gen_rtx_AND (VOIDmode, NULL_RTX, XEXP (src, 1));
    4988              : 
    4989      2307581 :           FOR_EACH_WIDER_MODE (tmode_iter, int_mode)
    4990              :             {
    4991      2307581 :               scalar_int_mode tmode = tmode_iter.require ();
    4992      4768938 :               if (GET_MODE_SIZE (tmode) > UNITS_PER_WORD)
    4993              :                 break;
    4994              : 
    4995      1523732 :               rtx inner = gen_lowpart (tmode, XEXP (src, 0));
    4996      1523732 :               struct table_elt *larger_elt;
    4997              : 
    4998      1523732 :               if (inner)
    4999              :                 {
    5000      1514020 :                   PUT_MODE (new_and, tmode);
    5001      1514020 :                   XEXP (new_and, 0) = inner;
    5002      1514020 :                   larger_elt = lookup (new_and, HASH (new_and, tmode), tmode);
    5003      1514020 :                   if (larger_elt == 0)
    5004      1513956 :                     continue;
    5005              : 
    5006           64 :                   for (larger_elt = larger_elt->first_same_value;
    5007           64 :                        larger_elt; larger_elt = larger_elt->next_same_value)
    5008           64 :                     if (REG_P (larger_elt->exp))
    5009              :                       {
    5010           64 :                         src_related
    5011           64 :                           = gen_lowpart (int_mode, larger_elt->exp);
    5012           64 :                         break;
    5013              :                       }
    5014              : 
    5015           64 :                   if (src_related)
    5016              :                     break;
    5017              :                 }
    5018              :             }
    5019              :         }
    5020              : 
    5021              :       /* If SRC_EQV is a CONST_INT, try looking up some related
    5022              :          constants (logical and arithmetic negation).  Those may
    5023              :          ultimately be cheaper to re-use.  */
    5024    198857988 :       if (GET_CODE (src) != CONST_INT
    5025              :           && GET_CODE (src) != REG
    5026              :           && GET_CODE (src) != SUBREG
    5027    123153479 :           && src_const
    5028     16463734 :           && GET_CODE (src_const) == CONST_INT)
    5029              :         {
    5030        68826 :           rtx trial_rtx = GEN_INT (~UINTVAL (src_const));
    5031        68826 :           struct table_elt *tmp = lookup (trial_rtx, HASH (trial_rtx, mode), mode);
    5032        68826 :           rtx_code code = NOT;
    5033        68826 :           if (!tmp)
    5034              :             {
    5035        67670 :               trial_rtx = GEN_INT (-UINTVAL (src_const));
    5036        67670 :               tmp = lookup (trial_rtx, HASH (trial_rtx, mode), mode);
    5037        67670 :               code = NEG;
    5038              :             }
    5039              : 
    5040        67670 :           if (tmp)
    5041              :             {
    5042        11206 :               src_related = gen_rtx_fmt_e (code, mode, tmp->first_same_value->exp);
    5043        11206 :               src_eqv_here = src_related;
    5044        11206 :               src_related_is_const_anchor = true;
    5045              :             }
    5046              : 
    5047              :         }
    5048              : 
    5049              :       /* See if a MEM has already been loaded with a widening operation;
    5050              :          if it has, we can use a subreg of that.  Many CISC machines
    5051              :          also have such operations, but this is only likely to be
    5052              :          beneficial on these machines.  */
    5053              : 
    5054    198857988 :       rtx_code extend_op;
    5055    198857988 :       if (flag_expensive_optimizations && src_related == 0
    5056              :           && MEM_P (src) && ! do_not_record
    5057              :           && is_a <scalar_int_mode> (mode, &int_mode)
    5058              :           && (extend_op = load_extend_op (int_mode)) != UNKNOWN)
    5059              :         {
    5060              : #if GCC_VERSION >= 5000
    5061              :           struct rtx_def memory_extend_buf;
    5062              :           rtx memory_extend_rtx = &memory_extend_buf;
    5063              : #else
    5064              :           /* Workaround GCC < 5 bug, fixed in r5-3834 as part of PR63362
    5065              :              fix.  */
    5066              :           alignas (rtx_def) unsigned char memory_extended_buf[sizeof (rtx_def)];
    5067              :           rtx memory_extend_rtx = (rtx) &memory_extended_buf[0];
    5068              : #endif
    5069              : 
    5070              :           /* Set what we are trying to extend and the operation it might
    5071              :              have been extended with.  */
    5072              :           memset (memory_extend_rtx, 0, sizeof (*memory_extend_rtx));
    5073              :           PUT_CODE (memory_extend_rtx, extend_op);
    5074              :           XEXP (memory_extend_rtx, 0) = src;
    5075              : 
    5076              :           opt_scalar_int_mode tmode_iter;
    5077              :           FOR_EACH_WIDER_MODE (tmode_iter, int_mode)
    5078              :             {
    5079              :               struct table_elt *larger_elt;
    5080              : 
    5081              :               scalar_int_mode tmode = tmode_iter.require ();
    5082              :               if (GET_MODE_SIZE (tmode) > UNITS_PER_WORD)
    5083              :                 break;
    5084              : 
    5085              :               PUT_MODE (memory_extend_rtx, tmode);
    5086              :               larger_elt = lookup (memory_extend_rtx,
    5087              :                                    HASH (memory_extend_rtx, tmode), tmode);
    5088              :               if (larger_elt == 0)
    5089              :                 continue;
    5090              : 
    5091              :               for (larger_elt = larger_elt->first_same_value;
    5092              :                    larger_elt; larger_elt = larger_elt->next_same_value)
    5093              :                 if (REG_P (larger_elt->exp))
    5094              :                   {
    5095              :                     src_related = gen_lowpart (int_mode, larger_elt->exp);
    5096              :                     break;
    5097              :                   }
    5098              : 
    5099              :               if (src_related)
    5100              :                 break;
    5101              :             }
    5102              :         }
    5103              : 
    5104              :       /* Try to express the constant using a register+offset expression
    5105              :          derived from a constant anchor.  */
    5106              : 
    5107    198857988 :       if (targetm.const_anchor
    5108            0 :           && !src_related
    5109            0 :           && src_const
    5110            0 :           && GET_CODE (src_const) == CONST_INT)
    5111              :         {
    5112            0 :           src_related = try_const_anchors (src_const, mode);
    5113            0 :           src_related_is_const_anchor = src_related != NULL_RTX;
    5114              :         }
    5115              : 
    5116              :       /* Try to re-materialize a vec_dup with an existing constant.   */
    5117    198857988 :       rtx src_elt;
    5118      5885097 :       if ((!src_eqv_here || CONSTANT_P (src_eqv_here))
    5119    198857988 :           && const_vec_duplicate_p (src, &src_elt))
    5120              :         {
    5121       680094 :            machine_mode const_mode = GET_MODE_INNER (GET_MODE (src));
    5122       680094 :            struct table_elt *related_elt
    5123       680094 :                 = lookup (src_elt, HASH (src_elt, const_mode), const_mode);
    5124       680094 :            if (related_elt)
    5125              :             {
    5126       270724 :               for (related_elt = related_elt->first_same_value;
    5127      2080625 :                    related_elt; related_elt = related_elt->next_same_value)
    5128      1841577 :                 if (REG_P (related_elt->exp))
    5129              :                   {
    5130              :                    /* We don't need to compare costs with an existing (constant)
    5131              :                       src_eqv_here, since any such src_eqv_here should already be
    5132              :                       available in src_const.  */
    5133        31676 :                     src_eqv_here
    5134        31676 :                         = gen_rtx_VEC_DUPLICATE (GET_MODE (src),
    5135              :                                                  related_elt->exp);
    5136        31676 :                     break;
    5137              :                   }
    5138              :             }
    5139              :         }
    5140              : 
    5141    198857988 :       if (src == src_folded)
    5142    194427175 :         src_folded = 0;
    5143              : 
    5144              :       /* At this point, ELT, if nonzero, points to a class of expressions
    5145              :          equivalent to the source of this SET and SRC, SRC_EQV, SRC_FOLDED,
    5146              :          and SRC_RELATED, if nonzero, each contain additional equivalent
    5147              :          expressions.  Prune these latter expressions by deleting expressions
    5148              :          already in the equivalence class.
    5149              : 
    5150              :          Check for an equivalent identical to the destination.  If found,
    5151              :          this is the preferred equivalent since it will likely lead to
    5152              :          elimination of the insn.  Indicate this by placing it in
    5153              :          `src_related'.  */
    5154              : 
    5155    198857988 :       if (elt)
    5156     35248765 :         elt = elt->first_same_value;
    5157    319419033 :       for (p = elt; p; p = p->next_same_value)
    5158              :         {
    5159    120561045 :           enum rtx_code code = GET_CODE (p->exp);
    5160              : 
    5161              :           /* If the expression is not valid, ignore it.  Then we do not
    5162              :              have to check for validity below.  In most cases, we can use
    5163              :              `rtx_equal_p', since canonicalization has already been done.  */
    5164    120561045 :           if (code != REG && ! exp_equiv_p (p->exp, p->exp, 1, false))
    5165         4225 :             continue;
    5166              : 
    5167              :           /* Also skip paradoxical subregs, unless that's what we're
    5168              :              looking for.  */
    5169    120556820 :           if (paradoxical_subreg_p (p->exp)
    5170      2803739 :               && ! (src != 0
    5171         3358 :                     && GET_CODE (src) == SUBREG
    5172         3358 :                     && GET_MODE (src) == GET_MODE (p->exp)
    5173         3358 :                     && partial_subreg_p (GET_MODE (SUBREG_REG (src)),
    5174              :                                          GET_MODE (SUBREG_REG (p->exp)))))
    5175         3610 :             continue;
    5176              : 
    5177    120553210 :           if (src && GET_CODE (src) == code && rtx_equal_p (src, p->exp))
    5178              :             src = 0;
    5179      2229929 :           else if (src_folded && GET_CODE (src_folded) == code
    5180     86333421 :                    && rtx_equal_p (src_folded, p->exp))
    5181              :             src_folded = 0;
    5182       780750 :           else if (src_eqv_here && GET_CODE (src_eqv_here) == code
    5183     85509760 :                    && rtx_equal_p (src_eqv_here, p->exp))
    5184              :             src_eqv_here = 0;
    5185       742110 :           else if (src_related && GET_CODE (src_related) == code
    5186     84952840 :                    && rtx_equal_p (src_related, p->exp))
    5187              :             src_related = 0;
    5188              : 
    5189              :           /* This is the same as the destination of the insns, we want
    5190              :              to prefer it.  The code below will then give it a negative
    5191              :              cost.  */
    5192    120553210 :           if (!dest_related
    5193    120553210 :               && GET_CODE (dest) == code && rtx_equal_p (p->exp, dest))
    5194       225300 :             dest_related = p->exp;
    5195              :         }
    5196              : 
    5197              :       /* Find the cheapest valid equivalent, trying all the available
    5198              :          possibilities.  Prefer items not in the hash table to ones
    5199              :          that are when they are equal cost.  Note that we can never
    5200              :          worsen an insn as the current contents will also succeed.
    5201              :          If we find an equivalent identical to the destination, use it as best,
    5202              :          since this insn will probably be eliminated in that case.  */
    5203    198857988 :       if (src)
    5204              :         {
    5205    164073276 :           if (rtx_equal_p (src, dest))
    5206              :             src_cost = src_regcost = -1;
    5207              :           else
    5208              :             {
    5209    164073270 :               src_cost = COST (src, mode);
    5210    164073270 :               src_regcost = approx_reg_cost (src);
    5211              :             }
    5212              :         }
    5213              : 
    5214    198857988 :       if (src_eqv_here)
    5215              :         {
    5216      5611589 :           if (rtx_equal_p (src_eqv_here, dest))
    5217              :             src_eqv_cost = src_eqv_regcost = -1;
    5218              :           else
    5219              :             {
    5220      5611589 :               src_eqv_cost = COST (src_eqv_here, mode);
    5221      5611589 :               src_eqv_regcost = approx_reg_cost (src_eqv_here);
    5222              :             }
    5223              :         }
    5224              : 
    5225    198857988 :       if (src_folded)
    5226              :         {
    5227      3866532 :           if (rtx_equal_p (src_folded, dest))
    5228              :             src_folded_cost = src_folded_regcost = -1;
    5229              :           else
    5230              :             {
    5231      3853734 :               src_folded_cost = COST (src_folded, mode);
    5232      3853734 :               src_folded_regcost = approx_reg_cost (src_folded);
    5233              :             }
    5234              :         }
    5235              : 
    5236    198857988 :       if (dest_related)
    5237              :         {
    5238              :           src_related_cost = src_related_regcost = -1;
    5239              :           /* Handle it as src_related.  */
    5240              :           src_related = dest_related;
    5241              :         }
    5242    198632688 :       else if (src_related)
    5243              :         {
    5244       473896 :           src_related_cost = COST (src_related, mode);
    5245       473896 :           src_related_regcost = approx_reg_cost (src_related);
    5246              : 
    5247              :           /* If a const-anchor is used to synthesize a constant that
    5248              :              normally requires multiple instructions then slightly prefer
    5249              :              it over the original sequence.  These instructions are likely
    5250              :              to become redundant now.  We can't compare against the cost
    5251              :              of src_eqv_here because, on MIPS for example, multi-insn
    5252              :              constants have zero cost; they are assumed to be hoisted from
    5253              :              loops.  */
    5254       473896 :           if (src_related_is_const_anchor
    5255       473896 :               && src_related_cost == src_cost
    5256         6129 :               && src_eqv_here)
    5257         6127 :             src_related_cost--;
    5258              :         }
    5259              : 
    5260              :       /* If this was an indirect jump insn, a known label will really be
    5261              :          cheaper even though it looks more expensive.  */
    5262    198857988 :       if (dest == pc_rtx && src_const && GET_CODE (src_const) == LABEL_REF)
    5263    198857988 :         src_folded = src_const, src_folded_cost = src_folded_regcost = -1;
    5264              : 
    5265              :       /* Terminate loop when replacement made.  This must terminate since
    5266              :          the current contents will be tested and will always be valid.  */
    5267    204229272 :       while (!is_fake_set)
    5268              :         {
    5269              :           rtx trial;
    5270              : 
    5271              :           /* Skip invalid entries.  */
    5272     36271923 :           while (elt && !REG_P (elt->exp)
    5273    212544068 :                  && ! exp_equiv_p (elt->exp, elt->exp, 1, false))
    5274           12 :             elt = elt->next_same_value;
    5275              : 
    5276              :           /* A paradoxical subreg would be bad here: it'll be the right
    5277              :              size, but later may be adjusted so that the upper bits aren't
    5278              :              what we want.  So reject it.  */
    5279    203555414 :           if (elt != 0
    5280     36271911 :               && paradoxical_subreg_p (elt->exp)
    5281              :               /* It is okay, though, if the rtx we're trying to match
    5282              :                  will ignore any of the bits we can't predict.  */
    5283    203556588 :               && ! (src != 0
    5284         1174 :                     && GET_CODE (src) == SUBREG
    5285         1174 :                     && GET_MODE (src) == GET_MODE (elt->exp)
    5286         1174 :                     && partial_subreg_p (GET_MODE (SUBREG_REG (src)),
    5287              :                                          GET_MODE (SUBREG_REG (elt->exp)))))
    5288              :             {
    5289         1174 :               elt = elt->next_same_value;
    5290         1174 :               continue;
    5291              :             }
    5292              : 
    5293    203553066 :           if (elt)
    5294              :             {
    5295     36270737 :               src_elt_cost = elt->cost;
    5296     36270737 :               src_elt_regcost = elt->regcost;
    5297              :             }
    5298              : 
    5299              :           /* Find cheapest and skip it for the next time.   For items
    5300              :              of equal cost, use this order:
    5301              :              src_folded, src, src_eqv, src_related and hash table entry.  */
    5302    203553066 :           if (src_folded
    5303      8123093 :               && preferable (src_folded_cost, src_folded_regcost,
    5304              :                              src_cost, src_regcost) <= 0
    5305      6254009 :               && preferable (src_folded_cost, src_folded_regcost,
    5306              :                              src_eqv_cost, src_eqv_regcost) <= 0
    5307      5366500 :               && preferable (src_folded_cost, src_folded_regcost,
    5308              :                              src_related_cost, src_related_regcost) <= 0
    5309    208915608 :               && preferable (src_folded_cost, src_folded_regcost,
    5310              :                              src_elt_cost, src_elt_regcost) <= 0)
    5311              :             trial = src_folded, src_folded_cost = MAX_COST;
    5312    198685349 :           else if (src
    5313    162914440 :                    && preferable (src_cost, src_regcost,
    5314              :                                   src_eqv_cost, src_eqv_regcost) <= 0
    5315    161362857 :                    && preferable (src_cost, src_regcost,
    5316              :                                   src_related_cost, src_related_regcost) <= 0
    5317    360021865 :                    && preferable (src_cost, src_regcost,
    5318              :                                   src_elt_cost, src_elt_regcost) <= 0)
    5319              :             trial = src, src_cost = MAX_COST;
    5320     37419278 :           else if (src_eqv_here
    5321      1761735 :                    && preferable (src_eqv_cost, src_eqv_regcost,
    5322              :                                   src_related_cost, src_related_regcost) <= 0
    5323     39175309 :                    && preferable (src_eqv_cost, src_eqv_regcost,
    5324              :                                   src_elt_cost, src_elt_regcost) <= 0)
    5325              :             trial = src_eqv_here, src_eqv_cost = MAX_COST;
    5326     35877242 :           else if (src_related
    5327     35877242 :                    && preferable (src_related_cost, src_related_regcost,
    5328              :                                   src_elt_cost, src_elt_regcost) <= 0)
    5329              :             trial = src_related, src_related_cost = MAX_COST;
    5330              :           else
    5331              :             {
    5332     35632147 :               trial = elt->exp;
    5333     35632147 :               elt = elt->next_same_value;
    5334     35632147 :               src_elt_cost = MAX_COST;
    5335              :             }
    5336              : 
    5337              :           /* Try to optimize
    5338              :              (set (reg:M N) (const_int A))
    5339              :              (set (reg:M2 O) (const_int B))
    5340              :              (set (zero_extract:M2 (reg:M N) (const_int C) (const_int D))
    5341              :                   (reg:M2 O)).  */
    5342    203553066 :           if (GET_CODE (SET_DEST (sets[i].rtl)) == ZERO_EXTRACT
    5343         4033 :               && CONST_INT_P (trial)
    5344          737 :               && CONST_INT_P (XEXP (SET_DEST (sets[i].rtl), 1))
    5345          737 :               && CONST_INT_P (XEXP (SET_DEST (sets[i].rtl), 2))
    5346          586 :               && REG_P (XEXP (SET_DEST (sets[i].rtl), 0))
    5347          106 :               && (known_ge
    5348              :                   (GET_MODE_PRECISION (GET_MODE (SET_DEST (sets[i].rtl))),
    5349              :                    INTVAL (XEXP (SET_DEST (sets[i].rtl), 1))))
    5350    203553172 :               && ((unsigned) INTVAL (XEXP (SET_DEST (sets[i].rtl), 1))
    5351          106 :                   + (unsigned) INTVAL (XEXP (SET_DEST (sets[i].rtl), 2))
    5352              :                   <= HOST_BITS_PER_WIDE_INT))
    5353              :             {
    5354          106 :               rtx dest_reg = XEXP (SET_DEST (sets[i].rtl), 0);
    5355          106 :               rtx width = XEXP (SET_DEST (sets[i].rtl), 1);
    5356          106 :               rtx pos = XEXP (SET_DEST (sets[i].rtl), 2);
    5357          106 :               unsigned int dest_hash = HASH (dest_reg, GET_MODE (dest_reg));
    5358          106 :               struct table_elt *dest_elt
    5359          106 :                 = lookup (dest_reg, dest_hash, GET_MODE (dest_reg));
    5360          106 :               rtx dest_cst = NULL;
    5361              : 
    5362          106 :               if (dest_elt)
    5363          153 :                 for (p = dest_elt->first_same_value; p; p = p->next_same_value)
    5364          104 :                   if (p->is_const && CONST_INT_P (p->exp))
    5365              :                     {
    5366              :                       dest_cst = p->exp;
    5367              :                       break;
    5368              :                     }
    5369           57 :               if (dest_cst)
    5370              :                 {
    5371            8 :                   HOST_WIDE_INT val = INTVAL (dest_cst);
    5372            8 :                   HOST_WIDE_INT mask;
    5373            8 :                   unsigned int shift;
    5374              :                   /* This is the mode of DEST_CST as well.  */
    5375            8 :                   scalar_int_mode dest_mode
    5376            8 :                     = as_a <scalar_int_mode> (GET_MODE (dest_reg));
    5377            8 :                   if (BITS_BIG_ENDIAN)
    5378              :                     shift = GET_MODE_PRECISION (dest_mode)
    5379              :                             - INTVAL (pos) - INTVAL (width);
    5380              :                   else
    5381            8 :                     shift = INTVAL (pos);
    5382            8 :                   if (INTVAL (width) == HOST_BITS_PER_WIDE_INT)
    5383              :                     mask = HOST_WIDE_INT_M1;
    5384              :                   else
    5385            8 :                     mask = (HOST_WIDE_INT_1 << INTVAL (width)) - 1;
    5386            8 :                   val &= ~(mask << shift);
    5387            8 :                   val |= (INTVAL (trial) & mask) << shift;
    5388            8 :                   val = trunc_int_for_mode (val, dest_mode);
    5389            8 :                   validate_unshare_change (insn, &SET_DEST (sets[i].rtl),
    5390              :                                            dest_reg, 1);
    5391            8 :                   validate_unshare_change (insn, &SET_SRC (sets[i].rtl),
    5392              :                                            GEN_INT (val), 1);
    5393            8 :                   if (apply_change_group ())
    5394              :                     {
    5395            8 :                       rtx note = find_reg_note (insn, REG_EQUAL, NULL_RTX);
    5396            8 :                       if (note)
    5397              :                         {
    5398            0 :                           remove_note (insn, note);
    5399            0 :                           df_notes_rescan (insn);
    5400              :                         }
    5401            8 :                       src_eqv = NULL_RTX;
    5402            8 :                       src_eqv_elt = NULL;
    5403            8 :                       src_eqv_volatile = 0;
    5404            8 :                       src_eqv_in_memory = 0;
    5405            8 :                       src_eqv_hash = 0;
    5406            8 :                       repeat = true;
    5407            8 :                       break;
    5408              :                     }
    5409              :                 }
    5410              :             }
    5411              : 
    5412              :           /* We don't normally have an insn matching (set (pc) (pc)), so
    5413              :              check for this separately here.  We will delete such an
    5414              :              insn below.
    5415              : 
    5416              :              For other cases such as a table jump or conditional jump
    5417              :              where we know the ultimate target, go ahead and replace the
    5418              :              operand.  While that may not make a valid insn, we will
    5419              :              reemit the jump below (and also insert any necessary
    5420              :              barriers).  */
    5421    200602201 :           if (n_sets == 1 && dest == pc_rtx
    5422    223960594 :               && (trial == pc_rtx
    5423     20394962 :                   || (GET_CODE (trial) == LABEL_REF
    5424        10978 :                       && ! condjump_p (insn))))
    5425              :             {
    5426              :               /* Don't substitute non-local labels, this confuses CFG.  */
    5427        15295 :               if (GET_CODE (trial) == LABEL_REF
    5428        13935 :                   && LABEL_REF_NONLOCAL_P (trial))
    5429         1360 :                 continue;
    5430              : 
    5431        12575 :               SET_SRC (sets[i].rtl) = trial;
    5432        12575 :               cse_jumps_altered = true;
    5433        12575 :               break;
    5434              :             }
    5435              : 
    5436              :           /* Similarly, lots of targets don't allow no-op
    5437              :              (set (mem x) (mem x)) moves.  Even (set (reg x) (reg x))
    5438              :              might be impossible for certain registers (like CC registers).  */
    5439    203539123 :           else if (n_sets == 1
    5440    200588266 :                    && !CALL_P (insn)
    5441    200098316 :                    && (MEM_P (trial) || REG_P (trial))
    5442     81048263 :                    && rtx_equal_p (trial, dest)
    5443       215115 :                    && !side_effects_p (dest)
    5444       215111 :                    && (cfun->can_delete_dead_exceptions
    5445        49373 :                        || insn_nothrow_p (insn))
    5446              :                    /* We can only remove the later store if the earlier aliases
    5447              :                       at least all accesses the later one.  */
    5448    203743773 :                    && (!MEM_P (trial)
    5449        25405 :                        || ((MEM_ALIAS_SET (dest) == MEM_ALIAS_SET (trial)
    5450         8620 :                             || alias_set_subset_of (MEM_ALIAS_SET (dest),
    5451         8620 :                                                     MEM_ALIAS_SET (trial)))
    5452        17245 :                             && (!MEM_EXPR (trial)
    5453        16264 :                                 || refs_same_for_tbaa_p (MEM_EXPR (trial),
    5454        16264 :                                                          MEM_EXPR (dest))))))
    5455              :             {
    5456       193511 :               SET_SRC (sets[i].rtl) = trial;
    5457       193511 :               noop_insn = true;
    5458       193511 :               break;
    5459              :             }
    5460              : 
    5461              :           /* Reject certain invalid forms of CONST that we create.  */
    5462    203345612 :           else if (CONSTANT_P (trial)
    5463     33579643 :                    && GET_CODE (trial) == CONST
    5464              :                    /* Reject cases that will cause decode_rtx_const to
    5465              :                       die.  On the alpha when simplifying a switch, we
    5466              :                       get (const (truncate (minus (label_ref)
    5467              :                       (label_ref)))).  */
    5468       589530 :                    && (GET_CODE (XEXP (trial, 0)) == TRUNCATE
    5469              :                        /* Likewise on IA-64, except without the
    5470              :                           truncate.  */
    5471       589530 :                        || (GET_CODE (XEXP (trial, 0)) == MINUS
    5472            0 :                            && GET_CODE (XEXP (XEXP (trial, 0), 0)) == LABEL_REF
    5473            0 :                            && GET_CODE (XEXP (XEXP (trial, 0), 1)) == LABEL_REF)))
    5474              :             /* Do nothing for this case.  */
    5475              :             ;
    5476              : 
    5477              :           /* Do not replace anything with a MEM, except the replacement
    5478              :              is a no-op.  This allows this loop to terminate.  */
    5479    203345612 :           else if (MEM_P (trial) && !rtx_equal_p (trial, SET_SRC(sets[i].rtl)))
    5480              :             /* Do nothing for this case.  */
    5481              :             ;
    5482              : 
    5483              :           /* Look for a substitution that makes a valid insn.  */
    5484    203261465 :           else if (validate_unshare_change (insn, &SET_SRC (sets[i].rtl),
    5485              :                                             trial, 0))
    5486              :             {
    5487    197976862 :               rtx new_rtx = canon_reg (SET_SRC (sets[i].rtl), insn);
    5488              : 
    5489              :               /* The result of apply_change_group can be ignored; see
    5490              :                  canon_reg.  */
    5491              : 
    5492    197976862 :               validate_change (insn, &SET_SRC (sets[i].rtl), new_rtx, 1);
    5493    197976862 :               apply_change_group ();
    5494              : 
    5495    197976862 :               break;
    5496              :             }
    5497              : 
    5498              :           /* If the current function uses a constant pool and this is a
    5499              :              constant, try making a pool entry. Put it in src_folded
    5500              :              unless we already have done this since that is where it
    5501              :              likely came from.  */
    5502              : 
    5503      5284603 :           else if (crtl->uses_const_pool
    5504      3875148 :                    && CONSTANT_P (trial)
    5505      2869312 :                    && !CONST_INT_P (trial)
    5506      2850689 :                    && (src_folded == 0 || !MEM_P (src_folded))
    5507      1975784 :                    && GET_MODE_CLASS (mode) != MODE_CC
    5508      1975784 :                    && mode != VOIDmode)
    5509              :             {
    5510      1975784 :               src_folded = force_const_mem (mode, trial);
    5511      1975784 :               if (src_folded)
    5512              :                 {
    5513      1975137 :                   src_folded_cost = COST (src_folded, mode);
    5514      1975137 :                   src_folded_regcost = approx_reg_cost (src_folded);
    5515              :                 }
    5516              :             }
    5517              :         }
    5518              : 
    5519              :       /* If we changed the insn too much, handle this set from scratch.  */
    5520    198182948 :       if (repeat)
    5521              :         {
    5522            8 :           i--;
    5523            8 :           continue;
    5524              :         }
    5525              : 
    5526    198857980 :       src = SET_SRC (sets[i].rtl);
    5527              : 
    5528              :       /* In general, it is good to have a SET with SET_SRC == SET_DEST.
    5529              :          However, there is an important exception:  If both are registers
    5530              :          that are not the head of their equivalence class, replace SET_SRC
    5531              :          with the head of the class.  If we do not do this, we will have
    5532              :          both registers live over a portion of the basic block.  This way,
    5533              :          their lifetimes will likely abut instead of overlapping.  */
    5534    198857980 :       if (!is_fake_set
    5535    198182948 :           && REG_P (dest)
    5536    346380478 :           && REGNO_QTY_VALID_P (REGNO (dest)))
    5537              :         {
    5538      8353443 :           int dest_q = REG_QTY (REGNO (dest));
    5539      8353443 :           struct qty_table_elem *dest_ent = &qty_table[dest_q];
    5540              : 
    5541      8353443 :           if (dest_ent->mode == GET_MODE (dest)
    5542      6501101 :               && dest_ent->first_reg != REGNO (dest)
    5543       119574 :               && REG_P (src) && REGNO (src) == REGNO (dest)
    5544              :               /* Don't do this if the original insn had a hard reg as
    5545              :                  SET_SRC or SET_DEST.  */
    5546         4387 :               && (!REG_P (sets[i].src)
    5547         3366 :                   || REGNO (sets[i].src) >= FIRST_PSEUDO_REGISTER)
    5548      8357816 :               && (!REG_P (dest) || REGNO (dest) >= FIRST_PSEUDO_REGISTER))
    5549              :             /* We can't call canon_reg here because it won't do anything if
    5550              :                SRC is a hard register.  */
    5551              :             {
    5552         4373 :               int src_q = REG_QTY (REGNO (src));
    5553         4373 :               struct qty_table_elem *src_ent = &qty_table[src_q];
    5554         4373 :               int first = src_ent->first_reg;
    5555         4373 :               rtx new_src
    5556              :                 = (first >= FIRST_PSEUDO_REGISTER
    5557         4373 :                    ? regno_reg_rtx[first] : gen_rtx_REG (GET_MODE (src), first));
    5558              : 
    5559              :               /* We must use validate-change even for this, because this
    5560              :                  might be a special no-op instruction, suitable only to
    5561              :                  tag notes onto.  */
    5562         4373 :               if (validate_change (insn, &SET_SRC (sets[i].rtl), new_src, 0))
    5563              :                 {
    5564         4373 :                   src = new_src;
    5565              :                   /* If we had a constant that is cheaper than what we are now
    5566              :                      setting SRC to, use that constant.  We ignored it when we
    5567              :                      thought we could make this into a no-op.  */
    5568         1556 :                   if (src_const && COST (src_const, mode) < COST (src, mode)
    5569         4373 :                       && validate_change (insn, &SET_SRC (sets[i].rtl),
    5570              :                                           src_const, 0))
    5571              :                     src = src_const;
    5572              :                 }
    5573              :             }
    5574              :         }
    5575              : 
    5576              :       /* If we made a change, recompute SRC values.  */
    5577    198857980 :       if (src != sets[i].src)
    5578              :         {
    5579      3398915 :           do_not_record = 0;
    5580      3398915 :           hash_arg_in_memory = 0;
    5581      3398915 :           sets[i].src = src;
    5582      3398915 :           sets[i].src_hash = HASH (src, mode);
    5583      3398915 :           sets[i].src_volatile = do_not_record;
    5584      3398915 :           sets[i].src_in_memory = hash_arg_in_memory;
    5585      3398915 :           sets[i].src_elt = lookup (src, sets[i].src_hash, mode);
    5586              :         }
    5587              : 
    5588              :       /* If this is a single SET, we are setting a register, and we have an
    5589              :          equivalent constant, we want to add a REG_EQUAL note if the constant
    5590              :          is different from the source.  We don't want to do it for a constant
    5591              :          pseudo since verifying that this pseudo hasn't been eliminated is a
    5592              :          pain; moreover such a note won't help anything.
    5593              : 
    5594              :          Avoid a REG_EQUAL note for (CONST (MINUS (LABEL_REF) (LABEL_REF)))
    5595              :          which can be created for a reference to a compile time computable
    5596              :          entry in a jump table.  */
    5597    198857980 :       if (n_sets == 1
    5598    195659585 :           && REG_P (dest)
    5599    145308514 :           && src_const
    5600     29953607 :           && !REG_P (src_const)
    5601     29926915 :           && !(GET_CODE (src_const) == SUBREG
    5602            0 :                && REG_P (SUBREG_REG (src_const)))
    5603     29926915 :           && !(GET_CODE (src_const) == CONST
    5604       389088 :                && GET_CODE (XEXP (src_const, 0)) == MINUS
    5605            0 :                && GET_CODE (XEXP (XEXP (src_const, 0), 0)) == LABEL_REF
    5606            0 :                && GET_CODE (XEXP (XEXP (src_const, 0), 1)) == LABEL_REF)
    5607    228784895 :           && !rtx_equal_p (src, src_const))
    5608              :         {
    5609              :           /* Make sure that the rtx is not shared.  */
    5610      7886240 :           src_const = copy_rtx (src_const);
    5611              : 
    5612              :           /* Record the actual constant value in a REG_EQUAL note,
    5613              :              making a new one if one does not already exist.  */
    5614      7886240 :           set_unique_reg_note (insn, REG_EQUAL, src_const);
    5615      7886240 :           df_notes_rescan (insn);
    5616              :         }
    5617              : 
    5618              :       /* Now deal with the destination.  */
    5619    198857980 :       do_not_record = 0;
    5620              : 
    5621              :       /* Look within any ZERO_EXTRACT to the MEM or REG within it.  */
    5622    198857980 :       while (GET_CODE (dest) == SUBREG
    5623    198875868 :              || GET_CODE (dest) == ZERO_EXTRACT
    5624    399402108 :              || GET_CODE (dest) == STRICT_LOW_PART)
    5625      1672285 :         dest = XEXP (dest, 0);
    5626              : 
    5627    198857980 :       sets[i].inner_dest = dest;
    5628              : 
    5629    198857980 :       if (MEM_P (dest))
    5630              :         {
    5631              : #ifdef PUSH_ROUNDING
    5632              :           /* Stack pushes invalidate the stack pointer.  */
    5633     29285228 :           rtx addr = XEXP (dest, 0);
    5634     29285228 :           if (GET_RTX_CLASS (GET_CODE (addr)) == RTX_AUTOINC
    5635      5550978 :               && XEXP (addr, 0) == stack_pointer_rtx)
    5636      5550978 :             invalidate (stack_pointer_rtx, VOIDmode);
    5637              : #endif
    5638     29285228 :           dest = fold_rtx (dest, insn);
    5639              :         }
    5640              : 
    5641              :       /* Compute the hash code of the destination now,
    5642              :          before the effects of this instruction are recorded,
    5643              :          since the register values used in the address computation
    5644              :          are those before this instruction.  */
    5645    198857980 :       sets[i].dest_hash = HASH (dest, mode);
    5646              : 
    5647              :       /* Don't enter a bit-field in the hash table
    5648              :          because the value in it after the store
    5649              :          may not equal what was stored, due to truncation.  */
    5650              : 
    5651    198857980 :       if (GET_CODE (SET_DEST (sets[i].rtl)) == ZERO_EXTRACT)
    5652              :         {
    5653         4025 :           rtx width = XEXP (SET_DEST (sets[i].rtl), 1);
    5654              : 
    5655         4025 :           if (src_const != 0 && CONST_INT_P (src_const)
    5656          729 :               && CONST_INT_P (width)
    5657          729 :               && INTVAL (width) < HOST_BITS_PER_WIDE_INT
    5658          729 :               && ! (INTVAL (src_const)
    5659          729 :                     & (HOST_WIDE_INT_M1U << INTVAL (width))))
    5660              :             /* Exception: if the value is constant,
    5661              :                and it won't be truncated, record it.  */
    5662              :             ;
    5663              :           else
    5664              :             {
    5665              :               /* This is chosen so that the destination will be invalidated
    5666              :                  but no new value will be recorded.
    5667              :                  We must invalidate because sometimes constant
    5668              :                  values can be recorded for bitfields.  */
    5669         3297 :               sets[i].src_elt = 0;
    5670         3297 :               sets[i].src_volatile = 1;
    5671         3297 :               src_eqv = 0;
    5672         3297 :               src_eqv_elt = 0;
    5673              :             }
    5674              :         }
    5675              : 
    5676              :       /* If only one set in a JUMP_INSN and it is now a no-op, we can delete
    5677              :          the insn.  */
    5678    198853955 :       else if (n_sets == 1 && dest == pc_rtx && src == pc_rtx)
    5679              :         {
    5680              :           /* One less use of the label this insn used to jump to.  */
    5681        12574 :           cse_cfg_altered |= delete_insn_and_edges (insn);
    5682        12574 :           cse_jumps_altered = true;
    5683              :           /* No more processing for this set.  */
    5684        12574 :           sets[i].rtl = 0;
    5685              :         }
    5686              : 
    5687              :       /* Similarly for no-op moves.  */
    5688    198841381 :       else if (noop_insn)
    5689              :         {
    5690       193511 :           if (cfun->can_throw_non_call_exceptions && can_throw_internal (insn))
    5691            0 :             cse_cfg_altered = true;
    5692       193511 :           cse_cfg_altered |= delete_insn_and_edges (insn);
    5693              :           /* No more processing for this set.  */
    5694       193511 :           sets[i].rtl = 0;
    5695              :         }
    5696              : 
    5697              :       /* If this SET is now setting PC to a label, we know it used to
    5698              :          be a conditional or computed branch.  */
    5699     20381091 :       else if (dest == pc_rtx && GET_CODE (src) == LABEL_REF
    5700    198657488 :                && !LABEL_REF_NONLOCAL_P (src))
    5701              :         {
    5702              :           /* We reemit the jump in as many cases as possible just in
    5703              :              case the form of an unconditional jump is significantly
    5704              :              different than a computed jump or conditional jump.
    5705              : 
    5706              :              If this insn has multiple sets, then reemitting the
    5707              :              jump is nontrivial.  So instead we just force rerecognition
    5708              :              and hope for the best.  */
    5709         9618 :           if (n_sets == 1)
    5710              :             {
    5711         9618 :               rtx_jump_insn *new_rtx;
    5712         9618 :               rtx note;
    5713              : 
    5714         9618 :               rtx_insn *seq = targetm.gen_jump (XEXP (src, 0));
    5715         9618 :               new_rtx = emit_jump_insn_before (seq, insn);
    5716         9618 :               JUMP_LABEL (new_rtx) = XEXP (src, 0);
    5717         9618 :               LABEL_NUSES (label_ref_label (src))++;
    5718              : 
    5719              :               /* Make sure to copy over REG_NON_LOCAL_GOTO.  */
    5720         9618 :               note = find_reg_note (insn, REG_NON_LOCAL_GOTO, 0);
    5721         9618 :               if (note)
    5722              :                 {
    5723            0 :                   XEXP (note, 1) = NULL_RTX;
    5724            0 :                   REG_NOTES (new_rtx) = note;
    5725              :                 }
    5726              : 
    5727         9618 :               cse_cfg_altered |= delete_insn_and_edges (insn);
    5728         9618 :               insn = new_rtx;
    5729              :             }
    5730              :           else
    5731            0 :             INSN_CODE (insn) = -1;
    5732              : 
    5733              :           /* Do not bother deleting any unreachable code, let jump do it.  */
    5734         9618 :           cse_jumps_altered = true;
    5735         9618 :           sets[i].rtl = 0;
    5736              :         }
    5737              : 
    5738              :       /* If destination is volatile, invalidate it and then do no further
    5739              :          processing for this assignment.  */
    5740              : 
    5741    198638252 :       else if (do_not_record)
    5742              :         {
    5743     56447247 :           invalidate_dest (dest);
    5744     56447247 :           sets[i].rtl = 0;
    5745              :         }
    5746              : 
    5747    198857980 :       if (sets[i].rtl != 0 && dest != SET_DEST (sets[i].rtl))
    5748              :         {
    5749      1742184 :           do_not_record = 0;
    5750      1742184 :           sets[i].dest_hash = HASH (SET_DEST (sets[i].rtl), mode);
    5751      1742184 :           if (do_not_record)
    5752              :             {
    5753         1181 :               invalidate_dest (SET_DEST (sets[i].rtl));
    5754         1181 :               sets[i].rtl = 0;
    5755              :             }
    5756              :         }
    5757              :     }
    5758              : 
    5759              :   /* Now enter all non-volatile source expressions in the hash table
    5760              :      if they are not already present.
    5761              :      Record their equivalence classes in src_elt.
    5762              :      This way we can insert the corresponding destinations into
    5763              :      the same classes even if the actual sources are no longer in them
    5764              :      (having been invalidated).  */
    5765              : 
    5766      5491672 :   if (src_eqv && src_eqv_elt == 0 && sets[0].rtl != 0 && ! src_eqv_volatile
    5767    416180726 :       && ! rtx_equal_p (src_eqv, SET_DEST (sets[0].rtl)))
    5768              :     {
    5769      4494291 :       struct table_elt *elt;
    5770      4494291 :       struct table_elt *classp = sets[0].src_elt;
    5771      4494291 :       rtx dest = SET_DEST (sets[0].rtl);
    5772      4494291 :       machine_mode eqvmode = GET_MODE (dest);
    5773              : 
    5774      4494291 :       if (GET_CODE (dest) == STRICT_LOW_PART)
    5775              :         {
    5776            0 :           eqvmode = GET_MODE (SUBREG_REG (XEXP (dest, 0)));
    5777            0 :           classp = 0;
    5778              :         }
    5779      4494291 :       if (insert_regs (src_eqv, classp, false))
    5780              :         {
    5781       155868 :           rehash_using_reg (src_eqv);
    5782       155868 :           src_eqv_hash = HASH (src_eqv, eqvmode);
    5783              :         }
    5784      4494291 :       elt = insert (src_eqv, classp, src_eqv_hash, eqvmode);
    5785      4494291 :       elt->in_memory = src_eqv_in_memory;
    5786      4494291 :       src_eqv_elt = elt;
    5787              : 
    5788              :       /* Check to see if src_eqv_elt is the same as a set source which
    5789              :          does not yet have an elt, and if so set the elt of the set source
    5790              :          to src_eqv_elt.  */
    5791      8988582 :       for (i = 0; i < n_sets; i++)
    5792      8988582 :         if (sets[i].rtl && sets[i].src_elt == 0
    5793      8855955 :             && rtx_equal_p (SET_SRC (sets[i].rtl), src_eqv))
    5794        98736 :           sets[i].src_elt = src_eqv_elt;
    5795              :     }
    5796              : 
    5797    610544415 :   for (i = 0; i < n_sets; i++)
    5798    341051829 :     if (sets[i].rtl && ! sets[i].src_volatile
    5799    327253277 :         && ! rtx_equal_p (SET_SRC (sets[i].rtl), SET_DEST (sets[i].rtl)))
    5800              :       {
    5801    128390274 :         if (GET_CODE (SET_DEST (sets[i].rtl)) == STRICT_LOW_PART)
    5802              :           {
    5803              :             /* REG_EQUAL in setting a STRICT_LOW_PART
    5804              :                gives an equivalent for the entire destination register,
    5805              :                not just for the subreg being stored in now.
    5806              :                This is a more interesting equivalence, so we arrange later
    5807              :                to treat the entire reg as the destination.  */
    5808        13863 :             sets[i].src_elt = src_eqv_elt;
    5809        13863 :             sets[i].src_hash = src_eqv_hash;
    5810              :           }
    5811              :         else
    5812              :           {
    5813              :             /* Insert source and constant equivalent into hash table, if not
    5814              :                already present.  */
    5815    128376411 :             struct table_elt *classp = src_eqv_elt;
    5816    128376411 :             rtx src = sets[i].src;
    5817    128376411 :             rtx dest = SET_DEST (sets[i].rtl);
    5818    268171986 :             machine_mode mode
    5819    128376411 :               = GET_MODE (src) == VOIDmode ? GET_MODE (dest) : GET_MODE (src);
    5820              : 
    5821              :             /* It's possible that we have a source value known to be
    5822              :                constant but don't have a REG_EQUAL note on the insn.
    5823              :                Lack of a note will mean src_eqv_elt will be NULL.  This
    5824              :                can happen where we've generated a SUBREG to access a
    5825              :                CONST_INT that is already in a register in a wider mode.
    5826              :                Ensure that the source expression is put in the proper
    5827              :                constant class.  */
    5828    128376411 :             if (!classp)
    5829    122926701 :               classp = sets[i].src_const_elt;
    5830              : 
    5831    128376411 :             if (sets[i].src_elt == 0)
    5832              :               {
    5833    105953003 :                 struct table_elt *elt;
    5834              : 
    5835              :                 /* Note that these insert_regs calls cannot remove
    5836              :                    any of the src_elt's, because they would have failed to
    5837              :                    match if not still valid.  */
    5838    105953003 :                 if (insert_regs (src, classp, false))
    5839              :                   {
    5840     17623098 :                     rehash_using_reg (src);
    5841     17623098 :                     sets[i].src_hash = HASH (src, mode);
    5842              :                   }
    5843    105953003 :                 elt = insert (src, classp, sets[i].src_hash, mode);
    5844    105953003 :                 elt->in_memory = sets[i].src_in_memory;
    5845              :                 /* If inline asm has any clobbers, ensure we only reuse
    5846              :                    existing inline asms and never try to put the ASM_OPERANDS
    5847              :                    into an insn that isn't inline asm.  */
    5848    105953003 :                 if (GET_CODE (src) == ASM_OPERANDS
    5849        20711 :                     && GET_CODE (x) == PARALLEL)
    5850        20693 :                   elt->cost = MAX_COST;
    5851    105953003 :                 sets[i].src_elt = classp = elt;
    5852              :               }
    5853    153658115 :             if (sets[i].src_const && sets[i].src_const_elt == 0
    5854     14989747 :                 && src != sets[i].src_const
    5855    130612016 :                 && ! rtx_equal_p (sets[i].src_const, src))
    5856      2235605 :               sets[i].src_elt = insert (sets[i].src_const, classp,
    5857      2235605 :                                         sets[i].src_const_hash, mode);
    5858              :           }
    5859              :       }
    5860     70467706 :     else if (sets[i].src_elt == 0)
    5861              :       /* If we did not insert the source into the hash table (e.g., it was
    5862              :          volatile), note the equivalence class for the REG_EQUAL value, if any,
    5863              :          so that the destination goes into that class.  */
    5864     57773795 :       sets[i].src_elt = src_eqv_elt;
    5865              : 
    5866              :   /* Record destination addresses in the hash table.  This allows us to
    5867              :      check if they are invalidated by other sets.  */
    5868    610544415 :   for (i = 0; i < n_sets; i++)
    5869              :     {
    5870    198857980 :       if (sets[i].rtl)
    5871              :         {
    5872    142193849 :           rtx x = sets[i].inner_dest;
    5873    142193849 :           struct table_elt *elt;
    5874    142193849 :           machine_mode mode;
    5875    142193849 :           unsigned hash;
    5876              : 
    5877    142193849 :           if (MEM_P (x))
    5878              :             {
    5879     22476167 :               x = XEXP (x, 0);
    5880     22476167 :               mode = GET_MODE (x);
    5881     22476167 :               hash = HASH (x, mode);
    5882     22476167 :               elt = lookup (x, hash, mode);
    5883     22476167 :               if (!elt)
    5884              :                 {
    5885     19599948 :                   if (insert_regs (x, NULL, false))
    5886              :                     {
    5887      2245646 :                       rtx dest = SET_DEST (sets[i].rtl);
    5888              : 
    5889      2245646 :                       rehash_using_reg (x);
    5890      2245646 :                       hash = HASH (x, mode);
    5891      2245646 :                       sets[i].dest_hash = HASH (dest, GET_MODE (dest));
    5892              :                     }
    5893     19599948 :                   elt = insert (x, NULL, hash, mode);
    5894              :                 }
    5895              : 
    5896     22476167 :               sets[i].dest_addr_elt = elt;
    5897              :             }
    5898              :           else
    5899    119717682 :             sets[i].dest_addr_elt = NULL;
    5900              :         }
    5901              :     }
    5902              : 
    5903    411686435 :   invalidate_from_clobbers (insn);
    5904              : 
    5905              :   /* Some registers are invalidated by subroutine calls.  Memory is
    5906              :      invalidated by non-constant calls.  */
    5907              : 
    5908    411686435 :   if (auto call = dyn_cast<rtx_call_insn *> (insn))
    5909              :     {
    5910     15857452 :       if (!(RTL_CONST_OR_PURE_CALL_P (call)))
    5911     13715915 :         invalidate_memory ();
    5912              :       else
    5913              :         /* For const/pure calls, invalidate any argument slots, because
    5914              :            those are owned by the callee.  */
    5915      6362469 :         for (tem = CALL_INSN_FUNCTION_USAGE (call); tem; tem = XEXP (tem, 1))
    5916      4220932 :           if (GET_CODE (XEXP (tem, 0)) == USE
    5917      4220931 :               && MEM_P (XEXP (XEXP (tem, 0), 0)))
    5918        71405 :             invalidate (XEXP (XEXP (tem, 0), 0), VOIDmode);
    5919     15857452 :       invalidate_for_call (call);
    5920              :     }
    5921              : 
    5922              :   /* Now invalidate everything set by this instruction.
    5923              :      If a SUBREG or other funny destination is being set,
    5924              :      sets[i].rtl is still nonzero, so here we invalidate the reg
    5925              :      a part of which is being set.  */
    5926              : 
    5927    610544415 :   for (i = 0; i < n_sets; i++)
    5928    198857980 :     if (sets[i].rtl)
    5929              :       {
    5930              :         /* We can't use the inner dest, because the mode associated with
    5931              :            a ZERO_EXTRACT is significant.  */
    5932    142193849 :         rtx dest = SET_DEST (sets[i].rtl);
    5933              : 
    5934              :         /* Needed for registers to remove the register from its
    5935              :            previous quantity's chain.
    5936              :            Needed for memory if this is a nonvarying address, unless
    5937              :            we have just done an invalidate_memory that covers even those.  */
    5938    142193849 :         if (REG_P (dest) || GET_CODE (dest) == SUBREG)
    5939    119700648 :           invalidate (dest, VOIDmode);
    5940     22493201 :         else if (MEM_P (dest))
    5941     22476167 :           invalidate (dest, VOIDmode);
    5942        17034 :         else if (GET_CODE (dest) == STRICT_LOW_PART
    5943         3171 :                  || GET_CODE (dest) == ZERO_EXTRACT)
    5944        16912 :           invalidate (XEXP (dest, 0), GET_MODE (dest));
    5945              :       }
    5946              : 
    5947              :   /* Don't cse over a call to setjmp; on some machines (eg VAX)
    5948              :      the regs restored by the longjmp come from a later time
    5949              :      than the setjmp.  */
    5950    411686435 :   if (CALL_P (insn) && find_reg_note (insn, REG_SETJMP, NULL))
    5951              :     {
    5952         2281 :       flush_hash_table ();
    5953         2281 :       goto done;
    5954              :     }
    5955              : 
    5956              :   /* Make sure registers mentioned in destinations
    5957              :      are safe for use in an expression to be inserted.
    5958              :      This removes from the hash table
    5959              :      any invalid entry that refers to one of these registers.
    5960              : 
    5961              :      We don't care about the return value from mention_regs because
    5962              :      we are going to hash the SET_DEST values unconditionally.  */
    5963              : 
    5964    610542134 :   for (i = 0; i < n_sets; i++)
    5965              :     {
    5966    198857980 :       if (sets[i].rtl)
    5967              :         {
    5968    142193849 :           rtx x = SET_DEST (sets[i].rtl);
    5969              : 
    5970    142193849 :           if (!REG_P (x))
    5971     24132188 :             mention_regs (x);
    5972              :           else
    5973              :             {
    5974              :               /* We used to rely on all references to a register becoming
    5975              :                  inaccessible when a register changes to a new quantity,
    5976              :                  since that changes the hash code.  However, that is not
    5977              :                  safe, since after HASH_SIZE new quantities we get a
    5978              :                  hash 'collision' of a register with its own invalid
    5979              :                  entries.  And since SUBREGs have been changed not to
    5980              :                  change their hash code with the hash code of the register,
    5981              :                  it wouldn't work any longer at all.  So we have to check
    5982              :                  for any invalid references lying around now.
    5983              :                  This code is similar to the REG case in mention_regs,
    5984              :                  but it knows that reg_tick has been incremented, and
    5985              :                  it leaves reg_in_table as -1 .  */
    5986    118061661 :               unsigned int regno = REGNO (x);
    5987    118061661 :               unsigned int endregno = END_REGNO (x);
    5988    118061661 :               unsigned int i;
    5989              : 
    5990    236123322 :               for (i = regno; i < endregno; i++)
    5991              :                 {
    5992    118061661 :                   if (REG_IN_TABLE (i) >= 0)
    5993              :                     {
    5994     13017047 :                       remove_invalid_refs (i);
    5995     13017047 :                       REG_IN_TABLE (i) = -1;
    5996              :                     }
    5997              :                 }
    5998              :             }
    5999              :         }
    6000              :     }
    6001              : 
    6002              :   /* We may have just removed some of the src_elt's from the hash table.
    6003              :      So replace each one with the current head of the same class.
    6004              :      Also check if destination addresses have been removed.  */
    6005              : 
    6006    610542134 :   for (i = 0; i < n_sets; i++)
    6007    198857980 :     if (sets[i].rtl)
    6008              :       {
    6009    142193849 :         if (sets[i].dest_addr_elt
    6010    142193849 :             && sets[i].dest_addr_elt->first_same_value == 0)
    6011              :           {
    6012              :             /* The elt was removed, which means this destination is not
    6013              :                valid after this instruction.  */
    6014            0 :             sets[i].rtl = NULL_RTX;
    6015              :           }
    6016    142193849 :         else if (sets[i].src_elt && sets[i].src_elt->first_same_value == 0)
    6017              :           /* If elt was removed, find current head of same class,
    6018              :              or 0 if nothing remains of that class.  */
    6019              :           {
    6020     10883684 :             struct table_elt *elt = sets[i].src_elt;
    6021              : 
    6022     10883684 :             while (elt && elt->prev_same_value)
    6023              :               elt = elt->prev_same_value;
    6024              : 
    6025     21695099 :             while (elt && elt->first_same_value == 0)
    6026     10847126 :               elt = elt->next_same_value;
    6027     10847973 :             sets[i].src_elt = elt ? elt->first_same_value : 0;
    6028              :           }
    6029              :       }
    6030              : 
    6031              :   /* Now insert the destinations into their equivalence classes.  */
    6032              : 
    6033    610542134 :   for (i = 0; i < n_sets; i++)
    6034    198857980 :     if (sets[i].rtl)
    6035              :       {
    6036    142193849 :         rtx dest = SET_DEST (sets[i].rtl);
    6037    142193849 :         struct table_elt *elt;
    6038              : 
    6039              :         /* Don't record value if we are not supposed to risk allocating
    6040              :            floating-point values in registers that might be wider than
    6041              :            memory.  */
    6042    166795239 :         if ((flag_float_store
    6043        12898 :              && MEM_P (dest)
    6044         4396 :              && FLOAT_MODE_P (GET_MODE (dest)))
    6045              :             /* Don't record BLKmode values, because we don't know the
    6046              :                size of it, and can't be sure that other BLKmode values
    6047              :                have the same or smaller size.  */
    6048    142191153 :             || GET_MODE (dest) == BLKmode
    6049              :             /* If we didn't put a REG_EQUAL value or a source into the hash
    6050              :                table, there is no point is recording DEST.  */
    6051    284385002 :             || sets[i].src_elt == 0)
    6052     24601390 :           continue;
    6053              : 
    6054              :         /* STRICT_LOW_PART isn't part of the value BEING set,
    6055              :            and neither is the SUBREG inside it.
    6056              :            Note that in this case SETS[I].SRC_ELT is really SRC_EQV_ELT.  */
    6057    117592459 :         if (GET_CODE (dest) == STRICT_LOW_PART)
    6058            0 :           dest = SUBREG_REG (XEXP (dest, 0));
    6059              : 
    6060    117592459 :         if (REG_P (dest) || GET_CODE (dest) == SUBREG)
    6061              :           /* Registers must also be inserted into chains for quantities.  */
    6062     95488189 :           if (insert_regs (dest, sets[i].src_elt, true))
    6063              :             {
    6064              :               /* If `insert_regs' changes something, the hash code must be
    6065              :                  recalculated.  */
    6066     94892991 :               rehash_using_reg (dest);
    6067     94892991 :               sets[i].dest_hash = HASH (dest, GET_MODE (dest));
    6068              :             }
    6069              : 
    6070              :         /* If DEST is a paradoxical SUBREG, don't record DEST since the bits
    6071              :            outside the mode of GET_MODE (SUBREG_REG (dest)) are undefined.  */
    6072    117592459 :         if (paradoxical_subreg_p (dest))
    6073        66303 :           continue;
    6074              : 
    6075    352578468 :         elt = insert (dest, sets[i].src_elt,
    6076    117526156 :                       sets[i].dest_hash, GET_MODE (dest));
    6077              : 
    6078              :         /* If this is a constant, insert the constant anchors with the
    6079              :            equivalent register-offset expressions using register DEST.  */
    6080    117526156 :         if (targetm.const_anchor
    6081            0 :             && REG_P (dest)
    6082            0 :             && SCALAR_INT_MODE_P (GET_MODE (dest))
    6083    117526156 :             && GET_CODE (sets[i].src_elt->exp) == CONST_INT)
    6084            0 :           insert_const_anchors (dest, sets[i].src_elt->exp, GET_MODE (dest));
    6085              : 
    6086    117526156 :         elt->in_memory = (MEM_P (sets[i].inner_dest)
    6087    117526156 :                           && !MEM_READONLY_P (sets[i].inner_dest));
    6088              : 
    6089              :         /* If we have (set (subreg:m1 (reg:m2 foo) 0) (bar:m1)), M1 is no
    6090              :            narrower than M2, and both M1 and M2 are the same number of words,
    6091              :            we are also doing (set (reg:m2 foo) (subreg:m2 (bar:m1) 0)) so
    6092              :            make that equivalence as well.
    6093              : 
    6094              :            However, BAR may have equivalences for which gen_lowpart
    6095              :            will produce a simpler value than gen_lowpart applied to
    6096              :            BAR (e.g., if BAR was ZERO_EXTENDed from M2), so we will scan all
    6097              :            BAR's equivalences.  If we don't get a simplified form, make
    6098              :            the SUBREG.  It will not be used in an equivalence, but will
    6099              :            cause two similar assignments to be detected.
    6100              : 
    6101              :            Note the loop below will find SUBREG_REG (DEST) since we have
    6102              :            already entered SRC and DEST of the SET in the table.  */
    6103              : 
    6104    117526156 :         if (GET_CODE (dest) == SUBREG
    6105              :             && (known_equal_after_align_down
    6106    200406346 :                 (GET_MODE_SIZE (GET_MODE (SUBREG_REG (dest))) - 1,
    6107      3011784 :                  GET_MODE_SIZE (GET_MODE (dest)) - 1,
    6108      1505892 :                  UNITS_PER_WORD))
    6109        94753 :             && !partial_subreg_p (dest)
    6110    117568630 :             && sets[i].src_elt != 0)
    6111              :           {
    6112        42474 :             machine_mode new_mode = GET_MODE (SUBREG_REG (dest));
    6113        42474 :             struct table_elt *elt, *classp = 0;
    6114              : 
    6115       471116 :             for (elt = sets[i].src_elt->first_same_value; elt;
    6116       428642 :                  elt = elt->next_same_value)
    6117              :               {
    6118       428642 :                 rtx new_src = 0;
    6119       428642 :                 unsigned src_hash;
    6120       428642 :                 struct table_elt *src_elt;
    6121              : 
    6122              :                 /* Ignore invalid entries.  */
    6123       428642 :                 if (!REG_P (elt->exp)
    6124       428642 :                     && ! exp_equiv_p (elt->exp, elt->exp, 1, false))
    6125            0 :                   continue;
    6126              : 
    6127              :                 /* We may have already been playing subreg games.  If the
    6128              :                    mode is already correct for the destination, use it.  */
    6129       428642 :                 if (GET_MODE (elt->exp) == new_mode)
    6130              :                   new_src = elt->exp;
    6131              :                 else
    6132              :                   {
    6133       428642 :                     poly_uint64 byte
    6134       428642 :                       = subreg_lowpart_offset (new_mode, GET_MODE (dest));
    6135       428642 :                     new_src = simplify_gen_subreg (new_mode, elt->exp,
    6136       428642 :                                                    GET_MODE (dest), byte);
    6137              :                   }
    6138              : 
    6139              :                 /* The call to simplify_gen_subreg fails if the value
    6140              :                    is VOIDmode, yet we can't do any simplification, e.g.
    6141              :                    for EXPR_LISTs denoting function call results.
    6142              :                    It is invalid to construct a SUBREG with a VOIDmode
    6143              :                    SUBREG_REG, hence a zero new_src means we can't do
    6144              :                    this substitution.  */
    6145       428642 :                 if (! new_src)
    6146            6 :                   continue;
    6147              : 
    6148       428636 :                 src_hash = HASH (new_src, new_mode);
    6149       428636 :                 src_elt = lookup (new_src, src_hash, new_mode);
    6150              : 
    6151              :                 /* Put the new source in the hash table is if isn't
    6152              :                    already.  */
    6153       428636 :                 if (src_elt == 0)
    6154              :                   {
    6155        54985 :                     if (insert_regs (new_src, classp, false))
    6156              :                       {
    6157            0 :                         rehash_using_reg (new_src);
    6158            0 :                         src_hash = HASH (new_src, new_mode);
    6159              :                       }
    6160        54985 :                     src_elt = insert (new_src, classp, src_hash, new_mode);
    6161        54985 :                     src_elt->in_memory = elt->in_memory;
    6162        54985 :                     if (GET_CODE (new_src) == ASM_OPERANDS
    6163            0 :                         && elt->cost == MAX_COST)
    6164            0 :                       src_elt->cost = MAX_COST;
    6165              :                   }
    6166       373651 :                 else if (classp && classp != src_elt->first_same_value)
    6167              :                   /* Show that two things that we've seen before are
    6168              :                      actually the same.  */
    6169          161 :                   merge_equiv_classes (src_elt, classp);
    6170              : 
    6171       428636 :                 classp = src_elt->first_same_value;
    6172              :                 /* Ignore invalid entries.  */
    6173       428636 :                 while (classp
    6174       428636 :                        && !REG_P (classp->exp)
    6175       806365 :                        && ! exp_equiv_p (classp->exp, classp->exp, 1, false))
    6176            0 :                   classp = classp->next_same_value;
    6177              :               }
    6178              :           }
    6179              :       }
    6180              : 
    6181              :   /* Special handling for (set REG0 REG1) where REG0 is the
    6182              :      "cheapest", cheaper than REG1.  After cse, REG1 will probably not
    6183              :      be used in the sequel, so (if easily done) change this insn to
    6184              :      (set REG1 REG0) and replace REG1 with REG0 in the previous insn
    6185              :      that computed their value.  Then REG1 will become a dead store
    6186              :      and won't cloud the situation for later optimizations.
    6187              : 
    6188              :      Do not make this change if REG1 is a hard register, because it will
    6189              :      then be used in the sequel and we may be changing a two-operand insn
    6190              :      into a three-operand insn.
    6191              : 
    6192              :      Also do not do this if we are operating on a copy of INSN.  */
    6193              : 
    6194    607343739 :   if (n_sets == 1 && sets[0].rtl)
    6195    139224217 :     try_back_substitute_reg (sets[0].rtl, insn);
    6196              : 
    6197    411686435 : done:;
    6198    411686435 : }
    6199              : 
    6200              : /* Remove from the hash table all expressions that reference memory.  */
    6201              : 
    6202              : static void
    6203     13715915 : invalidate_memory (void)
    6204              : {
    6205     13715915 :   int i;
    6206     13715915 :   struct table_elt *p, *next;
    6207              : 
    6208    452625195 :   for (i = 0; i < HASH_SIZE; i++)
    6209    632809438 :     for (p = table[i]; p; p = next)
    6210              :       {
    6211    193900158 :         next = p->next_same_hash;
    6212    193900158 :         if (p->in_memory)
    6213     19858225 :           remove_from_table (p, i);
    6214              :       }
    6215     13715915 : }
    6216              : 
    6217              : /* Perform invalidation on the basis of everything about INSN,
    6218              :    except for invalidating the actual places that are SET in it.
    6219              :    This includes the places CLOBBERed, and anything that might
    6220              :    alias with something that is SET or CLOBBERed.  */
    6221              : 
    6222              : static void
    6223    411686435 : invalidate_from_clobbers (rtx_insn *insn)
    6224              : {
    6225    411686435 :   rtx x = PATTERN (insn);
    6226              : 
    6227    411686435 :   if (GET_CODE (x) == CLOBBER)
    6228              :     {
    6229        69609 :       rtx ref = XEXP (x, 0);
    6230        69609 :       if (ref)
    6231              :         {
    6232        69609 :           if (REG_P (ref) || GET_CODE (ref) == SUBREG
    6233        13142 :               || MEM_P (ref))
    6234        69609 :             invalidate (ref, VOIDmode);
    6235            0 :           else if (GET_CODE (ref) == STRICT_LOW_PART
    6236            0 :                    || GET_CODE (ref) == ZERO_EXTRACT)
    6237            0 :             invalidate (XEXP (ref, 0), GET_MODE (ref));
    6238              :         }
    6239              :     }
    6240    411616826 :   else if (GET_CODE (x) == PARALLEL)
    6241              :     {
    6242     30964455 :       int i;
    6243     94286150 :       for (i = XVECLEN (x, 0) - 1; i >= 0; i--)
    6244              :         {
    6245     63321695 :           rtx y = XVECEXP (x, 0, i);
    6246     63321695 :           if (GET_CODE (y) == CLOBBER)
    6247              :             {
    6248     30557419 :               rtx ref = XEXP (y, 0);
    6249     30557419 :               if (REG_P (ref) || GET_CODE (ref) == SUBREG
    6250       259449 :                   || MEM_P (ref))
    6251     30361894 :                 invalidate (ref, VOIDmode);
    6252       195525 :               else if (GET_CODE (ref) == STRICT_LOW_PART
    6253       195525 :                        || GET_CODE (ref) == ZERO_EXTRACT)
    6254            0 :                 invalidate (XEXP (ref, 0), GET_MODE (ref));
    6255              :             }
    6256              :         }
    6257              :     }
    6258    411686435 : }
    6259              : 
    6260              : /* Perform invalidation on the basis of everything about INSN.
    6261              :    This includes the places CLOBBERed, and anything that might
    6262              :    alias with something that is SET or CLOBBERed.  */
    6263              : 
    6264              : static void
    6265    411686435 : invalidate_from_sets_and_clobbers (rtx_insn *insn)
    6266              : {
    6267    411686435 :   rtx tem;
    6268    411686435 :   rtx x = PATTERN (insn);
    6269              : 
    6270    411686435 :   if (CALL_P (insn))
    6271              :     {
    6272     46958816 :       for (tem = CALL_INSN_FUNCTION_USAGE (insn); tem; tem = XEXP (tem, 1))
    6273              :         {
    6274     31101364 :           rtx temx = XEXP (tem, 0);
    6275     31101364 :           if (GET_CODE (temx) == CLOBBER)
    6276            0 :             invalidate (SET_DEST (temx), VOIDmode);
    6277              :         }
    6278              :     }
    6279              : 
    6280              :   /* Ensure we invalidate the destination register of a CALL insn.
    6281              :      This is necessary for machines where this register is a fixed_reg,
    6282              :      because no other code would invalidate it.  */
    6283    411686435 :   if (GET_CODE (x) == SET && GET_CODE (SET_SRC (x)) == CALL)
    6284      7315589 :     invalidate (SET_DEST (x), VOIDmode);
    6285              : 
    6286    404370846 :   else if (GET_CODE (x) == PARALLEL)
    6287              :     {
    6288     31679152 :       int i;
    6289              : 
    6290     96435054 :       for (i = XVECLEN (x, 0) - 1; i >= 0; i--)
    6291              :         {
    6292     64755902 :           rtx y = XVECEXP (x, 0, i);
    6293     64755902 :           if (GET_CODE (y) == CLOBBER)
    6294              :             {
    6295     31276929 :               rtx clobbered = XEXP (y, 0);
    6296              : 
    6297     31276929 :               if (REG_P (clobbered)
    6298       264258 :                   || GET_CODE (clobbered) == SUBREG)
    6299     31012671 :                 invalidate (clobbered, VOIDmode);
    6300       264258 :               else if (GET_CODE (clobbered) == STRICT_LOW_PART
    6301       264258 :                        || GET_CODE (clobbered) == ZERO_EXTRACT)
    6302            0 :                 invalidate (XEXP (clobbered, 0), GET_MODE (clobbered));
    6303              :             }
    6304     33478973 :           else if (GET_CODE (y) == SET && GET_CODE (SET_SRC (y)) == CALL)
    6305        10207 :             invalidate (SET_DEST (y), VOIDmode);
    6306              :         }
    6307              :     }
    6308              : 
    6309              :   /* Any single register constraint may introduce a conflict, if the associated
    6310              :      hard register is live.  For example:
    6311              : 
    6312              :      r100=%1
    6313              :      r101=42
    6314              :      r102=exp(r101)
    6315              : 
    6316              :      If the first operand r101 of exp is constrained to hard register %1, then
    6317              :      r100 cannot be trivially substituted by %1 in the following since %1 got
    6318              :      clobbered.  Such conflicts may stem from single register classes as well
    6319              :      as hard register constraints.  Since prior RA we do not know which
    6320              :      alternative will be chosen, be conservative and consider any such hard
    6321              :      register from any alternative as a potential clobber.  */
    6322    411686435 :   extract_insn (insn);
    6323    884443519 :   for (int nop = recog_data.n_operands - 1; nop >= 0; --nop)
    6324              :     {
    6325    472757084 :       int c;
    6326    472757084 :       const char *p = recog_data.constraints[nop];
    6327  15816243464 :       for (; (c = *p); p += CONSTRAINT_LEN (c, p))
    6328  15343486380 :         if (c == ',')
    6329              :           ;
    6330  10027518259 :         else if (c == '{')
    6331              :           {
    6332          190 :             int regno = decode_hard_reg_constraint (p);
    6333          190 :             machine_mode mode = recog_data.operand_mode[nop];
    6334          190 :             invalidate_reg (gen_rtx_REG (mode, regno));
    6335              :           }
    6336              :     }
    6337    411686435 : }
    6338              : 
    6339              : static rtx cse_process_note (rtx);
    6340              : 
    6341              : /* A simplify_replace_fn_rtx callback for cse_process_note.  Process X,
    6342              :    part of the REG_NOTES of an insn.  Replace any registers with either
    6343              :    an equivalent constant or the canonical form of the register.
    6344              :    Only replace addresses if the containing MEM remains valid.
    6345              : 
    6346              :    Return the replacement for X, or null if it should be simplified
    6347              :    recursively.  */
    6348              : 
    6349              : static rtx
    6350     29802404 : cse_process_note_1 (rtx x, const_rtx, void *)
    6351              : {
    6352     29802404 :   if (MEM_P (x))
    6353              :     {
    6354      2072722 :       validate_change (x, &XEXP (x, 0), cse_process_note (XEXP (x, 0)), false);
    6355      1036361 :       return x;
    6356              :     }
    6357              : 
    6358     28766043 :   if (REG_P (x))
    6359              :     {
    6360      6290412 :       int i = REG_QTY (REGNO (x));
    6361              : 
    6362              :       /* Return a constant or a constant register.  */
    6363      6290412 :       if (REGNO_QTY_VALID_P (REGNO (x)))
    6364              :         {
    6365      1666810 :           struct qty_table_elem *ent = &qty_table[i];
    6366              : 
    6367      1666810 :           if (ent->const_rtx != NULL_RTX
    6368        21484 :               && (CONSTANT_P (ent->const_rtx)
    6369        15740 :                   || REG_P (ent->const_rtx)))
    6370              :             {
    6371         5744 :               rtx new_rtx = gen_lowpart (GET_MODE (x), ent->const_rtx);
    6372         5744 :               if (new_rtx)
    6373         5744 :                 return copy_rtx (new_rtx);
    6374              :             }
    6375              :         }
    6376              : 
    6377              :       /* Otherwise, canonicalize this register.  */
    6378      6284668 :       return canon_reg (x, NULL);
    6379              :     }
    6380              : 
    6381              :   return NULL_RTX;
    6382              : }
    6383              : 
    6384              : /* Process X, part of the REG_NOTES of an insn.  Replace any registers in it
    6385              :    with either an equivalent constant or the canonical form of the register.
    6386              :    Only replace addresses if the containing MEM remains valid.  */
    6387              : 
    6388              : static rtx
    6389     10129015 : cse_process_note (rtx x)
    6390              : {
    6391      1036361 :   return simplify_replace_fn_rtx (x, NULL_RTX, cse_process_note_1, NULL);
    6392              : }
    6393              : 
    6394              : 
    6395              : /* Find a path in the CFG, starting with FIRST_BB to perform CSE on.
    6396              : 
    6397              :    DATA is a pointer to a struct cse_basic_block_data, that is used to
    6398              :    describe the path.
    6399              :    It is filled with a queue of basic blocks, starting with FIRST_BB
    6400              :    and following a trace through the CFG.
    6401              : 
    6402              :    If all paths starting at FIRST_BB have been followed, or no new path
    6403              :    starting at FIRST_BB can be constructed, this function returns FALSE.
    6404              :    Otherwise, DATA->path is filled and the function returns TRUE indicating
    6405              :    that a path to follow was found.
    6406              : 
    6407              :    If FOLLOW_JUMPS is false, the maximum path length is 1 and the only
    6408              :    block in the path will be FIRST_BB.  */
    6409              : 
    6410              : static bool
    6411     40356459 : cse_find_path (basic_block first_bb, struct cse_basic_block_data *data,
    6412              :                bool follow_jumps)
    6413              : {
    6414     40356459 :   basic_block bb;
    6415     40356459 :   edge e;
    6416     40356459 :   int path_size;
    6417              : 
    6418     40356459 :   bitmap_set_bit (cse_visited_basic_blocks, first_bb->index);
    6419              : 
    6420              :   /* See if there is a previous path.  */
    6421     40356459 :   path_size = data->path_size;
    6422              : 
    6423              :   /* There is a previous path.  Make sure it started with FIRST_BB.  */
    6424     40356459 :   if (path_size)
    6425     21697211 :     gcc_assert (data->path[0].bb == first_bb);
    6426              : 
    6427              :   /* There was only one basic block in the last path.  Clear the path and
    6428              :      return, so that paths starting at another basic block can be tried.  */
    6429     21697211 :   if (path_size == 1)
    6430              :     {
    6431     14556004 :       path_size = 0;
    6432     14556004 :       goto done;
    6433              :     }
    6434              : 
    6435              :   /* If the path was empty from the beginning, construct a new path.  */
    6436     25800455 :   if (path_size == 0)
    6437     18659248 :     data->path[path_size++].bb = first_bb;
    6438              :   else
    6439              :     {
    6440              :       /* Otherwise, path_size must be equal to or greater than 2, because
    6441              :          a previous path exists that is at least two basic blocks long.
    6442              : 
    6443              :          Update the previous branch path, if any.  If the last branch was
    6444              :          previously along the branch edge, take the fallthrough edge now.  */
    6445     15784975 :       while (path_size >= 2)
    6446              :         {
    6447     11681731 :           basic_block last_bb_in_path, previous_bb_in_path;
    6448     11681731 :           edge e;
    6449              : 
    6450     11681731 :           --path_size;
    6451     11681731 :           last_bb_in_path = data->path[path_size].bb;
    6452     11681731 :           previous_bb_in_path = data->path[path_size - 1].bb;
    6453              : 
    6454              :           /* If we previously followed a path along the branch edge, try
    6455              :              the fallthru edge now.  */
    6456     20325499 :           if (EDGE_COUNT (previous_bb_in_path->succs) == 2
    6457     11342561 :               && any_condjump_p (BB_END (previous_bb_in_path))
    6458     11342561 :               && (e = find_edge (previous_bb_in_path, last_bb_in_path))
    6459     23024292 :               && e == BRANCH_EDGE (previous_bb_in_path))
    6460              :             {
    6461      4052361 :               bb = FALLTHRU_EDGE (previous_bb_in_path)->dest;
    6462      4052361 :               if (bb != EXIT_BLOCK_PTR_FOR_FN (cfun)
    6463      4052361 :                   && single_pred_p (bb)
    6464              :                   /* We used to assert here that we would only see blocks
    6465              :                      that we have not visited yet.  But we may end up
    6466              :                      visiting basic blocks twice if the CFG has changed
    6467              :                      in this run of cse_main, because when the CFG changes
    6468              :                      the topological sort of the CFG also changes.  A basic
    6469              :                      blocks that previously had more than two predecessors
    6470              :                      may now have a single predecessor, and become part of
    6471              :                      a path that starts at another basic block.
    6472              : 
    6473              :                      We still want to visit each basic block only once, so
    6474              :                      halt the path here if we have already visited BB.  */
    6475      7090324 :                   && !bitmap_bit_p (cse_visited_basic_blocks, bb->index))
    6476              :                 {
    6477      3037963 :                   bitmap_set_bit (cse_visited_basic_blocks, bb->index);
    6478      3037963 :                   data->path[path_size++].bb = bb;
    6479      3037963 :                   break;
    6480              :                 }
    6481              :             }
    6482              : 
    6483      8643768 :           data->path[path_size].bb = NULL;
    6484              :         }
    6485              : 
    6486              :       /* If only one block remains in the path, bail.  */
    6487      7141207 :       if (path_size == 1)
    6488              :         {
    6489      4103244 :           path_size = 0;
    6490      4103244 :           goto done;
    6491              :         }
    6492              :     }
    6493              : 
    6494              :   /* Extend the path if possible.  */
    6495     21697211 :   if (follow_jumps)
    6496              :     {
    6497     12389896 :       bb = data->path[path_size - 1].bb;
    6498     21037348 :       while (bb && path_size < param_max_cse_path_length)
    6499              :         {
    6500     20878595 :           if (single_succ_p (bb))
    6501      9141023 :             e = single_succ_edge (bb);
    6502     11737572 :           else if (EDGE_COUNT (bb->succs) == 2
    6503     11719339 :                    && any_condjump_p (BB_END (bb)))
    6504              :             {
    6505              :               /* First try to follow the branch.  If that doesn't lead
    6506              :                  to a useful path, follow the fallthru edge.  */
    6507      9252997 :               e = BRANCH_EDGE (bb);
    6508      9252997 :               if (!single_pred_p (e->dest))
    6509      5190301 :                 e = FALLTHRU_EDGE (bb);
    6510              :             }
    6511              :           else
    6512              :             e = NULL;
    6513              : 
    6514     18394020 :           if (e
    6515     18394020 :               && !((e->flags & EDGE_ABNORMAL_CALL) && cfun->has_nonlocal_label)
    6516     18392975 :               && e->dest != EXIT_BLOCK_PTR_FOR_FN (cfun)
    6517     16206963 :               && single_pred_p (e->dest)
    6518              :               /* Avoid visiting basic blocks twice.  The large comment
    6519              :                  above explains why this can happen.  */
    6520     27041484 :               && !bitmap_bit_p (cse_visited_basic_blocks, e->dest->index))
    6521              :             {
    6522      8647452 :               basic_block bb2 = e->dest;
    6523      8647452 :               bitmap_set_bit (cse_visited_basic_blocks, bb2->index);
    6524      8647452 :               data->path[path_size++].bb = bb2;
    6525      8647452 :               bb = bb2;
    6526              :             }
    6527              :           else
    6528              :             bb = NULL;
    6529              :         }
    6530              :     }
    6531              : 
    6532      9307315 : done:
    6533     40356459 :   data->path_size = path_size;
    6534     40356459 :   return path_size != 0;
    6535              : }
    6536              : 
    6537              : /* Dump the path in DATA to file F.  NSETS is the number of sets
    6538              :    in the path.  */
    6539              : 
    6540              : static void
    6541          317 : cse_dump_path (struct cse_basic_block_data *data, int nsets, FILE *f)
    6542              : {
    6543          317 :   int path_entry;
    6544              : 
    6545          317 :   fprintf (f, ";; Following path with %d sets: ", nsets);
    6546         1119 :   for (path_entry = 0; path_entry < data->path_size; path_entry++)
    6547          485 :     fprintf (f, "%d ", (data->path[path_entry].bb)->index);
    6548          317 :   fputc ('\n', f);
    6549          317 :   fflush (f);
    6550          317 : }
    6551              : 
    6552              : 
    6553              : /* Return true if BB has exception handling successor edges.  */
    6554              : 
    6555              : static bool
    6556      9070785 : have_eh_succ_edges (basic_block bb)
    6557              : {
    6558      9070785 :   edge e;
    6559      9070785 :   edge_iterator ei;
    6560              : 
    6561     21357205 :   FOR_EACH_EDGE (e, ei, bb->succs)
    6562     13285433 :     if (e->flags & EDGE_EH)
    6563              :       return true;
    6564              : 
    6565              :   return false;
    6566              : }
    6567              : 
    6568              : /* Record vec_duplicate match for SET in the same basic block, if any.  */
    6569              : static void
    6570    205521918 : cse_prescan_cache_vec_dup (struct cse_basic_block_data *data, basic_block bb,
    6571              :                            rtx_insn *insn, rtx set)
    6572              : {
    6573    205521918 :   rtx src = SET_SRC (set);
    6574    205521918 :   rtx dest = SET_DEST (set);
    6575    205521918 :   machine_mode mode = GET_MODE (src);
    6576    205521918 :   rtx scalar;
    6577              : 
    6578              :   /* Limit matching to duplicates of the same pseudo register, or an exact
    6579              :      SUBREG of such a pseudo.  */
    6580    205521918 :   if (!vec_duplicate_p (src, &scalar))
    6581    205181509 :     return;
    6582      1029783 :   if (REG_P (scalar))
    6583              :     {
    6584       340145 :       if (HARD_REGISTER_P (scalar))
    6585              :         return;
    6586              :     }
    6587       689638 :   else if (SUBREG_P (scalar))
    6588              :     {
    6589          756 :       if (!REG_P (SUBREG_REG (scalar))
    6590          756 :           || HARD_REGISTER_P (SUBREG_REG (scalar)))
    6591              :         return;
    6592              :     }
    6593              :   else
    6594              :     return;
    6595              : 
    6596              :   /* Check for matching cached vec_duplicates and save the match if found.  */
    6597      1239725 :   for (auto &entry : data->vec_duplicate_cache)
    6598              :     /* Create a match with existing cache entry if both scalar register
    6599              :        pseudos are matching and the new vec_duplicate mode is wider.  */
    6600       435028 :     if (GET_MODE_INNER (mode) == GET_MODE_INNER (entry.widest_mode)
    6601       208766 :         && rtx_equal_p (entry.scalar, scalar)
    6602       218008 :         && entry.last_checked_insn)
    6603              :       {
    6604          494 :         if (REG_P (dest)
    6605         1482 :             && known_gt (GET_MODE_SIZE (mode),
    6606              :                          GET_MODE_SIZE (entry.widest_mode)))
    6607              :           {
    6608           71 :             if (reg_set_between_p (scalar, entry.last_checked_insn, insn))
    6609              :               {
    6610            2 :                 entry.last_checked_insn = NULL;
    6611            2 :                 continue;
    6612              :               }
    6613           69 :             entry.widest_mode = mode;
    6614           69 :             entry.widest_insn = insn;
    6615           69 :             entry.last_checked_insn = insn;
    6616           69 :             entry.related_dups.safe_push (insn);
    6617           69 :             return;
    6618              :           }
    6619         1269 :         else if (known_le (GET_MODE_SIZE (mode),
    6620              :                            GET_MODE_SIZE (entry.widest_mode)))
    6621              :           {
    6622          423 :             if (reg_set_between_p (scalar, entry.last_checked_insn, insn))
    6623              :               {
    6624            0 :                 entry.last_checked_insn = NULL;
    6625            0 :                 continue;
    6626              :               }
    6627          423 :             entry.last_checked_insn = insn;
    6628          423 :             entry.related_dups.safe_push (insn);
    6629          423 :             return;
    6630              :           }
    6631              :       }
    6632              : 
    6633              :   /* Cache vec_duplicate as a new entry if no match was found.  */
    6634       340409 :   cse_vec_duplicate_match new_entry (bb, mode, scalar, insn, NULL, insn);
    6635       340409 :   new_entry.related_dups.safe_push (insn);
    6636       340409 :   data->vec_duplicate_cache.safe_push (new_entry);
    6637       340409 : }
    6638              : 
    6639              : 
    6640              : /* Scan to the end of the path described by DATA.  Return an estimate of
    6641              :    the total number of SETs of all insns in the path.  Also record any
    6642              :    matching vec_duplicate SETs of the same scalar value for different modes.  */
    6643              : 
    6644              : static void
    6645     21697211 : cse_prescan_path (struct cse_basic_block_data *data)
    6646              : {
    6647     21697211 :   int nsets = 0;
    6648     21697211 :   int path_size = data->path_size;
    6649     21697211 :   int path_entry;
    6650              : 
    6651     21697211 :   data->vec_duplicate_matches.truncate (0);
    6652              : 
    6653              :   /* Scan to end of each basic block in the path.  */
    6654     80365973 :   for (path_entry = 0; path_entry < path_size; path_entry++)
    6655              :     {
    6656     36971551 :       basic_block bb;
    6657     36971551 :       rtx_insn *insn;
    6658              : 
    6659     36971551 :       bb = data->path[path_entry].bb;
    6660     36971551 :       data->vec_duplicate_cache.truncate (0);
    6661              : 
    6662    507741470 :       FOR_BB_INSNS (bb, insn)
    6663              :         {
    6664    470769919 :           if (!INSN_P (insn))
    6665     59067691 :             continue;
    6666              : 
    6667    411702228 :           rtx pattern = PATTERN (insn);
    6668              : 
    6669              :           /* A PARALLEL can have lots of SETs in it,
    6670              :              especially if it is really an ASM_OPERANDS.  */
    6671    411702228 :           if (GET_CODE (pattern) == PARALLEL)
    6672              :             {
    6673     31683923 :               int len = XVECLEN (pattern, 0);
    6674              : 
    6675     31683923 :               nsets += len;
    6676     96449367 :               for (int i = 0; i < len; i++)
    6677              :                 {
    6678     64765444 :                   rtx elt = XVECEXP (pattern, 0, i);
    6679     64765444 :                   if (GET_CODE (elt) == SET)
    6680     32635683 :                     cse_prescan_cache_vec_dup (data, bb, insn, elt);
    6681              :                 }
    6682              :             }
    6683              :           else
    6684              :             {
    6685    380018305 :               if (GET_CODE (pattern) == SET)
    6686    172886235 :                 cse_prescan_cache_vec_dup (data, bb, insn, pattern);
    6687    380018305 :               nsets += 1;
    6688              :             }
    6689              :         }
    6690              : 
    6691              :       /* Record any vec_duplicate matches from this basic block.  */
    6692    111255062 :       for (auto &entry : data->vec_duplicate_cache)
    6693              :         {
    6694              :           /* If we recorded no wider vec_duplicate match then skip.  */
    6695       340409 :           if (entry.widest_insn == NULL_RTX)
    6696       340347 :             continue;
    6697              : 
    6698           62 :           data->vec_duplicate_matches.safe_push (entry);
    6699              :         }
    6700              :     }
    6701              : 
    6702     21697211 :   data->nsets = nsets;
    6703     21697211 : }
    6704              : 
    6705              : /* Return true if the pattern of INSN uses a LABEL_REF for which
    6706              :    there isn't a REG_LABEL_OPERAND note.  */
    6707              : 
    6708              : static bool
    6709    411519135 : check_for_label_ref (rtx_insn *insn)
    6710              : {
    6711              :   /* If this insn uses a LABEL_REF and there isn't a REG_LABEL_OPERAND
    6712              :      note for it, we must rerun jump since it needs to place the note.  If
    6713              :      this is a LABEL_REF for a CODE_LABEL that isn't in the insn chain,
    6714              :      don't do this since no REG_LABEL_OPERAND will be added.  */
    6715    411519135 :   subrtx_iterator::array_type array;
    6716   2046260168 :   FOR_EACH_SUBRTX (iter, array, PATTERN (insn), ALL)
    6717              :     {
    6718   1634742349 :       const_rtx x = *iter;
    6719   1634742349 :       if (GET_CODE (x) == LABEL_REF
    6720     20418069 :           && !LABEL_REF_NONLOCAL_P (x)
    6721     20417216 :           && (!JUMP_P (insn)
    6722     20375158 :               || !label_is_jump_target_p (label_ref_label (x), insn))
    6723        42059 :           && LABEL_P (label_ref_label (x))
    6724        41696 :           && INSN_UID (label_ref_label (x)) != 0
    6725   1634784045 :           && !find_reg_note (insn, REG_LABEL_OPERAND, label_ref_label (x)))
    6726         1316 :         return true;
    6727              :     }
    6728    411517819 :   return false;
    6729    411519135 : }
    6730              : 
    6731              : /* Process a single extended basic block described by EBB_DATA.  */
    6732              : 
    6733              : static void
    6734     21190623 : cse_extended_basic_block (struct cse_basic_block_data *ebb_data)
    6735              : {
    6736     21190623 :   int path_size = ebb_data->path_size;
    6737     21190623 :   int path_entry;
    6738     21190623 :   int num_insns = 0;
    6739              : 
    6740              :   /* Allocate the space needed by qty_table.  */
    6741     21190623 :   qty_table = XNEWVEC (struct qty_table_elem, max_qty);
    6742              : 
    6743     21190623 :   new_basic_block ();
    6744     21190623 :   cse_ebb_live_in = df_get_live_in (ebb_data->path[0].bb);
    6745     21190623 :   cse_ebb_live_out = df_get_live_out (ebb_data->path[path_size - 1].bb);
    6746     57651793 :   for (path_entry = 0; path_entry < path_size; path_entry++)
    6747              :     {
    6748     36461170 :       basic_block bb;
    6749     36461170 :       rtx_insn *insn;
    6750              : 
    6751     36461170 :       bb = ebb_data->path[path_entry].bb;
    6752              : 
    6753              :       /* Invalidate recorded information for eh regs if there is an EH
    6754              :          edge pointing to that bb.  */
    6755     36461170 :       if (bb_has_eh_pred (bb))
    6756              :         {
    6757       555333 :           df_ref def;
    6758              : 
    6759      2221332 :           FOR_EACH_ARTIFICIAL_DEF (def, bb->index)
    6760      1110666 :             if (DF_REF_FLAGS (def) & DF_REF_AT_TOP)
    6761      1110666 :               invalidate (DF_REF_REG (def), GET_MODE (DF_REF_REG (def)));
    6762              :         }
    6763              : 
    6764     36461170 :       optimize_this_for_speed_p = optimize_bb_for_speed_p (bb);
    6765    506308644 :       FOR_BB_INSNS (bb, insn)
    6766              :         {
    6767              :           /* If we have processed 1,000 insns, flush the hash table to
    6768              :              avoid extreme quadratic behavior.  We must not include NOTEs
    6769              :              in the count since there may be more of them when generating
    6770              :              debugging information.  If we clear the table at different
    6771              :              times, code generated with -g -O might be different than code
    6772              :              generated with -O but not -g.
    6773              : 
    6774              :              FIXME: This is a real kludge and needs to be done some other
    6775              :                     way.  */
    6776    469847474 :           if (NONDEBUG_INSN_P (insn)
    6777    469847474 :               && num_insns++ > param_max_cse_insns)
    6778              :             {
    6779         6033 :               flush_hash_table ();
    6780         6033 :               num_insns = 0;
    6781              :             }
    6782              : 
    6783    469847474 :           if (INSN_P (insn))
    6784              :             {
    6785              :               /* Process notes first so we have all notes in canonical forms
    6786              :                  when looking for duplicate operations.  */
    6787    411686435 :               bool changed = false;
    6788    651115914 :               for (rtx note = REG_NOTES (insn); note; note = XEXP (note, 1))
    6789    239429479 :                 if (REG_NOTE_KIND (note) == REG_EQUAL)
    6790              :                   {
    6791      9092654 :                     rtx newval = cse_process_note (XEXP (note, 0));
    6792      9092654 :                     if (newval != XEXP (note, 0))
    6793              :                       {
    6794        43371 :                         XEXP (note, 0) = newval;
    6795        43371 :                         changed = true;
    6796              :                       }
    6797              :                   }
    6798    411686435 :               if (changed)
    6799        43371 :                 df_notes_rescan (insn);
    6800              : 
    6801    411686435 :               cse_insn (insn);
    6802              : 
    6803              :               /* If we haven't already found an insn where we added a LABEL_REF,
    6804              :                  check this one.  */
    6805    411686435 :               if (INSN_P (insn) && !recorded_label_ref
    6806    823205570 :                   && check_for_label_ref (insn))
    6807         1316 :                 recorded_label_ref = true;
    6808              :             }
    6809              :         }
    6810              : 
    6811              :       /* With non-call exceptions, we are not always able to update
    6812              :          the CFG properly inside cse_insn.  So clean up possibly
    6813              :          redundant EH edges here.  */
    6814     36461170 :       if (cfun->can_throw_non_call_exceptions && have_eh_succ_edges (bb))
    6815       999013 :         cse_cfg_altered |= purge_dead_edges (bb);
    6816              : 
    6817              :       /* If we changed a conditional jump, we may have terminated
    6818              :          the path we are following.  Check that by verifying that
    6819              :          the edge we would take still exists.  If the edge does
    6820              :          not exist anymore, purge the remainder of the path.
    6821              :          Note that this will cause us to return to the caller.  */
    6822     36461170 :       if (path_entry < path_size - 1)
    6823              :         {
    6824     15273440 :           basic_block next_bb = ebb_data->path[path_entry + 1].bb;
    6825     15273440 :           if (!find_edge (bb, next_bb))
    6826              :             {
    6827         3684 :               do
    6828              :                 {
    6829         3684 :                   path_size--;
    6830              : 
    6831              :                   /* If we truncate the path, we must also reset the
    6832              :                      visited bit on the remaining blocks in the path,
    6833              :                      or we will never visit them at all.  */
    6834         3684 :                   bitmap_clear_bit (cse_visited_basic_blocks,
    6835         3684 :                              ebb_data->path[path_size].bb->index);
    6836         3684 :                   ebb_data->path[path_size].bb = NULL;
    6837              :                 }
    6838         3684 :               while (path_size - 1 != path_entry);
    6839         2893 :               ebb_data->path_size = path_size;
    6840              :             }
    6841              :         }
    6842              : 
    6843              :       /* If this is a conditional jump insn, record any known
    6844              :          equivalences due to the condition being tested.  */
    6845     36461170 :       insn = BB_END (bb);
    6846     36461170 :       if (path_entry < path_size - 1
    6847     51374292 :           && EDGE_COUNT (bb->succs) == 2
    6848     14913122 :           && JUMP_P (insn)
    6849     14913122 :           && single_set (insn)
    6850     14913122 :           && any_condjump_p (insn)
    6851              :           /* single_set may return non-NULL even for multiple sets
    6852              :              if there are REG_UNUSED notes.  record_jump_equiv only
    6853              :              looks at pc_set and doesn't consider other sets that
    6854              :              could affect the value, and the recorded equivalence
    6855              :              can extend the lifetime of the compared REG, so use
    6856              :              also !multiple_sets check to verify it is exactly one
    6857              :              set.  */
    6858     51374292 :           && !multiple_sets (insn))
    6859              :         {
    6860     14913122 :           basic_block next_bb = ebb_data->path[path_entry + 1].bb;
    6861     14913122 :           bool taken = (next_bb == BRANCH_EDGE (bb)->dest);
    6862     14913122 :           record_jump_equiv (insn, taken);
    6863              :         }
    6864              :     }
    6865              : 
    6866     21190623 :   gcc_assert (next_qty <= max_qty);
    6867              : 
    6868     21190623 :   free (qty_table);
    6869     21190623 : }
    6870              : 
    6871              : 
    6872              : /* Perform cse on the instructions of a function.
    6873              :    F is the first instruction.
    6874              :    NREGS is one plus the highest pseudo-reg number used in the instruction.
    6875              : 
    6876              :    Return 2 if jump optimizations should be redone due to simplifications
    6877              :    in conditional jump instructions.
    6878              :    Return 1 if the CFG should be cleaned up because it has been modified.
    6879              :    Return 0 otherwise.  */
    6880              : 
    6881              : static int
    6882      2345959 : cse_main (rtx_insn *f ATTRIBUTE_UNUSED, int nregs)
    6883              : {
    6884      2345959 :   struct cse_basic_block_data ebb_data;
    6885      2345959 :   basic_block bb;
    6886      2345959 :   int *rc_order = XNEWVEC (int, last_basic_block_for_fn (cfun));
    6887      2345959 :   int i, n_blocks;
    6888              : 
    6889              :   /* CSE doesn't use dominane info but can invalidate it in different ways.
    6890              :      For simplicity free dominance info here.  */
    6891      2345959 :   free_dominance_info (CDI_DOMINATORS);
    6892              : 
    6893      2345959 :   df_set_flags (DF_LR_RUN_DCE);
    6894      2345959 :   df_note_add_problem ();
    6895      2345959 :   df_analyze ();
    6896      2345959 :   df_set_flags (DF_DEFER_INSN_RESCAN);
    6897              : 
    6898      2345959 :   reg_scan (get_insns (), max_reg_num ());
    6899      2345959 :   init_cse_reg_info (nregs);
    6900              : 
    6901      2345959 :   ebb_data.path = XNEWVEC (struct branch_path,
    6902              :                            param_max_cse_path_length);
    6903              : 
    6904      2345959 :   cse_cfg_altered = false;
    6905      2345959 :   cse_jumps_altered = false;
    6906      2345959 :   recorded_label_ref = false;
    6907      2345959 :   ebb_data.path_size = 0;
    6908      2345959 :   ebb_data.nsets = 0;
    6909      2345959 :   rtl_hooks = cse_rtl_hooks;
    6910              : 
    6911      2345959 :   init_recog ();
    6912      2345959 :   init_alias_analysis ();
    6913              : 
    6914      2345959 :   reg_eqv_table = XNEWVEC (struct reg_eqv_elem, nregs);
    6915              : 
    6916              :   /* Set up the table of already visited basic blocks.  */
    6917      2345959 :   cse_visited_basic_blocks = sbitmap_alloc (last_basic_block_for_fn (cfun));
    6918      2345959 :   bitmap_clear (cse_visited_basic_blocks);
    6919              : 
    6920              :   /* Loop over basic blocks in reverse completion order (RPO),
    6921              :      excluding the ENTRY and EXIT blocks.  */
    6922      2345959 :   n_blocks = pre_and_rev_post_order_compute (NULL, rc_order, false);
    6923      2345959 :   i = 0;
    6924     23351166 :   while (i < n_blocks)
    6925              :     {
    6926              :       /* Find the first block in the RPO queue that we have not yet
    6927              :          processed before.  */
    6928     29922894 :       do
    6929              :         {
    6930     29922894 :           bb = BASIC_BLOCK_FOR_FN (cfun, rc_order[i++]);
    6931              :         }
    6932     29922894 :       while (bitmap_bit_p (cse_visited_basic_blocks, bb->index)
    6933     48582142 :              && i < n_blocks);
    6934              : 
    6935              :       /* Find all paths starting with BB, and process them.  */
    6936     40356459 :       while (cse_find_path (bb, &ebb_data, flag_cse_follow_jumps))
    6937              :         {
    6938              :           /* Pre-scan the path.  */
    6939     21697211 :           cse_prescan_path (&ebb_data);
    6940              : 
    6941              :           /* If this basic block has no sets, skip it.  */
    6942     21697211 :           if (ebb_data.nsets == 0)
    6943       506588 :             continue;
    6944              : 
    6945              :           /* If prescan discovers any vec_duplicate pairs where a wider
    6946              :              duplicate appears after a narrow duplicate of the same scalar,
    6947              :              move the widest duplicate before the first vec_duplicate of this
    6948              :              scalar and rewrite all related duplicates to reuse it.  */
    6949     63571931 :           for (auto &match : ebb_data.vec_duplicate_matches)
    6950              :             {
    6951           62 :               rtx wide_set = single_set (match.widest_insn);
    6952           62 :               if (!wide_set)
    6953            0 :                 continue;
    6954           62 :               rtx wide_reg = SET_DEST (wide_set);
    6955           62 :               rtx_insn *insert_after = PREV_INSN (match.first_insn);
    6956           62 :               if (!REG_P (wide_reg) || !insert_after)
    6957            0 :                 continue;
    6958              : 
    6959              :               /* Safety check that WIDE_REG is not otherwise used or redefined
    6960              :                  before its original defining insn.  */
    6961           62 :               if (reg_used_between_p (wide_reg, match.first_insn,
    6962           62 :                                       match.widest_insn)
    6963          124 :                   || reg_set_between_p (wide_reg, match.first_insn,
    6964           62 :                                         match.widest_insn))
    6965            0 :                 continue;
    6966              : 
    6967              :               /* Rewrite all related duplicates as a group so we either keep
    6968              :                  the whole transformation or none of it.  */
    6969           62 :               int prev_changes = num_changes_pending ();
    6970           62 :               bool abort_changes = false;
    6971              : 
    6972          319 :               for (auto &dup_insn : match.related_dups)
    6973              :                 {
    6974          133 :                   rtx dup_set;
    6975          133 :                   rtx dup_src;
    6976          133 :                   machine_mode dup_mode;
    6977              : 
    6978          133 :                   if (dup_insn == match.widest_insn)
    6979           62 :                     continue;
    6980              : 
    6981              :                   /* The earlier safety check already covered the range from
    6982              :                      FIRST_INSN up to WIDEST_INSN.  Only check the remaining
    6983              :                      suffix for duplicates that come later in the block.  This
    6984              :                      also covers call-clobbered hard registers via
    6985              :                      reg_set_between_p.  */
    6986           71 :                   bool reg_set = false;
    6987           71 :                   bool reg_set_with_widest_after = false;
    6988          735 :                   for (rtx_insn *scan = NEXT_INSN (match.widest_insn);
    6989         1470 :                        scan != NEXT_INSN (BB_END (match.bb))
    6990          735 :                        && scan != NULL_RTX;
    6991          664 :                        scan = NEXT_INSN (scan))
    6992              :                     {
    6993          666 :                       if (scan == dup_insn)
    6994              :                         {
    6995              :                           reg_set_with_widest_after = reg_set;
    6996              :                           break;
    6997              :                         }
    6998          664 :                       if (!reg_set
    6999          664 :                           && INSN_P (scan)
    7000         1328 :                           && reg_set_p (wide_reg, scan))
    7001              :                         reg_set = true;
    7002              :                     }
    7003              : 
    7004           71 :                   if (reg_set_with_widest_after)
    7005            0 :                     continue;
    7006              : 
    7007           71 :                   dup_set = single_set (dup_insn);
    7008           71 :                   if (!dup_set)
    7009              :                     {
    7010              :                       abort_changes = true;
    7011              :                       break;
    7012              :                     }
    7013              : 
    7014           71 :                   dup_mode = GET_MODE (SET_DEST (dup_set));
    7015           71 :                   if (dup_mode == GET_MODE (wide_reg))
    7016              :                     dup_src = wide_reg;
    7017              :                   else
    7018              :                     {
    7019           71 :                       dup_src = gen_lowpart (dup_mode, wide_reg);
    7020           71 :                       if (!dup_src)
    7021              :                         {
    7022              :                           abort_changes = true;
    7023              :                           break;
    7024              :                         }
    7025              :                     }
    7026              : 
    7027           71 :                   if (rtx_equal_p (dup_src, SET_DEST (dup_set)))
    7028            0 :                     continue;
    7029              : 
    7030           71 :                   if (!validate_change (dup_insn, &SET_SRC (dup_set),
    7031              :                                         dup_src, 1))
    7032              :                     {
    7033              :                       abort_changes = true;
    7034              :                       break;
    7035              :                     }
    7036              :                 }
    7037              : 
    7038           62 :               if (abort_changes)
    7039              :                 {
    7040            0 :                   cancel_changes (prev_changes);
    7041            0 :                   continue;
    7042              :                 }
    7043              : 
    7044           62 :               if (num_changes_pending () == prev_changes)
    7045            0 :                 continue;
    7046              : 
    7047           62 :               if (!apply_change_group ())
    7048            0 :                 continue;
    7049              : 
    7050           62 :               reorder_insns (match.widest_insn, match.widest_insn,
    7051              :                              insert_after);
    7052              : 
    7053           62 :               if (match.related_dups.contains (match.widest_insn))
    7054           62 :                 df_insn_rescan (match.widest_insn);
    7055              :             }
    7056              : 
    7057              :           /* Get a reasonable estimate for the maximum number of qty's
    7058              :              needed for this path.  For this, we take the number of sets
    7059              :              and multiply that by MAX_RECOG_OPERANDS.  */
    7060     21190623 :           max_qty = ebb_data.nsets * MAX_RECOG_OPERANDS;
    7061              : 
    7062              :           /* Dump the path we're about to process.  */
    7063     21190623 :           if (dump_file)
    7064          317 :             cse_dump_path (&ebb_data, ebb_data.nsets, dump_file);
    7065              : 
    7066     21190623 :           cse_extended_basic_block (&ebb_data);
    7067              :         }
    7068              :     }
    7069              : 
    7070              :   /* Clean up.  */
    7071      2345959 :   end_alias_analysis ();
    7072      2345959 :   free (reg_eqv_table);
    7073      2345959 :   free (ebb_data.path);
    7074      2345959 :   sbitmap_free (cse_visited_basic_blocks);
    7075      2345959 :   free (rc_order);
    7076      2345959 :   rtl_hooks = general_rtl_hooks;
    7077              : 
    7078      2345959 :   if (cse_jumps_altered || recorded_label_ref)
    7079              :     return 2;
    7080      2337876 :   else if (cse_cfg_altered)
    7081              :     return 1;
    7082              :   else
    7083      2328196 :     return 0;
    7084      2345959 : }
    7085              : 
    7086              : /* Count the number of times registers are used (not set) in X.
    7087              :    COUNTS is an array in which we accumulate the count, INCR is how much
    7088              :    we count each register usage.
    7089              : 
    7090              :    Don't count a usage of DEST, which is the SET_DEST of a SET which
    7091              :    contains X in its SET_SRC.  This is because such a SET does not
    7092              :    modify the liveness of DEST.
    7093              :    DEST is set to pc_rtx for a trapping insn, or for an insn with side effects.
    7094              :    We must then count uses of a SET_DEST regardless, because the insn can't be
    7095              :    deleted here.
    7096              :    Also count uses of a SET_DEST if it has been used by an earlier insn,
    7097              :    but in that case only when incrementing and not when decrementing, effectively
    7098              :    making setters of such a pseudo non-eliminable.  This is for cases like
    7099              :    (set (reg x) (expr))
    7100              :    ...
    7101              :    (set (reg y) (expr (reg (x))))
    7102              :    ...
    7103              :    (set (reg x) (expr (reg (x))))
    7104              :    where we can't eliminate the last insn because x is is still used, if y
    7105              :    is unused we can eliminate the middle insn and when considering the first insn
    7106              :    we used to eliminate it despite it being used in the last insn.  */
    7107              : 
    7108              : static void
    7109   2495610454 : count_reg_usage (rtx x, int *counts, rtx dest, int incr)
    7110              : {
    7111   2982978890 :   enum rtx_code code;
    7112   2982978890 :   rtx note;
    7113   2982978890 :   const char *fmt;
    7114   2982978890 :   int i, j;
    7115              : 
    7116   2982978890 :   if (x == 0)
    7117              :     return;
    7118              : 
    7119   2952683053 :   switch (code = GET_CODE (x))
    7120              :     {
    7121    560175572 :     case REG:
    7122    560175572 :       if (x != dest || (incr > 0 && counts[REGNO (x)]))
    7123    556296638 :         counts[REGNO (x)] += incr;
    7124              :       return;
    7125              : 
    7126              :     case PC:
    7127              :     case CONST:
    7128              :     CASE_CONST_ANY:
    7129              :     case SYMBOL_REF:
    7130              :     case LABEL_REF:
    7131              :       return;
    7132              : 
    7133    171519746 :     case CLOBBER:
    7134              :       /* If we are clobbering a MEM, mark any registers inside the address
    7135              :          as being used.  */
    7136    171519746 :       if (MEM_P (XEXP (x, 0)))
    7137       219252 :         count_reg_usage (XEXP (XEXP (x, 0), 0), counts, NULL_RTX, incr);
    7138              :       return;
    7139              : 
    7140    402952460 :     case SET:
    7141              :       /* Unless we are setting a REG, count everything in SET_DEST.  */
    7142    402952460 :       if (!REG_P (SET_DEST (x)))
    7143     98020684 :         count_reg_usage (SET_DEST (x), counts, NULL_RTX, incr);
    7144    402952460 :       count_reg_usage (SET_SRC (x), counts,
    7145              :                        dest ? dest : SET_DEST (x),
    7146              :                        incr);
    7147    402952460 :       return;
    7148              : 
    7149              :     case DEBUG_INSN:
    7150              :       return;
    7151              : 
    7152    423861921 :     case CALL_INSN:
    7153    423861921 :     case INSN:
    7154    423861921 :     case JUMP_INSN:
    7155              :       /* We expect dest to be NULL_RTX here.  If the insn may throw,
    7156              :          or if it cannot be deleted due to side-effects, mark this fact
    7157              :          by setting DEST to pc_rtx.  */
    7158    423861921 :       if ((!cfun->can_delete_dead_exceptions
    7159    164185118 :            && !insn_nothrow_p (as_a <rtx_insn *> (x)))
    7160    569896297 :           || side_effects_p (PATTERN (x)))
    7161     55661433 :         dest = pc_rtx;
    7162    423861921 :       if (auto call = dyn_cast<rtx_call_insn *> (x))
    7163     30295837 :         count_reg_usage (CALL_INSN_FUNCTION_USAGE (call), counts, dest, incr);
    7164    423861921 :       count_reg_usage (PATTERN (x), counts, dest, incr);
    7165              : 
    7166              :       /* Things used in a REG_EQUAL note aren't dead since loop may try to
    7167              :          use them.  */
    7168              : 
    7169    423861921 :       note = find_reg_equal_equiv_note (as_a <rtx_insn *> (x));
    7170    423861921 :       if (note)
    7171              :         {
    7172     23054495 :           rtx eqv = XEXP (note, 0);
    7173              : 
    7174     23054495 :           if (GET_CODE (eqv) == EXPR_LIST)
    7175              :           /* This REG_EQUAL note describes the result of a function call.
    7176              :              Process all the arguments.  */
    7177            0 :             do
    7178              :               {
    7179            0 :                 count_reg_usage (XEXP (eqv, 0), counts, dest, incr);
    7180            0 :                 eqv = XEXP (eqv, 1);
    7181              :               }
    7182            0 :             while (eqv && GET_CODE (eqv) == EXPR_LIST);
    7183              :           else
    7184              :             count_reg_usage (eqv, counts, dest, incr);
    7185              :         }
    7186              :       return;
    7187              : 
    7188     61142229 :     case EXPR_LIST:
    7189     61142229 :       if (REG_NOTE_KIND (x) == REG_EQUAL
    7190     61142229 :           || (REG_NOTE_KIND (x) != REG_NONNEG && GET_CODE (XEXP (x,0)) == USE)
    7191              :           /* FUNCTION_USAGE expression lists may include (CLOBBER (mem /u)),
    7192              :              involving registers in the address.  */
    7193       678584 :           || GET_CODE (XEXP (x, 0)) == CLOBBER)
    7194     60463645 :         count_reg_usage (XEXP (x, 0), counts, NULL_RTX, incr);
    7195              : 
    7196     61142229 :       count_reg_usage (XEXP (x, 1), counts, NULL_RTX, incr);
    7197     61142229 :       return;
    7198              : 
    7199       675372 :     case ASM_OPERANDS:
    7200              :       /* Iterate over just the inputs, not the constraints as well.  */
    7201      1361584 :       for (i = ASM_OPERANDS_INPUT_LENGTH (x) - 1; i >= 0; i--)
    7202       686212 :         count_reg_usage (ASM_OPERANDS_INPUT (x, i), counts, dest, incr);
    7203              :       return;
    7204              : 
    7205            0 :     case INSN_LIST:
    7206            0 :     case INT_LIST:
    7207            0 :       gcc_unreachable ();
    7208              : 
    7209    742730946 :     default:
    7210    742730946 :       break;
    7211              :     }
    7212              : 
    7213    742730946 :   fmt = GET_RTX_FORMAT (code);
    7214   2057157814 :   for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
    7215              :     {
    7216   1314426868 :       if (fmt[i] == 'e')
    7217   1000122191 :         count_reg_usage (XEXP (x, i), counts, dest, incr);
    7218    314304677 :       else if (fmt[i] == 'E')
    7219    209885678 :         for (j = XVECLEN (x, i) - 1; j >= 0; j--)
    7220    140620382 :           count_reg_usage (XVECEXP (x, i, j), counts, dest, incr);
    7221              :     }
    7222              : }
    7223              : 
    7224              : /* Return true if X is a dead register.  */
    7225              : 
    7226              : static inline bool
    7227    818723857 : is_dead_reg (const_rtx x, int *counts)
    7228              : {
    7229    818723857 :   return (REG_P (x)
    7230    355776724 :           && REGNO (x) >= FIRST_PSEUDO_REGISTER
    7231    221781653 :           && counts[REGNO (x)] == 0);
    7232              : }
    7233              : 
    7234              : /* Return true if set is live.  */
    7235              : static bool
    7236    380492622 : set_live_p (rtx set, int *counts)
    7237              : {
    7238    380492622 :   if (set_noop_p (set))
    7239              :     return false;
    7240              : 
    7241    380470399 :   if (!is_dead_reg (SET_DEST (set), counts)
    7242      8984898 :       || side_effects_p (SET_SRC (set)))
    7243    371551291 :     return true;
    7244              : 
    7245              :   return false;
    7246              : }
    7247              : 
    7248              : /* Return true if insn is live.  */
    7249              : 
    7250              : static bool
    7251    732434048 : insn_live_p (rtx_insn *insn, int *counts)
    7252              : {
    7253    732434048 :   int i;
    7254    732434048 :   if (!cfun->can_delete_dead_exceptions && !insn_nothrow_p (insn))
    7255              :     return true;
    7256    713967455 :   else if (GET_CODE (PATTERN (insn)) == SET)
    7257    320651241 :     return set_live_p (PATTERN (insn), counts);
    7258    393316214 :   else if (GET_CODE (PATTERN (insn)) == PARALLEL)
    7259              :     {
    7260    121066255 :       for (i = XVECLEN (PATTERN (insn), 0) - 1; i >= 0; i--)
    7261              :         {
    7262    120561263 :           rtx elt = XVECEXP (PATTERN (insn), 0, i);
    7263              : 
    7264    120561263 :           if (GET_CODE (elt) == SET)
    7265              :             {
    7266     59841381 :               if (set_live_p (elt, counts))
    7267              :                 return true;
    7268              :             }
    7269     60719882 :           else if (GET_CODE (elt) != CLOBBER && GET_CODE (elt) != USE)
    7270              :             return true;
    7271              :         }
    7272              :       return false;
    7273              :     }
    7274    333440619 :   else if (DEBUG_INSN_P (insn))
    7275              :     {
    7276    316895392 :       if (DEBUG_MARKER_INSN_P (insn))
    7277              :         return true;
    7278              : 
    7279    245349375 :       if (DEBUG_BIND_INSN_P (insn)
    7280    245349375 :           && TREE_VISITED (INSN_VAR_LOCATION_DECL (insn)))
    7281       449177 :         return false;
    7282              : 
    7283              :       return true;
    7284              :     }
    7285              :   else
    7286              :     return true;
    7287              : }
    7288              : 
    7289              : /* Count the number of stores into pseudo.  Callback for note_stores.  */
    7290              : 
    7291              : static void
    7292    260855720 : count_stores (rtx x, const_rtx set ATTRIBUTE_UNUSED, void *data)
    7293              : {
    7294    260855720 :   int *counts = (int *) data;
    7295    260855720 :   if (REG_P (x) && REGNO (x) >= FIRST_PSEUDO_REGISTER)
    7296     97623356 :     counts[REGNO (x)]++;
    7297    260855720 : }
    7298              : 
    7299              : /* Return if DEBUG_INSN pattern PAT needs to be reset because some dead
    7300              :    pseudo doesn't have a replacement.  COUNTS[X] is zero if register X
    7301              :    is dead and REPLACEMENTS[X] is null if it has no replacement.
    7302              :    Set *SEEN_REPL to true if we see a dead register that does have
    7303              :    a replacement.  */
    7304              : 
    7305              : static bool
    7306    245233290 : is_dead_debug_insn (const_rtx pat, int *counts, rtx *replacements,
    7307              :                     bool *seen_repl)
    7308              : {
    7309    245233290 :   subrtx_iterator::array_type array;
    7310    679250711 :   FOR_EACH_SUBRTX (iter, array, pat, NONCONST)
    7311              :     {
    7312    434040958 :       const_rtx x = *iter;
    7313    500764888 :       if (is_dead_reg (x, counts))
    7314              :         {
    7315        48942 :           if (replacements && replacements[REGNO (x)] != NULL_RTX)
    7316        25405 :             *seen_repl = true;
    7317              :           else
    7318        23537 :             return true;
    7319              :         }
    7320              :     }
    7321    245209753 :   return false;
    7322    245233290 : }
    7323              : 
    7324              : /* Replace a dead pseudo in a DEBUG_INSN with replacement DEBUG_EXPR.
    7325              :    Callback for simplify_replace_fn_rtx.  */
    7326              : 
    7327              : static rtx
    7328        40584 : replace_dead_reg (rtx x, const_rtx old_rtx ATTRIBUTE_UNUSED, void *data)
    7329              : {
    7330        40584 :   rtx *replacements = (rtx *) data;
    7331              : 
    7332        40584 :   if (REG_P (x)
    7333        26459 :       && REGNO (x) >= FIRST_PSEUDO_REGISTER
    7334        67004 :       && replacements[REGNO (x)] != NULL_RTX)
    7335              :     {
    7336        25405 :       if (GET_MODE (x) == GET_MODE (replacements[REGNO (x)]))
    7337              :         return replacements[REGNO (x)];
    7338            0 :       return lowpart_subreg (GET_MODE (x), replacements[REGNO (x)],
    7339            0 :                              GET_MODE (replacements[REGNO (x)]));
    7340              :     }
    7341              :   return NULL_RTX;
    7342              : }
    7343              : 
    7344              : /* Scan all the insns and delete any that are dead; i.e., they store a register
    7345              :    that is never used or they copy a register to itself.
    7346              : 
    7347              :    This is used to remove insns made obviously dead by cse, loop or other
    7348              :    optimizations.  It improves the heuristics in loop since it won't try to
    7349              :    move dead invariants out of loops or make givs for dead quantities.  The
    7350              :    remaining passes of the compilation are also sped up.  */
    7351              : 
    7352              : int
    7353      6492513 : delete_trivially_dead_insns (rtx_insn *insns, int nreg)
    7354              : {
    7355      6492513 :   int *counts;
    7356      6492513 :   rtx_insn *insn, *prev;
    7357      6492513 :   rtx *replacements = NULL;
    7358      6492513 :   int ndead = 0;
    7359              : 
    7360      6492513 :   timevar_push (TV_DELETE_TRIVIALLY_DEAD);
    7361              :   /* First count the number of times each register is used.  */
    7362      6492513 :   if (MAY_HAVE_DEBUG_BIND_INSNS)
    7363              :     {
    7364      2690380 :       counts = XCNEWVEC (int, nreg * 3);
    7365    634106366 :       for (insn = insns; insn; insn = NEXT_INSN (insn))
    7366    628725606 :         if (DEBUG_BIND_INSN_P (insn))
    7367              :           {
    7368    245665226 :             count_reg_usage (INSN_VAR_LOCATION_LOC (insn), counts + nreg,
    7369              :                              NULL_RTX, 1);
    7370    245665226 :             TREE_VISITED (INSN_VAR_LOCATION_DECL (insn)) = 0;
    7371              :           }
    7372    383060380 :         else if (INSN_P (insn))
    7373              :           {
    7374    308945922 :             count_reg_usage (insn, counts, NULL_RTX, 1);
    7375    308945922 :             note_stores (insn, count_stores, counts + nreg * 2);
    7376              :           }
    7377              :       /* If there can be debug insns, COUNTS are 3 consecutive arrays.
    7378              :          First one counts how many times each pseudo is used outside
    7379              :          of debug insns, second counts how many times each pseudo is
    7380              :          used in debug insns and third counts how many times a pseudo
    7381              :          is stored.  */
    7382              :     }
    7383              :   else
    7384              :     {
    7385      3802133 :       counts = XCNEWVEC (int, nreg);
    7386    233671781 :       for (insn = insns; insn; insn = NEXT_INSN (insn))
    7387    226067515 :         if (INSN_P (insn))
    7388    177822900 :           count_reg_usage (insn, counts, NULL_RTX, 1);
    7389              :       /* If no debug insns can be present, COUNTS is just an array
    7390              :          which counts how many times each pseudo is used.  */
    7391              :     }
    7392              :   /* Pseudo PIC register should be considered as used due to possible
    7393              :      new usages generated.  */
    7394      6492513 :   if (!reload_completed
    7395      6492513 :       && pic_offset_table_rtx
    7396      6711592 :       && REGNO (pic_offset_table_rtx) >= FIRST_PSEUDO_REGISTER)
    7397       219079 :     counts[REGNO (pic_offset_table_rtx)]++;
    7398              :   /* Go from the last insn to the first and delete insns that only set unused
    7399              :      registers or copy a register to itself.  As we delete an insn, remove
    7400              :      usage counts for registers it uses.
    7401              : 
    7402              :      The first jump optimization pass may leave a real insn as the last
    7403              :      insn in the function.   We must not skip that insn or we may end
    7404              :      up deleting code that is not really dead.
    7405              : 
    7406              :      If some otherwise unused register is only used in DEBUG_INSNs,
    7407              :      try to create a DEBUG_EXPR temporary and emit a DEBUG_INSN before
    7408              :      the setter.  Then go through DEBUG_INSNs and if a DEBUG_EXPR
    7409              :      has been created for the unused register, replace it with
    7410              :      the DEBUG_EXPR, otherwise reset the DEBUG_INSN.  */
    7411      6492513 :   auto_vec<tree, 32> later_debug_set_vars;
    7412    861285634 :   for (insn = get_last_insn (); insn; insn = prev)
    7413              :     {
    7414    854793121 :       int live_insn = 0;
    7415              : 
    7416    854793121 :       prev = PREV_INSN (insn);
    7417    854793121 :       if (!INSN_P (insn))
    7418    122359073 :         continue;
    7419              : 
    7420    732434048 :       live_insn = insn_live_p (insn, counts);
    7421              : 
    7422              :       /* If this is a dead insn, delete it and show registers in it aren't
    7423              :          being used.  */
    7424              : 
    7425    732434048 :       if (! live_insn && dbg_cnt (delete_trivial_dead))
    7426              :         {
    7427      9088293 :           if (DEBUG_INSN_P (insn))
    7428              :             {
    7429       449177 :               if (DEBUG_BIND_INSN_P (insn))
    7430       449177 :                 count_reg_usage (INSN_VAR_LOCATION_LOC (insn), counts + nreg,
    7431              :                                  NULL_RTX, -1);
    7432              :             }
    7433              :           else
    7434              :             {
    7435      8639116 :               rtx set;
    7436      8639116 :               if (MAY_HAVE_DEBUG_BIND_INSNS
    7437      4212500 :                   && (set = single_set (insn)) != NULL_RTX
    7438      4212500 :                   && is_dead_reg (SET_DEST (set), counts)
    7439              :                   /* Used at least once in some DEBUG_INSN.  */
    7440      4206563 :                   && counts[REGNO (SET_DEST (set)) + nreg] > 0
    7441              :                   /* And set exactly once.  */
    7442        18053 :                   && counts[REGNO (SET_DEST (set)) + nreg * 2] == 1
    7443        17242 :                   && !side_effects_p (SET_SRC (set))
    7444      8656358 :                   && asm_noperands (PATTERN (insn)) < 0)
    7445              :                 {
    7446        17241 :                   rtx dval, bind_var_loc;
    7447        17241 :                   rtx_insn *bind;
    7448              : 
    7449              :                   /* Create DEBUG_EXPR (and DEBUG_EXPR_DECL).  */
    7450        17241 :                   dval = make_debug_expr_from_rtl (SET_DEST (set));
    7451              : 
    7452              :                   /* Emit a debug bind insn before the insn in which
    7453              :                      reg dies.  */
    7454        17241 :                   bind_var_loc =
    7455        17241 :                     gen_rtx_VAR_LOCATION (GET_MODE (SET_DEST (set)),
    7456              :                                           DEBUG_EXPR_TREE_DECL (dval),
    7457              :                                           SET_SRC (set),
    7458              :                                           VAR_INIT_STATUS_INITIALIZED);
    7459        17241 :                   count_reg_usage (bind_var_loc, counts + nreg, NULL_RTX, 1);
    7460              : 
    7461        17241 :                   bind = emit_debug_insn_before (bind_var_loc, insn);
    7462        17241 :                   df_insn_rescan (bind);
    7463              : 
    7464        17241 :                   if (replacements == NULL)
    7465         9235 :                     replacements = XCNEWVEC (rtx, nreg);
    7466        17241 :                   replacements[REGNO (SET_DEST (set))] = dval;
    7467              :                 }
    7468              : 
    7469      8639116 :               count_reg_usage (insn, counts, NULL_RTX, -1);
    7470      8639116 :               ndead++;
    7471              :             }
    7472      9088293 :           cse_cfg_altered |= delete_insn_and_edges (insn);
    7473              :         }
    7474              :       else
    7475              :         {
    7476    723345755 :           if (!DEBUG_INSN_P (insn) || DEBUG_MARKER_INSN_P (insn))
    7477              :             {
    7478   1679303358 :               for (tree var : later_debug_set_vars)
    7479    244914240 :                 TREE_VISITED (var) = 0;
    7480    478129706 :               later_debug_set_vars.truncate (0);
    7481              :             }
    7482   1100009170 :           else if (DEBUG_BIND_INSN_P (insn)
    7483    245216049 :                    && !TREE_VISITED (INSN_VAR_LOCATION_DECL (insn)))
    7484              :             {
    7485    245210033 :               later_debug_set_vars.safe_push (INSN_VAR_LOCATION_DECL (insn));
    7486    245210033 :               TREE_VISITED (INSN_VAR_LOCATION_DECL (insn)) = 1;
    7487              :             }
    7488              :         }
    7489              :     }
    7490              : 
    7491      6492513 :   if (MAY_HAVE_DEBUG_BIND_INSNS)
    7492              :     {
    7493    626771550 :       for (insn = get_last_insn (); insn; insn = PREV_INSN (insn))
    7494    940859787 :         if (DEBUG_BIND_INSN_P (insn))
    7495              :           {
    7496              :             /* If this debug insn references a dead register that wasn't replaced
    7497              :                with an DEBUG_EXPR, reset the DEBUG_INSN.  */
    7498    245233290 :             bool seen_repl = false;
    7499    245233290 :             if (is_dead_debug_insn (INSN_VAR_LOCATION_LOC (insn),
    7500              :                                     counts, replacements, &seen_repl))
    7501              :               {
    7502        23537 :                 INSN_VAR_LOCATION_LOC (insn) = gen_rtx_UNKNOWN_VAR_LOC ();
    7503        23537 :                 df_insn_rescan (insn);
    7504              :               }
    7505    245209753 :             else if (seen_repl)
    7506              :               {
    7507        25386 :                 INSN_VAR_LOCATION_LOC (insn)
    7508        25386 :                   = simplify_replace_fn_rtx (INSN_VAR_LOCATION_LOC (insn),
    7509              :                                              NULL_RTX, replace_dead_reg,
    7510              :                                              replacements);
    7511        25386 :                 df_insn_rescan (insn);
    7512              :               }
    7513              :           }
    7514      2690380 :       free (replacements);
    7515              :     }
    7516              : 
    7517      6492513 :   if (dump_file && ndead)
    7518           26 :     fprintf (dump_file, "Deleted %i trivially dead insns\n",
    7519              :              ndead);
    7520              :   /* Clean up.  */
    7521      6492513 :   free (counts);
    7522      6492513 :   timevar_pop (TV_DELETE_TRIVIALLY_DEAD);
    7523      6492513 :   return ndead;
    7524      6492513 : }
    7525              : 
    7526              : /* If LOC contains references to NEWREG in a different mode, change them
    7527              :    to use NEWREG instead.  */
    7528              : 
    7529              : static void
    7530        58948 : cse_change_cc_mode (subrtx_ptr_iterator::array_type &array,
    7531              :                     rtx *loc, rtx_insn *insn, rtx newreg)
    7532              : {
    7533       364633 :   FOR_EACH_SUBRTX_PTR (iter, array, loc, NONCONST)
    7534              :     {
    7535       305685 :       rtx *loc = *iter;
    7536       305685 :       rtx x = *loc;
    7537       305685 :       if (x
    7538       276211 :           && REG_P (x)
    7539        68401 :           && REGNO (x) == REGNO (newreg)
    7540       355952 :           && GET_MODE (x) != GET_MODE (newreg))
    7541              :         {
    7542        50267 :           validate_change (insn, loc, newreg, 1);
    7543        50267 :           iter.skip_subrtxes ();
    7544              :         }
    7545              :     }
    7546        58948 : }
    7547              : 
    7548              : /* Change the mode of any reference to the register REGNO (NEWREG) to
    7549              :    GET_MODE (NEWREG) in INSN.  */
    7550              : 
    7551              : static void
    7552        29474 : cse_change_cc_mode_insn (rtx_insn *insn, rtx newreg)
    7553              : {
    7554        29474 :   int success;
    7555              : 
    7556        29474 :   if (!INSN_P (insn))
    7557            0 :     return;
    7558              : 
    7559        29474 :   subrtx_ptr_iterator::array_type array;
    7560        29474 :   cse_change_cc_mode (array, &PATTERN (insn), insn, newreg);
    7561        29474 :   cse_change_cc_mode (array, &REG_NOTES (insn), insn, newreg);
    7562              : 
    7563              :   /* If the following assertion was triggered, there is most probably
    7564              :      something wrong with the cc_modes_compatible back end function.
    7565              :      CC modes only can be considered compatible if the insn - with the mode
    7566              :      replaced by any of the compatible modes - can still be recognized.  */
    7567        29474 :   success = apply_change_group ();
    7568        29474 :   gcc_assert (success);
    7569        29474 : }
    7570              : 
    7571              : /* Change the mode of any reference to the register REGNO (NEWREG) to
    7572              :    GET_MODE (NEWREG), starting at START.  Stop before END.  Stop at
    7573              :    any instruction which modifies NEWREG.  */
    7574              : 
    7575              : static void
    7576        19911 : cse_change_cc_mode_insns (rtx_insn *start, rtx_insn *end, rtx newreg)
    7577              : {
    7578        19911 :   rtx_insn *insn;
    7579              : 
    7580        40199 :   for (insn = start; insn != end; insn = NEXT_INSN (insn))
    7581              :     {
    7582        21792 :       if (! INSN_P (insn))
    7583            0 :         continue;
    7584              : 
    7585        21792 :       if (reg_set_p (newreg, insn))
    7586              :         return;
    7587              : 
    7588        20288 :       cse_change_cc_mode_insn (insn, newreg);
    7589              :     }
    7590              : }
    7591              : 
    7592              : /* BB is a basic block which finishes with CC_REG as a condition code
    7593              :    register which is set to CC_SRC.  Look through the successors of BB
    7594              :    to find blocks which have a single predecessor (i.e., this one),
    7595              :    and look through those blocks for an assignment to CC_REG which is
    7596              :    equivalent to CC_SRC.  CAN_CHANGE_MODE indicates whether we are
    7597              :    permitted to change the mode of CC_SRC to a compatible mode.  This
    7598              :    returns VOIDmode if no equivalent assignments were found.
    7599              :    Otherwise it returns the mode which CC_SRC should wind up with.
    7600              :    ORIG_BB should be the same as BB in the outermost cse_cc_succs call,
    7601              :    but is passed unmodified down to recursive calls in order to prevent
    7602              :    endless recursion.
    7603              : 
    7604              :    The main complexity in this function is handling the mode issues.
    7605              :    We may have more than one duplicate which we can eliminate, and we
    7606              :    try to find a mode which will work for multiple duplicates.  */
    7607              : 
    7608              : static machine_mode
    7609      5653224 : cse_cc_succs (basic_block bb, basic_block orig_bb, rtx cc_reg, rtx cc_src,
    7610              :               bool can_change_mode)
    7611              : {
    7612      5653224 :   bool found_equiv;
    7613      5653224 :   machine_mode mode;
    7614      5653224 :   unsigned int insn_count;
    7615      5653224 :   edge e;
    7616      5653224 :   rtx_insn *insns[2];
    7617      5653224 :   machine_mode modes[2];
    7618      5653224 :   rtx_insn *last_insns[2];
    7619      5653224 :   unsigned int i;
    7620      5653224 :   rtx newreg;
    7621      5653224 :   edge_iterator ei;
    7622              : 
    7623              :   /* We expect to have two successors.  Look at both before picking
    7624              :      the final mode for the comparison.  If we have more successors
    7625              :      (i.e., some sort of table jump, although that seems unlikely),
    7626              :      then we require all beyond the first two to use the same
    7627              :      mode.  */
    7628              : 
    7629      5653224 :   found_equiv = false;
    7630      5653224 :   mode = GET_MODE (cc_src);
    7631      5653224 :   insn_count = 0;
    7632     16011397 :   FOR_EACH_EDGE (e, ei, bb->succs)
    7633              :     {
    7634     10358173 :       rtx_insn *insn;
    7635     10358173 :       rtx_insn *end;
    7636              : 
    7637     10358173 :       if (e->flags & EDGE_COMPLEX)
    7638        32683 :         continue;
    7639              : 
    7640     10325490 :       if (EDGE_COUNT (e->dest->preds) != 1
    7641      5769225 :           || e->dest == EXIT_BLOCK_PTR_FOR_FN (cfun)
    7642              :           /* Avoid endless recursion on unreachable blocks.  */
    7643     15995850 :           || e->dest == orig_bb)
    7644      4655130 :         continue;
    7645              : 
    7646      5670360 :       end = NEXT_INSN (BB_END (e->dest));
    7647     36362337 :       for (insn = BB_HEAD (e->dest); insn != end; insn = NEXT_INSN (insn))
    7648              :         {
    7649     35206966 :           rtx set;
    7650              : 
    7651     35206966 :           if (! INSN_P (insn))
    7652      7691780 :             continue;
    7653              : 
    7654              :           /* If CC_SRC is modified, we have to stop looking for
    7655              :              something which uses it.  */
    7656     27515186 :           if (modified_in_p (cc_src, insn))
    7657              :             break;
    7658              : 
    7659              :           /* Check whether INSN sets CC_REG to CC_SRC.  */
    7660     27018879 :           set = single_set (insn);
    7661     27018879 :           if (set
    7662     11571568 :               && REG_P (SET_DEST (set))
    7663     37097789 :               && REGNO (SET_DEST (set)) == REGNO (cc_reg))
    7664              :             {
    7665      1408806 :               bool found;
    7666      1408806 :               machine_mode set_mode;
    7667      1408806 :               machine_mode comp_mode;
    7668              : 
    7669      1408806 :               found = false;
    7670      1408806 :               set_mode = GET_MODE (SET_SRC (set));
    7671      1408806 :               comp_mode = set_mode;
    7672      1408806 :               if (rtx_equal_p (cc_src, SET_SRC (set)))
    7673              :                 found = true;
    7674      1402714 :               else if (GET_CODE (cc_src) == COMPARE
    7675      1333804 :                        && GET_CODE (SET_SRC (set)) == COMPARE
    7676      1283126 :                        && mode != set_mode
    7677       353435 :                        && rtx_equal_p (XEXP (cc_src, 0),
    7678       353435 :                                        XEXP (SET_SRC (set), 0))
    7679      1464326 :                        && rtx_equal_p (XEXP (cc_src, 1),
    7680        61612 :                                        XEXP (SET_SRC (set), 1)))
    7681              : 
    7682              :                 {
    7683        19915 :                   comp_mode = targetm.cc_modes_compatible (mode, set_mode);
    7684        19915 :                   if (comp_mode != VOIDmode
    7685        19915 :                       && (can_change_mode || comp_mode == mode))
    7686              :                     found = true;
    7687              :                 }
    7688              : 
    7689        26003 :               if (found)
    7690              :                 {
    7691        26003 :                   found_equiv = true;
    7692        26003 :                   if (insn_count < ARRAY_SIZE (insns))
    7693              :                     {
    7694        26003 :                       insns[insn_count] = insn;
    7695        26003 :                       modes[insn_count] = set_mode;
    7696        26003 :                       last_insns[insn_count] = end;
    7697        26003 :                       ++insn_count;
    7698              : 
    7699        26003 :                       if (mode != comp_mode)
    7700              :                         {
    7701         9186 :                           gcc_assert (can_change_mode);
    7702         9186 :                           mode = comp_mode;
    7703              : 
    7704              :                           /* The modified insn will be re-recognized later.  */
    7705         9186 :                           PUT_MODE (cc_src, mode);
    7706              :                         }
    7707              :                     }
    7708              :                   else
    7709              :                     {
    7710            0 :                       if (set_mode != mode)
    7711              :                         {
    7712              :                           /* We found a matching expression in the
    7713              :                              wrong mode, but we don't have room to
    7714              :                              store it in the array.  Punt.  This case
    7715              :                              should be rare.  */
    7716              :                           break;
    7717              :                         }
    7718              :                       /* INSN sets CC_REG to a value equal to CC_SRC
    7719              :                          with the right mode.  We can simply delete
    7720              :                          it.  */
    7721            0 :                       delete_insn (insn);
    7722              :                     }
    7723              : 
    7724              :                   /* We found an instruction to delete.  Keep looking,
    7725              :                      in the hopes of finding a three-way jump.  */
    7726        26003 :                   continue;
    7727              :                 }
    7728              : 
    7729              :               /* We found an instruction which sets the condition
    7730              :                  code, so don't look any farther.  */
    7731              :               break;
    7732              :             }
    7733              : 
    7734              :           /* If INSN sets CC_REG in some other way, don't look any
    7735              :              farther.  */
    7736     25610073 :           if (reg_set_p (cc_reg, insn))
    7737              :             break;
    7738              :         }
    7739              : 
    7740              :       /* If we fell off the bottom of the block, we can keep looking
    7741              :          through successors.  We pass CAN_CHANGE_MODE as false because
    7742              :          we aren't prepared to handle compatibility between the
    7743              :          further blocks and this block.  */
    7744      5670360 :       if (insn == end)
    7745              :         {
    7746      1155371 :           machine_mode submode;
    7747              : 
    7748      1155371 :           submode = cse_cc_succs (e->dest, orig_bb, cc_reg, cc_src, false);
    7749      1155371 :           if (submode != VOIDmode)
    7750              :             {
    7751           52 :               gcc_assert (submode == mode);
    7752              :               found_equiv = true;
    7753              :               can_change_mode = false;
    7754              :             }
    7755              :         }
    7756              :     }
    7757              : 
    7758      5653224 :   if (! found_equiv)
    7759              :     return VOIDmode;
    7760              : 
    7761              :   /* Now INSN_COUNT is the number of instructions we found which set
    7762              :      CC_REG to a value equivalent to CC_SRC.  The instructions are in
    7763              :      INSNS.  The modes used by those instructions are in MODES.  */
    7764              : 
    7765              :   newreg = NULL_RTX;
    7766        52051 :   for (i = 0; i < insn_count; ++i)
    7767              :     {
    7768        26003 :       if (modes[i] != mode)
    7769              :         {
    7770              :           /* We need to change the mode of CC_REG in INSNS[i] and
    7771              :              subsequent instructions.  */
    7772        10725 :           if (! newreg)
    7773              :             {
    7774        10725 :               if (GET_MODE (cc_reg) == mode)
    7775              :                 newreg = cc_reg;
    7776              :               else
    7777         7822 :                 newreg = gen_rtx_REG (mode, REGNO (cc_reg));
    7778              :             }
    7779        10725 :           cse_change_cc_mode_insns (NEXT_INSN (insns[i]), last_insns[i],
    7780              :                                     newreg);
    7781              :         }
    7782              : 
    7783        26003 :       cse_cfg_altered |= delete_insn_and_edges (insns[i]);
    7784              :     }
    7785              : 
    7786              :   return mode;
    7787              : }
    7788              : 
    7789              : /* If we have a fixed condition code register (or two), walk through
    7790              :    the instructions and try to eliminate duplicate assignments.  */
    7791              : 
    7792              : static void
    7793       982740 : cse_condition_code_reg (void)
    7794              : {
    7795       982740 :   unsigned int cc_regno_1;
    7796       982740 :   unsigned int cc_regno_2;
    7797       982740 :   rtx cc_reg_1;
    7798       982740 :   rtx cc_reg_2;
    7799       982740 :   basic_block bb;
    7800              : 
    7801       982740 :   if (! targetm.fixed_condition_code_regs (&cc_regno_1, &cc_regno_2))
    7802            0 :     return;
    7803              : 
    7804       982740 :   cc_reg_1 = gen_rtx_REG (CCmode, cc_regno_1);
    7805       982740 :   if (cc_regno_2 != INVALID_REGNUM)
    7806            0 :     cc_reg_2 = gen_rtx_REG (CCmode, cc_regno_2);
    7807              :   else
    7808              :     cc_reg_2 = NULL_RTX;
    7809              : 
    7810     11025329 :   FOR_EACH_BB_FN (bb, cfun)
    7811              :     {
    7812     10042589 :       rtx_insn *last_insn;
    7813     10042589 :       rtx cc_reg;
    7814     10042589 :       rtx_insn *insn;
    7815     10042589 :       rtx_insn *cc_src_insn;
    7816     10042589 :       rtx cc_src;
    7817     10042589 :       machine_mode mode;
    7818     10042589 :       machine_mode orig_mode;
    7819              : 
    7820              :       /* Look for blocks which end with a conditional jump based on a
    7821              :          condition code register.  Then look for the instruction which
    7822              :          sets the condition code register.  Then look through the
    7823              :          successor blocks for instructions which set the condition
    7824              :          code register to the same value.  There are other possible
    7825              :          uses of the condition code register, but these are by far the
    7826              :          most common and the ones which we are most likely to be able
    7827              :          to optimize.  */
    7828              : 
    7829     10042589 :       last_insn = BB_END (bb);
    7830     10042589 :       if (!JUMP_P (last_insn))
    7831      5393299 :         continue;
    7832              : 
    7833      4649290 :       if (reg_referenced_p (cc_reg_1, PATTERN (last_insn)))
    7834              :         cc_reg = cc_reg_1;
    7835        14832 :       else if (cc_reg_2 && reg_referenced_p (cc_reg_2, PATTERN (last_insn)))
    7836              :         cc_reg = cc_reg_2;
    7837              :       else
    7838        14832 :         continue;
    7839              : 
    7840      4634458 :       cc_src_insn = NULL;
    7841      4634458 :       cc_src = NULL_RTX;
    7842      4808872 :       for (insn = PREV_INSN (last_insn);
    7843      4808872 :            insn && insn != PREV_INSN (BB_HEAD (bb));
    7844       174414 :            insn = PREV_INSN (insn))
    7845              :         {
    7846      4689349 :           rtx set;
    7847              : 
    7848      4689349 :           if (! INSN_P (insn))
    7849       128181 :             continue;
    7850      4561168 :           set = single_set (insn);
    7851      4561168 :           if (set
    7852      4505579 :               && REG_P (SET_DEST (set))
    7853      9065833 :               && REGNO (SET_DEST (set)) == REGNO (cc_reg))
    7854              :             {
    7855      4497867 :               cc_src_insn = insn;
    7856      4497867 :               cc_src = SET_SRC (set);
    7857      4497867 :               break;
    7858              :             }
    7859        63301 :           else if (reg_set_p (cc_reg, insn))
    7860              :             break;
    7861              :         }
    7862              : 
    7863      4634458 :       if (! cc_src_insn)
    7864       136591 :         continue;
    7865              : 
    7866      4497867 :       if (modified_between_p (cc_src, cc_src_insn, NEXT_INSN (last_insn)))
    7867           14 :         continue;
    7868              : 
    7869              :       /* Now CC_REG is a condition code register used for a
    7870              :          conditional jump at the end of the block, and CC_SRC, in
    7871              :          CC_SRC_INSN, is the value to which that condition code
    7872              :          register is set, and CC_SRC is still meaningful at the end of
    7873              :          the basic block.  */
    7874              : 
    7875      4497853 :       orig_mode = GET_MODE (cc_src);
    7876      4497853 :       mode = cse_cc_succs (bb, bb, cc_reg, cc_src, true);
    7877      4497853 :       if (mode != VOIDmode)
    7878              :         {
    7879        25996 :           gcc_assert (mode == GET_MODE (cc_src));
    7880        25996 :           if (mode != orig_mode)
    7881              :             {
    7882         9186 :               rtx newreg = gen_rtx_REG (mode, REGNO (cc_reg));
    7883              : 
    7884         9186 :               cse_change_cc_mode_insn (cc_src_insn, newreg);
    7885              : 
    7886              :               /* Do the same in the following insns that use the
    7887              :                  current value of CC_REG within BB.  */
    7888         9186 :               cse_change_cc_mode_insns (NEXT_INSN (cc_src_insn),
    7889              :                                         NEXT_INSN (last_insn),
    7890              :                                         newreg);
    7891              :             }
    7892              :         }
    7893              :     }
    7894              : }
    7895              : 
    7896              : 
    7897              : /* Perform common subexpression elimination.  Nonzero value from
    7898              :    `cse_main' means that jumps were simplified and some code may now
    7899              :    be unreachable, so do jump optimization again.  */
    7900              : static unsigned int
    7901      1064962 : rest_of_handle_cse (void)
    7902              : {
    7903      1064962 :   int tem;
    7904              : 
    7905      1064962 :   if (dump_file)
    7906           32 :     dump_flow_info (dump_file, dump_flags);
    7907              : 
    7908      1064962 :   tem = cse_main (get_insns (), max_reg_num ());
    7909              : 
    7910              :   /* If we are not running more CSE passes, then we are no longer
    7911              :      expecting CSE to be run.  But always rerun it in a cheap mode.  */
    7912      1064962 :   cse_not_expected = !flag_rerun_cse_after_loop && !flag_gcse;
    7913              : 
    7914      1064962 :   if (tem == 2)
    7915              :     {
    7916         5615 :       timevar_push (TV_JUMP);
    7917         5615 :       rebuild_jump_labels (get_insns ());
    7918         5615 :       cse_cfg_altered |= cleanup_cfg (CLEANUP_CFG_CHANGED);
    7919         5615 :       timevar_pop (TV_JUMP);
    7920              :     }
    7921      1059347 :   else if (tem == 1 || optimize > 1)
    7922       977474 :     cse_cfg_altered |= cleanup_cfg (0);
    7923              : 
    7924      1064962 :   return 0;
    7925              : }
    7926              : 
    7927              : namespace {
    7928              : 
    7929              : const pass_data pass_data_cse =
    7930              : {
    7931              :   RTL_PASS, /* type */
    7932              :   "cse1", /* name */
    7933              :   OPTGROUP_NONE, /* optinfo_flags */
    7934              :   TV_CSE, /* tv_id */
    7935              :   0, /* properties_required */
    7936              :   0, /* properties_provided */
    7937              :   0, /* properties_destroyed */
    7938              :   0, /* todo_flags_start */
    7939              :   TODO_df_finish, /* todo_flags_finish */
    7940              : };
    7941              : 
    7942              : class pass_cse : public rtl_opt_pass
    7943              : {
    7944              : public:
    7945       297180 :   pass_cse (gcc::context *ctxt)
    7946       594360 :     : rtl_opt_pass (pass_data_cse, ctxt)
    7947              :   {}
    7948              : 
    7949              :   /* opt_pass methods: */
    7950      1517135 :   bool gate (function *) final override { return optimize > 0; }
    7951      1064962 :   unsigned int execute (function *) final override
    7952              :   {
    7953      1064962 :     return rest_of_handle_cse ();
    7954              :   }
    7955              : 
    7956              : }; // class pass_cse
    7957              : 
    7958              : } // anon namespace
    7959              : 
    7960              : rtl_opt_pass *
    7961       297180 : make_pass_cse (gcc::context *ctxt)
    7962              : {
    7963       297180 :   return new pass_cse (ctxt);
    7964              : }
    7965              : 
    7966              : 
    7967              : /* Run second CSE pass after loop optimizations.  */
    7968              : static unsigned int
    7969       982740 : rest_of_handle_cse2 (void)
    7970              : {
    7971       982740 :   int tem;
    7972              : 
    7973       982740 :   if (dump_file)
    7974           22 :     dump_flow_info (dump_file, dump_flags);
    7975              : 
    7976       982740 :   tem = cse_main (get_insns (), max_reg_num ());
    7977              : 
    7978              :   /* Run a pass to eliminate duplicated assignments to condition code
    7979              :      registers.  We have to run this after bypass_jumps, because it
    7980              :      makes it harder for that pass to determine whether a jump can be
    7981              :      bypassed safely.  */
    7982       982740 :   cse_condition_code_reg ();
    7983              : 
    7984       982740 :   delete_trivially_dead_insns (get_insns (), max_reg_num ());
    7985              : 
    7986       982740 :   if (tem == 2)
    7987              :     {
    7988         2208 :       timevar_push (TV_JUMP);
    7989         2208 :       rebuild_jump_labels (get_insns ());
    7990         2208 :       cse_cfg_altered |= cleanup_cfg (CLEANUP_CFG_CHANGED);
    7991         2208 :       timevar_pop (TV_JUMP);
    7992              :     }
    7993       980532 :   else if (tem == 1 || cse_cfg_altered)
    7994          108 :     cse_cfg_altered |= cleanup_cfg (0);
    7995              : 
    7996       982740 :   cse_not_expected = 1;
    7997       982740 :   return 0;
    7998              : }
    7999              : 
    8000              : 
    8001              : namespace {
    8002              : 
    8003              : const pass_data pass_data_cse2 =
    8004              : {
    8005              :   RTL_PASS, /* type */
    8006              :   "cse2", /* name */
    8007              :   OPTGROUP_NONE, /* optinfo_flags */
    8008              :   TV_CSE2, /* tv_id */
    8009              :   0, /* properties_required */
    8010              :   0, /* properties_provided */
    8011              :   0, /* properties_destroyed */
    8012              :   0, /* todo_flags_start */
    8013              :   TODO_df_finish, /* todo_flags_finish */
    8014              : };
    8015              : 
    8016              : class pass_cse2 : public rtl_opt_pass
    8017              : {
    8018              : public:
    8019       297180 :   pass_cse2 (gcc::context *ctxt)
    8020       594360 :     : rtl_opt_pass (pass_data_cse2, ctxt)
    8021              :   {}
    8022              : 
    8023              :   /* opt_pass methods: */
    8024      1517135 :   bool gate (function *) final override
    8025              :     {
    8026      1517135 :       return optimize > 0 && flag_rerun_cse_after_loop;
    8027              :     }
    8028              : 
    8029       982740 :   unsigned int execute (function *) final override
    8030              :   {
    8031       982740 :     return rest_of_handle_cse2 ();
    8032              :   }
    8033              : 
    8034              : }; // class pass_cse2
    8035              : 
    8036              : } // anon namespace
    8037              : 
    8038              : rtl_opt_pass *
    8039       297180 : make_pass_cse2 (gcc::context *ctxt)
    8040              : {
    8041       297180 :   return new pass_cse2 (ctxt);
    8042              : }
    8043              : 
    8044              : /* Run second CSE pass after loop optimizations.  */
    8045              : static unsigned int
    8046       298257 : rest_of_handle_cse_after_global_opts (void)
    8047              : {
    8048       298257 :   int save_cfj;
    8049       298257 :   int tem;
    8050              : 
    8051              :   /* We only want to do local CSE, so don't follow jumps.  */
    8052       298257 :   save_cfj = flag_cse_follow_jumps;
    8053       298257 :   flag_cse_follow_jumps = 0;
    8054              : 
    8055       298257 :   rebuild_jump_labels (get_insns ());
    8056       298257 :   tem = cse_main (get_insns (), max_reg_num ());
    8057       298257 :   cse_cfg_altered |= purge_all_dead_edges ();
    8058       298257 :   delete_trivially_dead_insns (get_insns (), max_reg_num ());
    8059              : 
    8060       298257 :   cse_not_expected = !flag_rerun_cse_after_loop;
    8061              : 
    8062              :   /* If cse altered any jumps, rerun jump opts to clean things up.  */
    8063       298257 :   if (tem == 2)
    8064              :     {
    8065          260 :       timevar_push (TV_JUMP);
    8066          260 :       rebuild_jump_labels (get_insns ());
    8067          260 :       cse_cfg_altered |= cleanup_cfg (CLEANUP_CFG_CHANGED);
    8068          260 :       timevar_pop (TV_JUMP);
    8069              :     }
    8070       297997 :   else if (tem == 1 || cse_cfg_altered)
    8071         4786 :     cse_cfg_altered |= cleanup_cfg (0);
    8072              : 
    8073       298257 :   flag_cse_follow_jumps = save_cfj;
    8074       298257 :   return 0;
    8075              : }
    8076              : 
    8077              : namespace {
    8078              : 
    8079              : const pass_data pass_data_cse_after_global_opts =
    8080              : {
    8081              :   RTL_PASS, /* type */
    8082              :   "cse_local", /* name */
    8083              :   OPTGROUP_NONE, /* optinfo_flags */
    8084              :   TV_CSE, /* tv_id */
    8085              :   0, /* properties_required */
    8086              :   0, /* properties_provided */
    8087              :   0, /* properties_destroyed */
    8088              :   0, /* todo_flags_start */
    8089              :   TODO_df_finish, /* todo_flags_finish */
    8090              : };
    8091              : 
    8092              : class pass_cse_after_global_opts : public rtl_opt_pass
    8093              : {
    8094              : public:
    8095       297180 :   pass_cse_after_global_opts (gcc::context *ctxt)
    8096       594360 :     : rtl_opt_pass (pass_data_cse_after_global_opts, ctxt)
    8097              :   {}
    8098              : 
    8099              :   /* opt_pass methods: */
    8100      1517135 :   bool gate (function *) final override
    8101              :     {
    8102      1517135 :       return optimize > 0 && flag_rerun_cse_after_global_opts;
    8103              :     }
    8104              : 
    8105       298257 :   unsigned int execute (function *) final override
    8106              :     {
    8107       298257 :       return rest_of_handle_cse_after_global_opts ();
    8108              :     }
    8109              : 
    8110              : }; // class pass_cse_after_global_opts
    8111              : 
    8112              : } // anon namespace
    8113              : 
    8114              : rtl_opt_pass *
    8115       297180 : make_pass_cse_after_global_opts (gcc::context *ctxt)
    8116              : {
    8117       297180 :   return new pass_cse_after_global_opts (ctxt);
    8118              : }
        

Generated by: LCOV version 2.5.0-beta

LCOV profile is generated on x86_64 machine using following configure options: configure --disable-bootstrap --enable-coverage=opt --enable-languages=c,c++,fortran,go,jit,lto,rust,m2 --enable-host-shared. GCC test suite is run with the built compiler.