Line data Source code
1 : /* Integrated Register Allocator (IRA) entry point.
2 : Copyright (C) 2006-2026 Free Software Foundation, Inc.
3 : Contributed by Vladimir Makarov <vmakarov@redhat.com>.
4 :
5 : This file is part of GCC.
6 :
7 : GCC is free software; you can redistribute it and/or modify it under
8 : the terms of the GNU General Public License as published by the Free
9 : Software Foundation; either version 3, or (at your option) any later
10 : version.
11 :
12 : GCC is distributed in the hope that it will be useful, but WITHOUT ANY
13 : WARRANTY; without even the implied warranty of MERCHANTABILITY or
14 : FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
15 : for more details.
16 :
17 : You should have received a copy of the GNU General Public License
18 : along with GCC; see the file COPYING3. If not see
19 : <http://www.gnu.org/licenses/>. */
20 :
21 : /* The integrated register allocator (IRA) is a
22 : regional register allocator performing graph coloring on a top-down
23 : traversal of nested regions. Graph coloring in a region is based
24 : on Chaitin-Briggs algorithm. It is called integrated because
25 : register coalescing, register live range splitting, and choosing a
26 : better hard register are done on-the-fly during coloring. Register
27 : coalescing and choosing a cheaper hard register is done by hard
28 : register preferencing during hard register assigning. The live
29 : range splitting is a byproduct of the regional register allocation.
30 :
31 : Major IRA notions are:
32 :
33 : o *Region* is a part of CFG where graph coloring based on
34 : Chaitin-Briggs algorithm is done. IRA can work on any set of
35 : nested CFG regions forming a tree. Currently the regions are
36 : the entire function for the root region and natural loops for
37 : the other regions. Therefore data structure representing a
38 : region is called loop_tree_node.
39 :
40 : o *Allocno class* is a register class used for allocation of
41 : given allocno. It means that only hard register of given
42 : register class can be assigned to given allocno. In reality,
43 : even smaller subset of (*profitable*) hard registers can be
44 : assigned. In rare cases, the subset can be even smaller
45 : because our modification of Chaitin-Briggs algorithm requires
46 : that sets of hard registers can be assigned to allocnos forms a
47 : forest, i.e. the sets can be ordered in a way where any
48 : previous set is not intersected with given set or is a superset
49 : of given set.
50 :
51 : o *Pressure class* is a register class belonging to a set of
52 : register classes containing all of the hard-registers available
53 : for register allocation. The set of all pressure classes for a
54 : target is defined in the corresponding machine-description file
55 : according some criteria. Register pressure is calculated only
56 : for pressure classes and it affects some IRA decisions as
57 : forming allocation regions.
58 :
59 : o *Allocno* represents the live range of a pseudo-register in a
60 : region. Besides the obvious attributes like the corresponding
61 : pseudo-register number, allocno class, conflicting allocnos and
62 : conflicting hard-registers, there are a few allocno attributes
63 : which are important for understanding the allocation algorithm:
64 :
65 : - *Live ranges*. This is a list of ranges of *program points*
66 : where the allocno lives. Program points represent places
67 : where a pseudo can be born or become dead (there are
68 : approximately two times more program points than the insns)
69 : and they are represented by integers starting with 0. The
70 : live ranges are used to find conflicts between allocnos.
71 : They also play very important role for the transformation of
72 : the IRA internal representation of several regions into a one
73 : region representation. The later is used during the reload
74 : pass work because each allocno represents all of the
75 : corresponding pseudo-registers.
76 :
77 : - *Hard-register costs*. This is a vector of size equal to the
78 : number of available hard-registers of the allocno class. The
79 : cost of a callee-clobbered hard-register for an allocno is
80 : increased by the cost of save/restore code around the calls
81 : through the given allocno's life. If the allocno is a move
82 : instruction operand and another operand is a hard-register of
83 : the allocno class, the cost of the hard-register is decreased
84 : by the move cost.
85 :
86 : When an allocno is assigned, the hard-register with minimal
87 : full cost is used. Initially, a hard-register's full cost is
88 : the corresponding value from the hard-register's cost vector.
89 : If the allocno is connected by a *copy* (see below) to
90 : another allocno which has just received a hard-register, the
91 : cost of the hard-register is decreased. Before choosing a
92 : hard-register for an allocno, the allocno's current costs of
93 : the hard-registers are modified by the conflict hard-register
94 : costs of all of the conflicting allocnos which are not
95 : assigned yet.
96 :
97 : - *Conflict hard-register costs*. This is a vector of the same
98 : size as the hard-register costs vector. To permit an
99 : unassigned allocno to get a better hard-register, IRA uses
100 : this vector to calculate the final full cost of the
101 : available hard-registers. Conflict hard-register costs of an
102 : unassigned allocno are also changed with a change of the
103 : hard-register cost of the allocno when a copy involving the
104 : allocno is processed as described above. This is done to
105 : show other unassigned allocnos that a given allocno prefers
106 : some hard-registers in order to remove the move instruction
107 : corresponding to the copy.
108 :
109 : o *Cap*. If a pseudo-register does not live in a region but
110 : lives in a nested region, IRA creates a special allocno called
111 : a cap in the outer region. A region cap is also created for a
112 : subregion cap.
113 :
114 : o *Copy*. Allocnos can be connected by copies. Copies are used
115 : to modify hard-register costs for allocnos during coloring.
116 : Such modifications reflects a preference to use the same
117 : hard-register for the allocnos connected by copies. Usually
118 : copies are created for move insns (in this case it results in
119 : register coalescing). But IRA also creates copies for operands
120 : of an insn which should be assigned to the same hard-register
121 : due to constraints in the machine description (it usually
122 : results in removing a move generated in reload to satisfy
123 : the constraints) and copies referring to the allocno which is
124 : the output operand of an instruction and the allocno which is
125 : an input operand dying in the instruction (creation of such
126 : copies results in less register shuffling). IRA *does not*
127 : create copies between the same register allocnos from different
128 : regions because we use another technique for propagating
129 : hard-register preference on the borders of regions.
130 :
131 : Allocnos (including caps) for the upper region in the region tree
132 : *accumulate* information important for coloring from allocnos with
133 : the same pseudo-register from nested regions. This includes
134 : hard-register and memory costs, conflicts with hard-registers,
135 : allocno conflicts, allocno copies and more. *Thus, attributes for
136 : allocnos in a region have the same values as if the region had no
137 : subregions*. It means that attributes for allocnos in the
138 : outermost region corresponding to the function have the same values
139 : as though the allocation used only one region which is the entire
140 : function. It also means that we can look at IRA work as if the
141 : first IRA did allocation for all function then it improved the
142 : allocation for loops then their subloops and so on.
143 :
144 : IRA major passes are:
145 :
146 : o Building IRA internal representation which consists of the
147 : following subpasses:
148 :
149 : * First, IRA builds regions and creates allocnos (file
150 : ira-build.cc) and initializes most of their attributes.
151 :
152 : * Then IRA finds an allocno class for each allocno and
153 : calculates its initial (non-accumulated) cost of memory and
154 : each hard-register of its allocno class (file ira-cost.c).
155 :
156 : * IRA creates live ranges of each allocno, calculates register
157 : pressure for each pressure class in each region, sets up
158 : conflict hard registers for each allocno and info about calls
159 : the allocno lives through (file ira-lives.cc).
160 :
161 : * IRA removes low register pressure loops from the regions
162 : mostly to speed IRA up (file ira-build.cc).
163 :
164 : * IRA propagates accumulated allocno info from lower region
165 : allocnos to corresponding upper region allocnos (file
166 : ira-build.cc).
167 :
168 : * IRA creates all caps (file ira-build.cc).
169 :
170 : * Having live-ranges of allocnos and their classes, IRA creates
171 : conflicting allocnos for each allocno. Conflicting allocnos
172 : are stored as a bit vector or array of pointers to the
173 : conflicting allocnos whatever is more profitable (file
174 : ira-conflicts.cc). At this point IRA creates allocno copies.
175 :
176 : o Coloring. Now IRA has all necessary info to start graph coloring
177 : process. It is done in each region on top-down traverse of the
178 : region tree (file ira-color.cc). There are following subpasses:
179 :
180 : * Finding profitable hard registers of corresponding allocno
181 : class for each allocno. For example, only callee-saved hard
182 : registers are frequently profitable for allocnos living
183 : through colors. If the profitable hard register set of
184 : allocno does not form a tree based on subset relation, we use
185 : some approximation to form the tree. This approximation is
186 : used to figure out trivial colorability of allocnos. The
187 : approximation is a pretty rare case.
188 :
189 : * Putting allocnos onto the coloring stack. IRA uses Briggs
190 : optimistic coloring which is a major improvement over
191 : Chaitin's coloring. Therefore IRA does not spill allocnos at
192 : this point. There is some freedom in the order of putting
193 : allocnos on the stack which can affect the final result of
194 : the allocation. IRA uses some heuristics to improve the
195 : order. The major one is to form *threads* from colorable
196 : allocnos and push them on the stack by threads. Thread is a
197 : set of non-conflicting colorable allocnos connected by
198 : copies. The thread contains allocnos from the colorable
199 : bucket or colorable allocnos already pushed onto the coloring
200 : stack. Pushing thread allocnos one after another onto the
201 : stack increases chances of removing copies when the allocnos
202 : get the same hard reg.
203 :
204 : We also use a modification of Chaitin-Briggs algorithm which
205 : works for intersected register classes of allocnos. To
206 : figure out trivial colorability of allocnos, the mentioned
207 : above tree of hard register sets is used. To get an idea how
208 : the algorithm works in i386 example, let us consider an
209 : allocno to which any general hard register can be assigned.
210 : If the allocno conflicts with eight allocnos to which only
211 : EAX register can be assigned, given allocno is still
212 : trivially colorable because all conflicting allocnos might be
213 : assigned only to EAX and all other general hard registers are
214 : still free.
215 :
216 : To get an idea of the used trivial colorability criterion, it
217 : is also useful to read article "Graph-Coloring Register
218 : Allocation for Irregular Architectures" by Michael D. Smith
219 : and Glen Holloway. Major difference between the article
220 : approach and approach used in IRA is that Smith's approach
221 : takes register classes only from machine description and IRA
222 : calculate register classes from intermediate code too
223 : (e.g. an explicit usage of hard registers in RTL code for
224 : parameter passing can result in creation of additional
225 : register classes which contain or exclude the hard
226 : registers). That makes IRA approach useful for improving
227 : coloring even for architectures with regular register files
228 : and in fact some benchmarking shows the improvement for
229 : regular class architectures is even bigger than for irregular
230 : ones. Another difference is that Smith's approach chooses
231 : intersection of classes of all insn operands in which a given
232 : pseudo occurs. IRA can use bigger classes if it is still
233 : more profitable than memory usage.
234 :
235 : * Popping the allocnos from the stack and assigning them hard
236 : registers. If IRA cannot assign a hard register to an
237 : allocno and the allocno is coalesced, IRA undoes the
238 : coalescing and puts the uncoalesced allocnos onto the stack in
239 : the hope that some such allocnos will get a hard register
240 : separately. If IRA fails to assign hard register or memory
241 : is more profitable for it, IRA spills the allocno. IRA
242 : assigns the allocno the hard-register with minimal full
243 : allocation cost which reflects the cost of usage of the
244 : hard-register for the allocno and cost of usage of the
245 : hard-register for allocnos conflicting with given allocno.
246 :
247 : * Chaitin-Briggs coloring assigns as many pseudos as possible
248 : to hard registers. After coloring we try to improve
249 : allocation with cost point of view. We improve the
250 : allocation by spilling some allocnos and assigning the freed
251 : hard registers to other allocnos if it decreases the overall
252 : allocation cost.
253 :
254 : * After allocno assigning in the region, IRA modifies the hard
255 : register and memory costs for the corresponding allocnos in
256 : the subregions to reflect the cost of possible loads, stores,
257 : or moves on the border of the region and its subregions.
258 : When default regional allocation algorithm is used
259 : (-fira-algorithm=mixed), IRA just propagates the assignment
260 : for allocnos if the register pressure in the region for the
261 : corresponding pressure class is less than number of available
262 : hard registers for given pressure class.
263 :
264 : o Spill/restore code moving. When IRA performs an allocation
265 : by traversing regions in top-down order, it does not know what
266 : happens below in the region tree. Therefore, sometimes IRA
267 : misses opportunities to perform a better allocation. A simple
268 : optimization tries to improve allocation in a region having
269 : subregions and containing in another region. If the
270 : corresponding allocnos in the subregion are spilled, it spills
271 : the region allocno if it is profitable. The optimization
272 : implements a simple iterative algorithm performing profitable
273 : transformations while they are still possible. It is fast in
274 : practice, so there is no real need for a better time complexity
275 : algorithm.
276 :
277 : o Code change. After coloring, two allocnos representing the
278 : same pseudo-register outside and inside a region respectively
279 : may be assigned to different locations (hard-registers or
280 : memory). In this case IRA creates and uses a new
281 : pseudo-register inside the region and adds code to move allocno
282 : values on the region's borders. This is done during top-down
283 : traversal of the regions (file ira-emit.cc). In some
284 : complicated cases IRA can create a new allocno to move allocno
285 : values (e.g. when a swap of values stored in two hard-registers
286 : is needed). At this stage, the new allocno is marked as
287 : spilled. IRA still creates the pseudo-register and the moves
288 : on the region borders even when both allocnos were assigned to
289 : the same hard-register. If the reload pass spills a
290 : pseudo-register for some reason, the effect will be smaller
291 : because another allocno will still be in the hard-register. In
292 : most cases, this is better then spilling both allocnos. If
293 : reload does not change the allocation for the two
294 : pseudo-registers, the trivial move will be removed by
295 : post-reload optimizations. IRA does not generate moves for
296 : allocnos assigned to the same hard register when the default
297 : regional allocation algorithm is used and the register pressure
298 : in the region for the corresponding pressure class is less than
299 : number of available hard registers for given pressure class.
300 : IRA also does some optimizations to remove redundant stores and
301 : to reduce code duplication on the region borders.
302 :
303 : o Flattening internal representation. After changing code, IRA
304 : transforms its internal representation for several regions into
305 : one region representation (file ira-build.cc). This process is
306 : called IR flattening. Such process is more complicated than IR
307 : rebuilding would be, but is much faster.
308 :
309 : o After IR flattening, IRA tries to assign hard registers to all
310 : spilled allocnos. This is implemented by a simple and fast
311 : priority coloring algorithm (see function
312 : ira_reassign_conflict_allocnos::ira-color.cc). Here new allocnos
313 : created during the code change pass can be assigned to hard
314 : registers.
315 :
316 : o At the end IRA calls the reload pass. The reload pass
317 : communicates with IRA through several functions in file
318 : ira-color.cc to improve its decisions in
319 :
320 : * sharing stack slots for the spilled pseudos based on IRA info
321 : about pseudo-register conflicts.
322 :
323 : * reassigning hard-registers to all spilled pseudos at the end
324 : of each reload iteration.
325 :
326 : * choosing a better hard-register to spill based on IRA info
327 : about pseudo-register live ranges and the register pressure
328 : in places where the pseudo-register lives.
329 :
330 : IRA uses a lot of data representing the target processors. These
331 : data are initialized in file ira.cc.
332 :
333 : If function has no loops (or the loops are ignored when
334 : -fira-algorithm=CB is used), we have classic Chaitin-Briggs
335 : coloring (only instead of separate pass of coalescing, we use hard
336 : register preferencing). In such case, IRA works much faster
337 : because many things are not made (like IR flattening, the
338 : spill/restore optimization, and the code change).
339 :
340 : Literature is worth to read for better understanding the code:
341 :
342 : o Preston Briggs, Keith D. Cooper, Linda Torczon. Improvements to
343 : Graph Coloring Register Allocation.
344 :
345 : o David Callahan, Brian Koblenz. Register allocation via
346 : hierarchical graph coloring.
347 :
348 : o Keith Cooper, Anshuman Dasgupta, Jason Eckhardt. Revisiting Graph
349 : Coloring Register Allocation: A Study of the Chaitin-Briggs and
350 : Callahan-Koblenz Algorithms.
351 :
352 : o Guei-Yuan Lueh, Thomas Gross, and Ali-Reza Adl-Tabatabai. Global
353 : Register Allocation Based on Graph Fusion.
354 :
355 : o Michael D. Smith and Glenn Holloway. Graph-Coloring Register
356 : Allocation for Irregular Architectures
357 :
358 : o Vladimir Makarov. The Integrated Register Allocator for GCC.
359 :
360 : o Vladimir Makarov. The top-down register allocator for irregular
361 : register file architectures.
362 :
363 : */
364 :
365 :
366 : #include "config.h"
367 : #include "system.h"
368 : #include "coretypes.h"
369 : #include "backend.h"
370 : #include "target.h"
371 : #include "rtl.h"
372 : #include "tree.h"
373 : #include "df.h"
374 : #include "memmodel.h"
375 : #include "tm_p.h"
376 : #include "insn-config.h"
377 : #include "regs.h"
378 : #include "ira.h"
379 : #include "ira-int.h"
380 : #include "diagnostic-core.h"
381 : #include "cfgrtl.h"
382 : #include "cfgbuild.h"
383 : #include "cfgcleanup.h"
384 : #include "expr.h"
385 : #include "tree-pass.h"
386 : #include "output.h"
387 : #include "reload.h"
388 : #include "cfgloop.h"
389 : #include "lra.h"
390 : #include "dce.h"
391 : #include "dbgcnt.h"
392 : #include "rtl-iter.h"
393 : #include "shrink-wrap.h"
394 : #include "print-rtl.h"
395 :
396 : struct target_ira default_target_ira;
397 : class target_ira_int default_target_ira_int;
398 : #if SWITCHABLE_TARGET
399 : struct target_ira *this_target_ira = &default_target_ira;
400 : class target_ira_int *this_target_ira_int = &default_target_ira_int;
401 : #endif
402 :
403 : /* A modified value of flag `-fira-verbose' used internally. */
404 : int internal_flag_ira_verbose;
405 :
406 : /* Dump file of the allocator if it is not NULL. */
407 : FILE *ira_dump_file;
408 :
409 : /* The number of elements in the following array. */
410 : int ira_spilled_reg_stack_slots_num;
411 :
412 : /* The following array contains info about spilled pseudo-registers
413 : stack slots used in current function so far. */
414 : class ira_spilled_reg_stack_slot *ira_spilled_reg_stack_slots;
415 :
416 : /* Correspondingly overall cost of the allocation, overall cost before
417 : reload, cost of the allocnos assigned to hard-registers, cost of
418 : the allocnos assigned to memory, cost of loads, stores and register
419 : move insns generated for pseudo-register live range splitting (see
420 : ira-emit.cc). */
421 : int64_t ira_overall_cost, overall_cost_before;
422 : int64_t ira_reg_cost, ira_mem_cost;
423 : int64_t ira_load_cost, ira_store_cost, ira_shuffle_cost;
424 : int ira_move_loops_num, ira_additional_jumps_num;
425 :
426 : /* All registers that can be eliminated. */
427 :
428 : HARD_REG_SET eliminable_regset;
429 :
430 : /* Value of max_reg_num () before IRA work start. This value helps
431 : us to recognize a situation when new pseudos were created during
432 : IRA work. */
433 : static int max_regno_before_ira;
434 :
435 : /* Temporary hard reg set used for a different calculation. */
436 : static HARD_REG_SET temp_hard_regset;
437 :
438 : #define last_mode_for_init_move_cost \
439 : (this_target_ira_int->x_last_mode_for_init_move_cost)
440 :
441 :
442 : /* The function sets up the map IRA_REG_MODE_HARD_REGSET. */
443 : static void
444 222149 : setup_reg_mode_hard_regset (void)
445 : {
446 222149 : int i, m, hard_regno;
447 :
448 27768625 : for (m = 0; m < NUM_MACHINE_MODES; m++)
449 2616915220 : for (hard_regno = 0; hard_regno < FIRST_PSEUDO_REGISTER; hard_regno++)
450 : {
451 2589368744 : CLEAR_HARD_REG_SET (ira_reg_mode_hard_regset[hard_regno][m]);
452 7787469372 : for (i = hard_regno_nregs (hard_regno, (machine_mode) m) - 1;
453 7787469372 : i >= 0; i--)
454 5198100628 : if (hard_regno + i < FIRST_PSEUDO_REGISTER)
455 4902526366 : SET_HARD_REG_BIT (ira_reg_mode_hard_regset[hard_regno][m],
456 : hard_regno + i);
457 : }
458 222149 : }
459 :
460 :
461 : #define no_unit_alloc_regs \
462 : (this_target_ira_int->x_no_unit_alloc_regs)
463 :
464 : /* The function sets up the three arrays declared above. */
465 : static void
466 222149 : setup_class_hard_regs (void)
467 : {
468 222149 : int cl, i, hard_regno, n;
469 222149 : unsigned int j;
470 222149 : HARD_REG_SET processed_hard_reg_set;
471 :
472 222149 : ira_assert (SHRT_MAX >= FIRST_PSEUDO_REGISTER);
473 7775215 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
474 : {
475 7553066 : temp_hard_regset = reg_class_contents[cl] & ~no_unit_alloc_regs;
476 7553066 : CLEAR_HARD_REG_SET (processed_hard_reg_set);
477 717541270 : for (i = 0; i < FIRST_PSEUDO_REGISTER; i++)
478 : {
479 709988204 : ira_non_ordered_class_hard_regs[cl][i] = -1;
480 709988204 : ira_class_hard_reg_index[cl][i] = -1;
481 : }
482 717541270 : for (n = 0, i = 0; i < FIRST_PSEUDO_REGISTER; i++)
483 : {
484 : #ifdef REG_ALLOC_ORDER
485 709988204 : hard_regno = reg_alloc_order[i];
486 : #else
487 : hard_regno = i;
488 : #endif
489 709988204 : if (TEST_HARD_REG_BIT (processed_hard_reg_set, hard_regno))
490 45318396 : continue;
491 664669808 : SET_HARD_REG_BIT (processed_hard_reg_set, hard_regno);
492 664669808 : if (! TEST_HARD_REG_BIT (temp_hard_regset, hard_regno))
493 588095366 : ira_class_hard_reg_index[cl][hard_regno] = -1;
494 : else
495 : {
496 76574442 : ira_class_hard_reg_index[cl][hard_regno] = n;
497 76574442 : ira_class_hard_regs[cl][n++] = hard_regno;
498 : }
499 : }
500 7553066 : ira_class_hard_regs_num[cl] = n;
501 7553066 : n = 0;
502 7553066 : j = 0;
503 7553066 : hard_reg_set_iterator hrsi;
504 84127508 : EXECUTE_IF_SET_IN_HARD_REG_SET (temp_hard_regset, 0, j, hrsi)
505 76574442 : ira_non_ordered_class_hard_regs[cl][n++] = j;
506 :
507 7553066 : ira_assert (ira_class_hard_regs_num[cl] == n);
508 : }
509 222149 : }
510 :
511 : /* Set up global variables defining info about hard registers for the
512 : allocation. These depend on USE_HARD_FRAME_P whose TRUE value means
513 : that we can use the hard frame pointer for the allocation. */
514 : static void
515 222149 : setup_alloc_regs (bool use_hard_frame_p)
516 : {
517 : #ifdef ADJUST_REG_ALLOC_ORDER
518 222149 : ADJUST_REG_ALLOC_ORDER;
519 : #endif
520 222149 : no_unit_alloc_regs = fixed_nonglobal_reg_set;
521 222149 : if (! use_hard_frame_p)
522 69380 : add_to_hard_reg_set (&no_unit_alloc_regs, Pmode,
523 : HARD_FRAME_POINTER_REGNUM);
524 222149 : setup_class_hard_regs ();
525 222149 : }
526 :
527 :
528 :
529 : #define alloc_reg_class_subclasses \
530 : (this_target_ira_int->x_alloc_reg_class_subclasses)
531 :
532 : /* Initialize the table of subclasses of each reg class. */
533 : static void
534 222149 : setup_reg_subclasses (void)
535 : {
536 222149 : int i, j;
537 222149 : HARD_REG_SET temp_hard_regset2;
538 :
539 7775215 : for (i = 0; i < N_REG_CLASSES; i++)
540 264357310 : for (j = 0; j < N_REG_CLASSES; j++)
541 256804244 : alloc_reg_class_subclasses[i][j] = LIM_REG_CLASSES;
542 :
543 7775215 : for (i = 0; i < N_REG_CLASSES; i++)
544 : {
545 7553066 : if (i == (int) NO_REGS)
546 222149 : continue;
547 :
548 7330917 : temp_hard_regset = reg_class_contents[i] & ~no_unit_alloc_regs;
549 14661834 : if (hard_reg_set_empty_p (temp_hard_regset))
550 440308 : continue;
551 241171315 : for (j = 0; j < N_REG_CLASSES; j++)
552 234280706 : if (i != j)
553 : {
554 227390097 : enum reg_class *p;
555 :
556 227390097 : temp_hard_regset2 = reg_class_contents[j] & ~no_unit_alloc_regs;
557 454780194 : if (! hard_reg_set_subset_p (temp_hard_regset,
558 : temp_hard_regset2))
559 173279275 : continue;
560 54110822 : p = &alloc_reg_class_subclasses[j][0];
561 520455245 : while (*p != LIM_REG_CLASSES) p++;
562 54110822 : *p = (enum reg_class) i;
563 : }
564 : }
565 222149 : }
566 :
567 :
568 :
569 : /* Set up IRA_MEMORY_MOVE_COST and IRA_MAX_MEMORY_MOVE_COST. */
570 : static void
571 222149 : setup_class_subset_and_memory_move_costs (void)
572 : {
573 222149 : int cl, cl2, mode, cost;
574 222149 : HARD_REG_SET temp_hard_regset2;
575 :
576 27768625 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
577 27546476 : ira_memory_move_cost[mode][NO_REGS][0]
578 27546476 : = ira_memory_move_cost[mode][NO_REGS][1] = SHRT_MAX;
579 7775215 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
580 : {
581 7553066 : if (cl != (int) NO_REGS)
582 916364625 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
583 : {
584 1818067416 : ira_max_memory_move_cost[mode][cl][0]
585 909033708 : = ira_memory_move_cost[mode][cl][0]
586 909033708 : = memory_move_cost ((machine_mode) mode,
587 : (reg_class_t) cl, false);
588 1818067416 : ira_max_memory_move_cost[mode][cl][1]
589 909033708 : = ira_memory_move_cost[mode][cl][1]
590 909033708 : = memory_move_cost ((machine_mode) mode,
591 : (reg_class_t) cl, true);
592 : /* Costs for NO_REGS are used in cost calculation on the
593 : 1st pass when the preferred register classes are not
594 : known yet. In this case we take the best scenario. */
595 909033708 : if (!targetm.hard_regno_mode_ok (ira_class_hard_regs[cl][0],
596 : (machine_mode) mode))
597 670323945 : continue;
598 :
599 238709763 : if (ira_memory_move_cost[mode][NO_REGS][0]
600 238709763 : > ira_memory_move_cost[mode][cl][0])
601 13416994 : ira_max_memory_move_cost[mode][NO_REGS][0]
602 13416994 : = ira_memory_move_cost[mode][NO_REGS][0]
603 13416994 : = ira_memory_move_cost[mode][cl][0];
604 238709763 : if (ira_memory_move_cost[mode][NO_REGS][1]
605 238709763 : > ira_memory_move_cost[mode][cl][1])
606 13407705 : ira_max_memory_move_cost[mode][NO_REGS][1]
607 13407705 : = ira_memory_move_cost[mode][NO_REGS][1]
608 13407705 : = ira_memory_move_cost[mode][cl][1];
609 : }
610 : }
611 7775215 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
612 264357310 : for (cl2 = (int) N_REG_CLASSES - 1; cl2 >= 0; cl2--)
613 : {
614 256804244 : temp_hard_regset = reg_class_contents[cl] & ~no_unit_alloc_regs;
615 256804244 : temp_hard_regset2 = reg_class_contents[cl2] & ~no_unit_alloc_regs;
616 256804244 : ira_class_subset_p[cl][cl2]
617 256804244 : = hard_reg_set_subset_p (temp_hard_regset, temp_hard_regset2);
618 256804244 : if (! hard_reg_set_empty_p (temp_hard_regset2)
619 491084950 : && hard_reg_set_subset_p (reg_class_contents[cl2],
620 : reg_class_contents[cl]))
621 6913744875 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
622 : {
623 6858434916 : cost = ira_memory_move_cost[mode][cl2][0];
624 6858434916 : if (cost > ira_max_memory_move_cost[mode][cl][0])
625 117092007 : ira_max_memory_move_cost[mode][cl][0] = cost;
626 6858434916 : cost = ira_memory_move_cost[mode][cl2][1];
627 6858434916 : if (cost > ira_max_memory_move_cost[mode][cl][1])
628 117146707 : ira_max_memory_move_cost[mode][cl][1] = cost;
629 : }
630 : }
631 7775215 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
632 944133250 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
633 : {
634 936580184 : ira_memory_move_cost[mode][cl][0]
635 936580184 : = ira_max_memory_move_cost[mode][cl][0];
636 936580184 : ira_memory_move_cost[mode][cl][1]
637 936580184 : = ira_max_memory_move_cost[mode][cl][1];
638 : }
639 222149 : setup_reg_subclasses ();
640 222149 : }
641 :
642 :
643 :
644 : /* Define the following macro if allocation through malloc if
645 : preferable. */
646 : #define IRA_NO_OBSTACK
647 :
648 : #ifndef IRA_NO_OBSTACK
649 : /* Obstack used for storing all dynamic data (except bitmaps) of the
650 : IRA. */
651 : static struct obstack ira_obstack;
652 : #endif
653 :
654 : /* Obstack used for storing all bitmaps of the IRA. */
655 : static struct bitmap_obstack ira_bitmap_obstack;
656 :
657 : /* Allocate memory of size LEN for IRA data. */
658 : void *
659 210191736 : ira_allocate (size_t len)
660 : {
661 210191736 : void *res;
662 :
663 : #ifndef IRA_NO_OBSTACK
664 : res = obstack_alloc (&ira_obstack, len);
665 : #else
666 210191736 : res = xmalloc (len);
667 : #endif
668 210191736 : return res;
669 : }
670 :
671 : /* Free memory ADDR allocated for IRA data. */
672 : void
673 210191736 : ira_free (void *addr ATTRIBUTE_UNUSED)
674 : {
675 : #ifndef IRA_NO_OBSTACK
676 : /* do nothing */
677 : #else
678 210191736 : free (addr);
679 : #endif
680 210191736 : }
681 :
682 :
683 : /* Allocate and returns bitmap for IRA. */
684 : bitmap
685 10841262 : ira_allocate_bitmap (void)
686 : {
687 10841262 : return BITMAP_ALLOC (&ira_bitmap_obstack);
688 : }
689 :
690 : /* Free bitmap B allocated for IRA. */
691 : void
692 10841262 : ira_free_bitmap (bitmap b ATTRIBUTE_UNUSED)
693 : {
694 : /* do nothing */
695 10841262 : }
696 :
697 :
698 :
699 : /* Output information about allocation of all allocnos (except for
700 : caps) into file F. */
701 : void
702 95 : ira_print_disposition (FILE *f)
703 : {
704 95 : int i, n, max_regno;
705 95 : ira_allocno_t a;
706 95 : basic_block bb;
707 :
708 95 : fprintf (f, "Disposition:");
709 95 : max_regno = max_reg_num ();
710 1981 : for (n = 0, i = FIRST_PSEUDO_REGISTER; i < max_regno; i++)
711 1791 : for (a = ira_regno_allocno_map[i];
712 2386 : a != NULL;
713 595 : a = ALLOCNO_NEXT_REGNO_ALLOCNO (a))
714 : {
715 595 : if (n % 4 == 0)
716 178 : fprintf (f, "\n");
717 595 : n++;
718 595 : fprintf (f, " %4d:r%-4d", ALLOCNO_NUM (a), ALLOCNO_REGNO (a));
719 595 : if ((bb = ALLOCNO_LOOP_TREE_NODE (a)->bb) != NULL)
720 0 : fprintf (f, "b%-3d", bb->index);
721 : else
722 595 : fprintf (f, "l%-3d", ALLOCNO_LOOP_TREE_NODE (a)->loop_num);
723 595 : if (ALLOCNO_HARD_REGNO (a) >= 0)
724 594 : fprintf (f, " %3d", ALLOCNO_HARD_REGNO (a));
725 : else
726 1 : fprintf (f, " mem");
727 : }
728 95 : fprintf (f, "\n");
729 95 : }
730 :
731 : /* Outputs information about allocation of all allocnos into
732 : stderr. */
733 : void
734 0 : ira_debug_disposition (void)
735 : {
736 0 : ira_print_disposition (stderr);
737 0 : }
738 :
739 :
740 :
741 : /* Set up ira_stack_reg_pressure_class which is the biggest pressure
742 : register class containing stack registers or NO_REGS if there are
743 : no stack registers. To find this class, we iterate through all
744 : register pressure classes and choose the first register pressure
745 : class containing all the stack registers and having the biggest
746 : size. */
747 : static void
748 222149 : setup_stack_reg_pressure_class (void)
749 : {
750 222149 : ira_stack_reg_pressure_class = NO_REGS;
751 : #ifdef STACK_REGS
752 222149 : {
753 222149 : int i, best, size;
754 222149 : enum reg_class cl;
755 222149 : HARD_REG_SET temp_hard_regset2;
756 :
757 222149 : CLEAR_HARD_REG_SET (temp_hard_regset);
758 1999341 : for (i = FIRST_STACK_REG; i <= LAST_STACK_REG; i++)
759 1777192 : SET_HARD_REG_BIT (temp_hard_regset, i);
760 : best = 0;
761 1112857 : for (i = 0; i < ira_pressure_classes_num; i++)
762 : {
763 890708 : cl = ira_pressure_classes[i];
764 890708 : temp_hard_regset2 = temp_hard_regset & reg_class_contents[cl];
765 890708 : size = hard_reg_set_popcount (temp_hard_regset2);
766 890708 : if (best < size)
767 : {
768 221617 : best = size;
769 221617 : ira_stack_reg_pressure_class = cl;
770 : }
771 : }
772 : }
773 : #endif
774 222149 : }
775 :
776 : /* Find pressure classes which are register classes for which we
777 : calculate register pressure in IRA, register pressure sensitive
778 : insn scheduling, and register pressure sensitive loop invariant
779 : motion.
780 :
781 : To make register pressure calculation easy, we always use
782 : non-intersected register pressure classes. A move of hard
783 : registers from one register pressure class is not more expensive
784 : than load and store of the hard registers. Most likely an allocno
785 : class will be a subset of a register pressure class and in many
786 : cases a register pressure class. That makes usage of register
787 : pressure classes a good approximation to find a high register
788 : pressure. */
789 : static void
790 222149 : setup_pressure_classes (void)
791 : {
792 222149 : int cost, i, n, curr;
793 222149 : int cl, cl2;
794 222149 : enum reg_class pressure_classes[N_REG_CLASSES];
795 222149 : int m;
796 222149 : HARD_REG_SET temp_hard_regset2;
797 222149 : bool insert_p;
798 :
799 222149 : if (targetm.compute_pressure_classes)
800 0 : n = targetm.compute_pressure_classes (pressure_classes);
801 : else
802 : {
803 : n = 0;
804 7775215 : for (cl = 0; cl < N_REG_CLASSES; cl++)
805 : {
806 7553066 : if (ira_class_hard_regs_num[cl] == 0)
807 662457 : continue;
808 6890609 : if (ira_class_hard_regs_num[cl] != 1
809 : /* A register class without subclasses may contain a few
810 : hard registers and movement between them is costly
811 : (e.g. SPARC FPCC registers). We still should consider it
812 : as a candidate for a pressure class. */
813 4892939 : && alloc_reg_class_subclasses[cl][0] < cl)
814 : {
815 : /* Check that the moves between any hard registers of the
816 : current class are not more expensive for a legal mode
817 : than load/store of the hard registers of the current
818 : class. Such class is a potential candidate to be a
819 : register pressure class. */
820 227223524 : for (m = 0; m < NUM_MACHINE_MODES; m++)
821 : {
822 225891145 : temp_hard_regset
823 225891145 : = (reg_class_contents[cl]
824 225891145 : & ~(no_unit_alloc_regs
825 225891145 : | ira_prohibited_class_mode_regs[cl][m]));
826 451782290 : if (hard_reg_set_empty_p (temp_hard_regset))
827 166977987 : continue;
828 58913158 : ira_init_register_move_cost_if_necessary ((machine_mode) m);
829 58913158 : cost = ira_register_move_cost[m][cl][cl];
830 58913158 : if (cost <= ira_max_memory_move_cost[m][cl][1]
831 55577993 : || cost <= ira_max_memory_move_cost[m][cl][0])
832 : break;
833 : }
834 4667544 : if (m >= NUM_MACHINE_MODES)
835 1332379 : continue;
836 : }
837 5558230 : curr = 0;
838 5558230 : insert_p = true;
839 5558230 : temp_hard_regset = reg_class_contents[cl] & ~no_unit_alloc_regs;
840 : /* Remove so far added pressure classes which are subset of the
841 : current candidate class. Prefer GENERAL_REGS as a pressure
842 : register class to another class containing the same
843 : allocatable hard registers. We do this because machine
844 : dependent cost hooks might give wrong costs for the latter
845 : class but always give the right cost for the former class
846 : (GENERAL_REGS). */
847 19943521 : for (i = 0; i < n; i++)
848 : {
849 14385291 : cl2 = pressure_classes[i];
850 14385291 : temp_hard_regset2 = (reg_class_contents[cl2]
851 14385291 : & ~no_unit_alloc_regs);
852 14385291 : if (hard_reg_set_subset_p (temp_hard_regset, temp_hard_regset2)
853 15570493 : && (temp_hard_regset != temp_hard_regset2
854 1115993 : || cl2 == (int) GENERAL_REGS))
855 : {
856 735469 : pressure_classes[curr++] = (enum reg_class) cl2;
857 735469 : insert_p = false;
858 735469 : continue;
859 : }
860 17137797 : if (hard_reg_set_subset_p (temp_hard_regset2, temp_hard_regset)
861 17581875 : && (temp_hard_regset2 != temp_hard_regset
862 449733 : || cl == (int) GENERAL_REGS))
863 3487975 : continue;
864 10161847 : if (temp_hard_regset2 == temp_hard_regset)
865 444078 : insert_p = false;
866 10161847 : pressure_classes[curr++] = (enum reg_class) cl2;
867 : }
868 : /* If the current candidate is a subset of a so far added
869 : pressure class, don't add it to the list of the pressure
870 : classes. */
871 5558230 : if (insert_p)
872 4378683 : pressure_classes[curr++] = (enum reg_class) cl;
873 : n = curr;
874 : }
875 : }
876 : #ifdef ENABLE_IRA_CHECKING
877 222149 : {
878 222149 : HARD_REG_SET ignore_hard_regs;
879 :
880 : /* Check pressure classes correctness: here we check that hard
881 : registers from all register pressure classes contains all hard
882 : registers available for the allocation. */
883 888596 : CLEAR_HARD_REG_SET (temp_hard_regset);
884 222149 : CLEAR_HARD_REG_SET (temp_hard_regset2);
885 222149 : ignore_hard_regs = no_unit_alloc_regs;
886 7775215 : for (cl = 0; cl < LIM_REG_CLASSES; cl++)
887 : {
888 : /* For some targets (like MIPS with MD_REGS), there are some
889 : classes with hard registers available for allocation but
890 : not able to hold value of any mode. */
891 214553420 : for (m = 0; m < NUM_MACHINE_MODES; m++)
892 213890963 : if (contains_reg_of_mode[cl][m])
893 : break;
894 7553066 : if (m >= NUM_MACHINE_MODES)
895 : {
896 662457 : ignore_hard_regs |= reg_class_contents[cl];
897 662457 : continue;
898 : }
899 32308212 : for (i = 0; i < n; i++)
900 26308311 : if ((int) pressure_classes[i] == cl)
901 : break;
902 6890609 : temp_hard_regset2 |= reg_class_contents[cl];
903 6890609 : if (i < n)
904 7553066 : temp_hard_regset |= reg_class_contents[cl];
905 : }
906 21104155 : for (i = 0; i < FIRST_PSEUDO_REGISTER; i++)
907 : /* Some targets (like SPARC with ICC reg) have allocatable regs
908 : for which no reg class is defined. */
909 20882006 : if (REGNO_REG_CLASS (i) == NO_REGS)
910 888596 : SET_HARD_REG_BIT (ignore_hard_regs, i);
911 222149 : temp_hard_regset &= ~ignore_hard_regs;
912 222149 : temp_hard_regset2 &= ~ignore_hard_regs;
913 444298 : ira_assert (hard_reg_set_subset_p (temp_hard_regset2, temp_hard_regset));
914 : }
915 : #endif
916 222149 : ira_pressure_classes_num = 0;
917 1112857 : for (i = 0; i < n; i++)
918 : {
919 890708 : cl = (int) pressure_classes[i];
920 890708 : ira_reg_pressure_class_p[cl] = true;
921 890708 : ira_pressure_classes[ira_pressure_classes_num++] = (enum reg_class) cl;
922 : }
923 222149 : setup_stack_reg_pressure_class ();
924 222149 : }
925 :
926 : /* Set up IRA_UNIFORM_CLASS_P. Uniform class is a register class
927 : whose register move cost between any registers of the class is the
928 : same as for all its subclasses. We use the data to speed up the
929 : 2nd pass of calculations of allocno costs. */
930 : static void
931 222149 : setup_uniform_class_p (void)
932 : {
933 222149 : int i, cl, cl2, m;
934 :
935 7775215 : for (cl = 0; cl < N_REG_CLASSES; cl++)
936 : {
937 7553066 : ira_uniform_class_p[cl] = false;
938 7553066 : if (ira_class_hard_regs_num[cl] == 0)
939 662457 : continue;
940 : /* We cannot use alloc_reg_class_subclasses here because move
941 : cost hooks does not take into account that some registers are
942 : unavailable for the subtarget. E.g. for i686, INT_SSE_REGS
943 : is element of alloc_reg_class_subclasses for GENERAL_REGS
944 : because SSE regs are unavailable. */
945 26873393 : for (i = 0; (cl2 = reg_class_subclasses[cl][i]) != LIM_REG_CLASSES; i++)
946 : {
947 21315164 : if (ira_class_hard_regs_num[cl2] == 0)
948 437 : continue;
949 2525759984 : for (m = 0; m < NUM_MACHINE_MODES; m++)
950 2505777637 : if (contains_reg_of_mode[cl][m] && contains_reg_of_mode[cl2][m])
951 : {
952 602090130 : ira_init_register_move_cost_if_necessary ((machine_mode) m);
953 602090130 : if (ira_register_move_cost[m][cl][cl]
954 602090130 : != ira_register_move_cost[m][cl2][cl2])
955 : break;
956 : }
957 21314727 : if (m < NUM_MACHINE_MODES)
958 : break;
959 : }
960 6890609 : if (cl2 == LIM_REG_CLASSES)
961 5558229 : ira_uniform_class_p[cl] = true;
962 : }
963 222149 : }
964 :
965 : /* Set up IRA_ALLOCNO_CLASSES, IRA_ALLOCNO_CLASSES_NUM,
966 : IRA_IMPORTANT_CLASSES, and IRA_IMPORTANT_CLASSES_NUM.
967 :
968 : Target may have many subtargets and not all target hard registers can
969 : be used for allocation, e.g. x86 port in 32-bit mode cannot use
970 : hard registers introduced in x86-64 like r8-r15). Some classes
971 : might have the same allocatable hard registers, e.g. INDEX_REGS
972 : and GENERAL_REGS in x86 port in 32-bit mode. To decrease different
973 : calculations efforts we introduce allocno classes which contain
974 : unique non-empty sets of allocatable hard-registers.
975 :
976 : Pseudo class cost calculation in ira-costs.cc is very expensive.
977 : Therefore we are trying to decrease number of classes involved in
978 : such calculation. Register classes used in the cost calculation
979 : are called important classes. They are allocno classes and other
980 : non-empty classes whose allocatable hard register sets are inside
981 : of an allocno class hard register set. From the first sight, it
982 : looks like that they are just allocno classes. It is not true. In
983 : example of x86-port in 32-bit mode, allocno classes will contain
984 : GENERAL_REGS but not LEGACY_REGS (because allocatable hard
985 : registers are the same for the both classes). The important
986 : classes will contain GENERAL_REGS and LEGACY_REGS. It is done
987 : because a machine description insn constraint may refers for
988 : LEGACY_REGS and code in ira-costs.cc is mostly base on investigation
989 : of the insn constraints. */
990 : static void
991 222149 : setup_allocno_and_important_classes (void)
992 : {
993 222149 : int i, j, n, cl;
994 222149 : bool set_p;
995 222149 : HARD_REG_SET temp_hard_regset2;
996 222149 : static enum reg_class classes[LIM_REG_CLASSES + 1];
997 :
998 222149 : n = 0;
999 : /* Collect classes which contain unique sets of allocatable hard
1000 : registers. Prefer GENERAL_REGS to other classes containing the
1001 : same set of hard registers. */
1002 7775215 : for (i = 0; i < LIM_REG_CLASSES; i++)
1003 : {
1004 7553066 : temp_hard_regset = reg_class_contents[i] & ~no_unit_alloc_regs;
1005 98301784 : for (j = 0; j < n; j++)
1006 : {
1007 92525867 : cl = classes[j];
1008 92525867 : temp_hard_regset2 = reg_class_contents[cl] & ~no_unit_alloc_regs;
1009 185051734 : if (temp_hard_regset == temp_hard_regset2)
1010 : break;
1011 : }
1012 7553066 : if (j >= n || targetm.additional_allocno_class_p (i))
1013 5775917 : classes[n++] = (enum reg_class) i;
1014 1777149 : else if (i == GENERAL_REGS)
1015 : /* Prefer general regs. For i386 example, it means that
1016 : we prefer GENERAL_REGS over INDEX_REGS or LEGACY_REGS
1017 : (all of them consists of the same available hard
1018 : registers). */
1019 5655 : classes[j] = (enum reg_class) i;
1020 : }
1021 222149 : classes[n] = LIM_REG_CLASSES;
1022 :
1023 : /* Set up classes which can be used for allocnos as classes
1024 : containing non-empty unique sets of allocatable hard
1025 : registers. */
1026 222149 : ira_allocno_classes_num = 0;
1027 5998066 : for (i = 0; (cl = classes[i]) != LIM_REG_CLASSES; i++)
1028 5775917 : if (ira_class_hard_regs_num[cl] > 0)
1029 5553768 : ira_allocno_classes[ira_allocno_classes_num++] = (enum reg_class) cl;
1030 222149 : ira_important_classes_num = 0;
1031 : /* Add non-allocno classes containing to non-empty set of
1032 : allocatable hard regs. */
1033 7775215 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1034 7553066 : if (ira_class_hard_regs_num[cl] > 0)
1035 : {
1036 6890609 : temp_hard_regset = reg_class_contents[cl] & ~no_unit_alloc_regs;
1037 6890609 : set_p = false;
1038 106965592 : for (j = 0; j < ira_allocno_classes_num; j++)
1039 : {
1040 105628751 : temp_hard_regset2 = (reg_class_contents[ira_allocno_classes[j]]
1041 105628751 : & ~no_unit_alloc_regs);
1042 105628751 : if ((enum reg_class) cl == ira_allocno_classes[j])
1043 : break;
1044 100074983 : else if (hard_reg_set_subset_p (temp_hard_regset,
1045 : temp_hard_regset2))
1046 6955525 : set_p = true;
1047 : }
1048 6890609 : if (set_p && j >= ira_allocno_classes_num)
1049 1336841 : ira_important_classes[ira_important_classes_num++]
1050 1336841 : = (enum reg_class) cl;
1051 : }
1052 : /* Now add allocno classes to the important classes. */
1053 5775917 : for (j = 0; j < ira_allocno_classes_num; j++)
1054 5553768 : ira_important_classes[ira_important_classes_num++]
1055 5553768 : = ira_allocno_classes[j];
1056 7775215 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1057 : {
1058 7553066 : ira_reg_allocno_class_p[cl] = false;
1059 7553066 : ira_reg_pressure_class_p[cl] = false;
1060 : }
1061 5775917 : for (j = 0; j < ira_allocno_classes_num; j++)
1062 5553768 : ira_reg_allocno_class_p[ira_allocno_classes[j]] = true;
1063 222149 : setup_pressure_classes ();
1064 222149 : setup_uniform_class_p ();
1065 222149 : }
1066 :
1067 : /* Setup translation in CLASS_TRANSLATE of all classes into a class
1068 : given by array CLASSES of length CLASSES_NUM. The function is used
1069 : make translation any reg class to an allocno class or to an
1070 : pressure class. This translation is necessary for some
1071 : calculations when we can use only allocno or pressure classes and
1072 : such translation represents an approximate representation of all
1073 : classes.
1074 :
1075 : The translation in case when allocatable hard register set of a
1076 : given class is subset of allocatable hard register set of a class
1077 : in CLASSES is pretty simple. We use smallest classes from CLASSES
1078 : containing a given class. If allocatable hard register set of a
1079 : given class is not a subset of any corresponding set of a class
1080 : from CLASSES, we use the cheapest (with load/store point of view)
1081 : class from CLASSES whose set intersects with given class set. */
1082 : static void
1083 444298 : setup_class_translate_array (enum reg_class *class_translate,
1084 : int classes_num, enum reg_class *classes)
1085 : {
1086 444298 : int cl, mode;
1087 444298 : enum reg_class aclass, best_class, *cl_ptr;
1088 444298 : int i, cost, min_cost, best_cost;
1089 :
1090 15550430 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1091 15106132 : class_translate[cl] = NO_REGS;
1092 :
1093 6888774 : for (i = 0; i < classes_num; i++)
1094 : {
1095 6444476 : aclass = classes[i];
1096 47320050 : for (cl_ptr = &alloc_reg_class_subclasses[aclass][0];
1097 47320050 : (cl = *cl_ptr) != LIM_REG_CLASSES;
1098 : cl_ptr++)
1099 40875574 : if (class_translate[cl] == NO_REGS)
1100 6292584 : class_translate[cl] = aclass;
1101 6444476 : class_translate[aclass] = aclass;
1102 : }
1103 : /* For classes which are not fully covered by one of given classes
1104 : (in other words covered by more one given class), use the
1105 : cheapest class. */
1106 15550430 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1107 : {
1108 15106132 : if (cl == NO_REGS || class_translate[cl] != NO_REGS)
1109 13112248 : continue;
1110 : best_class = NO_REGS;
1111 : best_cost = INT_MAX;
1112 19113678 : for (i = 0; i < classes_num; i++)
1113 : {
1114 17119794 : aclass = classes[i];
1115 17119794 : temp_hard_regset = (reg_class_contents[aclass]
1116 17119794 : & reg_class_contents[cl]
1117 17119794 : & ~no_unit_alloc_regs);
1118 34239588 : if (! hard_reg_set_empty_p (temp_hard_regset))
1119 : {
1120 : min_cost = INT_MAX;
1121 361874125 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
1122 : {
1123 358979132 : cost = (ira_memory_move_cost[mode][aclass][0]
1124 358979132 : + ira_memory_move_cost[mode][aclass][1]);
1125 358979132 : if (min_cost > cost)
1126 : min_cost = cost;
1127 : }
1128 2894993 : if (best_class == NO_REGS || best_cost > min_cost)
1129 : {
1130 17119794 : best_class = aclass;
1131 17119794 : best_cost = min_cost;
1132 : }
1133 : }
1134 : }
1135 1993884 : class_translate[cl] = best_class;
1136 : }
1137 444298 : }
1138 :
1139 : /* Set up array IRA_ALLOCNO_CLASS_TRANSLATE and
1140 : IRA_PRESSURE_CLASS_TRANSLATE. */
1141 : static void
1142 222149 : setup_class_translate (void)
1143 : {
1144 222149 : setup_class_translate_array (ira_allocno_class_translate,
1145 222149 : ira_allocno_classes_num, ira_allocno_classes);
1146 222149 : setup_class_translate_array (ira_pressure_class_translate,
1147 222149 : ira_pressure_classes_num, ira_pressure_classes);
1148 222149 : }
1149 :
1150 : /* Order numbers of allocno classes in original target allocno class
1151 : array, -1 for non-allocno classes. */
1152 : static int allocno_class_order[N_REG_CLASSES];
1153 :
1154 : /* The function used to sort the important classes. */
1155 : static int
1156 185220780 : comp_reg_classes_func (const void *v1p, const void *v2p)
1157 : {
1158 185220780 : enum reg_class cl1 = *(const enum reg_class *) v1p;
1159 185220780 : enum reg_class cl2 = *(const enum reg_class *) v2p;
1160 185220780 : enum reg_class tcl1, tcl2;
1161 185220780 : int diff;
1162 :
1163 185220780 : tcl1 = ira_allocno_class_translate[cl1];
1164 185220780 : tcl2 = ira_allocno_class_translate[cl2];
1165 185220780 : if (tcl1 != NO_REGS && tcl2 != NO_REGS
1166 185220780 : && (diff = allocno_class_order[tcl1] - allocno_class_order[tcl2]) != 0)
1167 : return diff;
1168 8381408 : return (int) cl1 - (int) cl2;
1169 : }
1170 :
1171 : /* For correct work of function setup_reg_class_relation we need to
1172 : reorder important classes according to the order of their allocno
1173 : classes. It places important classes containing the same
1174 : allocatable hard register set adjacent to each other and allocno
1175 : class with the allocatable hard register set right after the other
1176 : important classes with the same set.
1177 :
1178 : In example from comments of function
1179 : setup_allocno_and_important_classes, it places LEGACY_REGS and
1180 : GENERAL_REGS close to each other and GENERAL_REGS is after
1181 : LEGACY_REGS. */
1182 : static void
1183 222149 : reorder_important_classes (void)
1184 : {
1185 222149 : int i;
1186 :
1187 7775215 : for (i = 0; i < N_REG_CLASSES; i++)
1188 7553066 : allocno_class_order[i] = -1;
1189 5775917 : for (i = 0; i < ira_allocno_classes_num; i++)
1190 5553768 : allocno_class_order[ira_allocno_classes[i]] = i;
1191 222149 : qsort (ira_important_classes, ira_important_classes_num,
1192 : sizeof (enum reg_class), comp_reg_classes_func);
1193 7334907 : for (i = 0; i < ira_important_classes_num; i++)
1194 6890609 : ira_important_class_nums[ira_important_classes[i]] = i;
1195 222149 : }
1196 :
1197 : /* Set up IRA_REG_CLASS_SUBUNION, IRA_REG_CLASS_SUPERUNION,
1198 : IRA_REG_CLASS_SUPER_CLASSES, IRA_REG_CLASSES_INTERSECT, and
1199 : IRA_REG_CLASSES_INTERSECT_P. For the meaning of the relations,
1200 : please see corresponding comments in ira-int.h. */
1201 : static void
1202 222149 : setup_reg_class_relations (void)
1203 : {
1204 222149 : int i, cl1, cl2, cl3;
1205 222149 : HARD_REG_SET intersection_set, union_set, temp_set2;
1206 222149 : bool important_class_p[N_REG_CLASSES];
1207 :
1208 222149 : memset (important_class_p, 0, sizeof (important_class_p));
1209 7112758 : for (i = 0; i < ira_important_classes_num; i++)
1210 6890609 : important_class_p[ira_important_classes[i]] = true;
1211 7775215 : for (cl1 = 0; cl1 < N_REG_CLASSES; cl1++)
1212 : {
1213 7553066 : ira_reg_class_super_classes[cl1][0] = LIM_REG_CLASSES;
1214 264357310 : for (cl2 = 0; cl2 < N_REG_CLASSES; cl2++)
1215 : {
1216 256804244 : ira_reg_classes_intersect_p[cl1][cl2] = false;
1217 256804244 : ira_reg_class_intersect[cl1][cl2] = NO_REGS;
1218 256804244 : ira_reg_class_subset[cl1][cl2] = NO_REGS;
1219 256804244 : temp_hard_regset = reg_class_contents[cl1] & ~no_unit_alloc_regs;
1220 256804244 : temp_set2 = reg_class_contents[cl2] & ~no_unit_alloc_regs;
1221 256804244 : if (hard_reg_set_empty_p (temp_hard_regset)
1222 279327782 : && hard_reg_set_empty_p (temp_set2))
1223 : {
1224 : /* The both classes have no allocatable hard registers
1225 : -- take all class hard registers into account and use
1226 : reg_class_subunion and reg_class_superunion. */
1227 754730 : for (i = 0;; i++)
1228 : {
1229 2796539 : cl3 = reg_class_subclasses[cl1][i];
1230 2796539 : if (cl3 == LIM_REG_CLASSES)
1231 : break;
1232 754730 : if (reg_class_subset_p (ira_reg_class_intersect[cl1][cl2],
1233 : (enum reg_class) cl3))
1234 716836 : ira_reg_class_intersect[cl1][cl2] = (enum reg_class) cl3;
1235 : }
1236 2041809 : ira_reg_class_subunion[cl1][cl2] = reg_class_subunion[cl1][cl2];
1237 2041809 : ira_reg_class_superunion[cl1][cl2] = reg_class_superunion[cl1][cl2];
1238 2041809 : continue;
1239 : }
1240 : ira_reg_classes_intersect_p[cl1][cl2]
1241 254762435 : = hard_reg_set_intersect_p (temp_hard_regset, temp_set2);
1242 234280706 : if (important_class_p[cl1] && important_class_p[cl2]
1243 468561412 : && hard_reg_set_subset_p (temp_hard_regset, temp_set2))
1244 : {
1245 : /* CL1 and CL2 are important classes and CL1 allocatable
1246 : hard register set is inside of CL2 allocatable hard
1247 : registers -- make CL1 a superset of CL2. */
1248 61001431 : enum reg_class *p;
1249 :
1250 61001431 : p = &ira_reg_class_super_classes[cl1][0];
1251 368156619 : while (*p != LIM_REG_CLASSES)
1252 307155188 : p++;
1253 61001431 : *p++ = (enum reg_class) cl2;
1254 61001431 : *p = LIM_REG_CLASSES;
1255 : }
1256 254762435 : ira_reg_class_subunion[cl1][cl2] = NO_REGS;
1257 254762435 : ira_reg_class_superunion[cl1][cl2] = NO_REGS;
1258 254762435 : intersection_set = (reg_class_contents[cl1]
1259 254762435 : & reg_class_contents[cl2]
1260 254762435 : & ~no_unit_alloc_regs);
1261 254762435 : union_set = ((reg_class_contents[cl1] | reg_class_contents[cl2])
1262 254762435 : & ~no_unit_alloc_regs);
1263 8916685225 : for (cl3 = 0; cl3 < N_REG_CLASSES; cl3++)
1264 : {
1265 8661922790 : temp_hard_regset = reg_class_contents[cl3] & ~no_unit_alloc_regs;
1266 17323845580 : if (hard_reg_set_empty_p (temp_hard_regset))
1267 758966937 : continue;
1268 :
1269 7902955853 : if (hard_reg_set_subset_p (temp_hard_regset, intersection_set))
1270 : {
1271 : /* CL3 allocatable hard register set is inside of
1272 : intersection of allocatable hard register sets
1273 : of CL1 and CL2. */
1274 675311807 : if (important_class_p[cl3])
1275 : {
1276 675311807 : temp_set2
1277 675311807 : = (reg_class_contents
1278 675311807 : [ira_reg_class_intersect[cl1][cl2]]);
1279 675311807 : temp_set2 &= ~no_unit_alloc_regs;
1280 675311807 : if (! hard_reg_set_subset_p (temp_hard_regset, temp_set2)
1281 : /* If the allocatable hard register sets are
1282 : the same, prefer GENERAL_REGS or the
1283 : smallest class for debugging
1284 : purposes. */
1285 784951023 : || (temp_hard_regset == temp_set2
1286 105289211 : && (cl3 == GENERAL_REGS
1287 104605007 : || ((ira_reg_class_intersect[cl1][cl2]
1288 : != GENERAL_REGS)
1289 35029601 : && hard_reg_set_subset_p
1290 35029601 : (reg_class_contents[cl3],
1291 : reg_class_contents
1292 : [(int)
1293 : ira_reg_class_intersect[cl1][cl2]])))))
1294 593239320 : ira_reg_class_intersect[cl1][cl2] = (enum reg_class) cl3;
1295 : }
1296 675311807 : temp_set2
1297 675311807 : = (reg_class_contents[ira_reg_class_subset[cl1][cl2]]
1298 675311807 : & ~no_unit_alloc_regs);
1299 675311807 : if (! hard_reg_set_subset_p (temp_hard_regset, temp_set2)
1300 : /* Ignore unavailable hard registers and prefer
1301 : smallest class for debugging purposes. */
1302 784951023 : || (temp_hard_regset == temp_set2
1303 105289211 : && hard_reg_set_subset_p
1304 105289211 : (reg_class_contents[cl3],
1305 : reg_class_contents
1306 : [(int) ira_reg_class_subset[cl1][cl2]])))
1307 627616763 : ira_reg_class_subset[cl1][cl2] = (enum reg_class) cl3;
1308 : }
1309 7902955853 : if (important_class_p[cl3]
1310 15805911706 : && hard_reg_set_subset_p (temp_hard_regset, union_set))
1311 : {
1312 : /* CL3 allocatable hard register set is inside of
1313 : union of allocatable hard register sets of CL1
1314 : and CL2. */
1315 3517557061 : temp_set2
1316 3517557061 : = (reg_class_contents[ira_reg_class_subunion[cl1][cl2]]
1317 3517557061 : & ~no_unit_alloc_regs);
1318 3517557061 : if (ira_reg_class_subunion[cl1][cl2] == NO_REGS
1319 6780351687 : || (hard_reg_set_subset_p (temp_set2, temp_hard_regset)
1320 :
1321 1085638475 : && (temp_set2 != temp_hard_regset
1322 451903460 : || cl3 == GENERAL_REGS
1323 : /* If the allocatable hard register sets are the
1324 : same, prefer GENERAL_REGS or the smallest
1325 : class for debugging purposes. */
1326 448301418 : || (ira_reg_class_subunion[cl1][cl2] != GENERAL_REGS
1327 29712142 : && hard_reg_set_subset_p
1328 29712142 : (reg_class_contents[cl3],
1329 : reg_class_contents
1330 : [(int) ira_reg_class_subunion[cl1][cl2]])))))
1331 915204101 : ira_reg_class_subunion[cl1][cl2] = (enum reg_class) cl3;
1332 : }
1333 15805911706 : if (hard_reg_set_subset_p (union_set, temp_hard_regset))
1334 : {
1335 : /* CL3 allocatable hard register set contains union
1336 : of allocatable hard register sets of CL1 and
1337 : CL2. */
1338 1465525699 : temp_set2
1339 1465525699 : = (reg_class_contents[ira_reg_class_superunion[cl1][cl2]]
1340 1465525699 : & ~no_unit_alloc_regs);
1341 1465525699 : if (ira_reg_class_superunion[cl1][cl2] == NO_REGS
1342 2676288963 : || (hard_reg_set_subset_p (temp_hard_regset, temp_set2)
1343 :
1344 207832420 : && (temp_set2 != temp_hard_regset
1345 206925462 : || cl3 == GENERAL_REGS
1346 : /* If the allocatable hard register sets are the
1347 : same, prefer GENERAL_REGS or the smallest
1348 : class for debugging purposes. */
1349 205360300 : || (ira_reg_class_superunion[cl1][cl2] != GENERAL_REGS
1350 20466346 : && hard_reg_set_subset_p
1351 20466346 : (reg_class_contents[cl3],
1352 : reg_class_contents
1353 : [(int) ira_reg_class_superunion[cl1][cl2]])))))
1354 271936732 : ira_reg_class_superunion[cl1][cl2] = (enum reg_class) cl3;
1355 : }
1356 : }
1357 : }
1358 : }
1359 222149 : }
1360 :
1361 : /* Output all uniform and important classes into file F. */
1362 : static void
1363 0 : print_uniform_and_important_classes (FILE *f)
1364 : {
1365 0 : int i, cl;
1366 :
1367 0 : fprintf (f, "Uniform classes:\n");
1368 0 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1369 0 : if (ira_uniform_class_p[cl])
1370 0 : fprintf (f, " %s", reg_class_names[cl]);
1371 0 : fprintf (f, "\nImportant classes:\n");
1372 0 : for (i = 0; i < ira_important_classes_num; i++)
1373 0 : fprintf (f, " %s", reg_class_names[ira_important_classes[i]]);
1374 0 : fprintf (f, "\n");
1375 0 : }
1376 :
1377 : /* Output all possible allocno or pressure classes and their
1378 : translation map into file F. */
1379 : static void
1380 0 : print_translated_classes (FILE *f, bool pressure_p)
1381 : {
1382 0 : int classes_num = (pressure_p
1383 0 : ? ira_pressure_classes_num : ira_allocno_classes_num);
1384 0 : enum reg_class *classes = (pressure_p
1385 0 : ? ira_pressure_classes : ira_allocno_classes);
1386 0 : enum reg_class *class_translate = (pressure_p
1387 : ? ira_pressure_class_translate
1388 : : ira_allocno_class_translate);
1389 0 : int i;
1390 :
1391 0 : fprintf (f, "%s classes:\n", pressure_p ? "Pressure" : "Allocno");
1392 0 : for (i = 0; i < classes_num; i++)
1393 0 : fprintf (f, " %s", reg_class_names[classes[i]]);
1394 0 : fprintf (f, "\nClass translation:\n");
1395 0 : for (i = 0; i < N_REG_CLASSES; i++)
1396 0 : fprintf (f, " %s -> %s\n", reg_class_names[i],
1397 0 : reg_class_names[class_translate[i]]);
1398 0 : }
1399 :
1400 : /* Output all possible allocno and translation classes and the
1401 : translation maps into stderr. */
1402 : void
1403 0 : ira_debug_allocno_classes (void)
1404 : {
1405 0 : print_uniform_and_important_classes (stderr);
1406 0 : print_translated_classes (stderr, false);
1407 0 : print_translated_classes (stderr, true);
1408 0 : }
1409 :
1410 : /* Set up different arrays concerning class subsets, allocno and
1411 : important classes. */
1412 : static void
1413 222149 : find_reg_classes (void)
1414 : {
1415 222149 : setup_allocno_and_important_classes ();
1416 222149 : setup_class_translate ();
1417 222149 : reorder_important_classes ();
1418 222149 : setup_reg_class_relations ();
1419 222149 : }
1420 :
1421 :
1422 :
1423 : /* Set up array ira_hard_regno_allocno_class. */
1424 : static void
1425 222149 : setup_hard_regno_aclass (void)
1426 : {
1427 222149 : int i;
1428 :
1429 21104155 : for (i = 0; i < FIRST_PSEUDO_REGISTER; i++)
1430 : {
1431 41764012 : ira_hard_regno_allocno_class[i]
1432 31261272 : = (TEST_HARD_REG_BIT (no_unit_alloc_regs, i)
1433 20882006 : ? NO_REGS
1434 10379266 : : ira_allocno_class_translate[REGNO_REG_CLASS (i)]);
1435 : }
1436 222149 : }
1437 :
1438 :
1439 :
1440 : /* Form IRA_REG_CLASS_MAX_NREGS and IRA_REG_CLASS_MIN_NREGS maps. */
1441 : static void
1442 222149 : setup_reg_class_nregs (void)
1443 : {
1444 222149 : int i, cl, cl2, m;
1445 :
1446 27768625 : for (m = 0; m < MAX_MACHINE_MODE; m++)
1447 : {
1448 964126660 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1449 1873160368 : ira_reg_class_max_nregs[cl][m]
1450 1873160368 : = ira_reg_class_min_nregs[cl][m]
1451 936580184 : = targetm.class_max_nregs ((reg_class_t) cl, (machine_mode) m);
1452 964126660 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1453 6709741928 : for (i = 0;
1454 7646322112 : (cl2 = alloc_reg_class_subclasses[cl][i]) != LIM_REG_CLASSES;
1455 : i++)
1456 6709741928 : if (ira_reg_class_min_nregs[cl2][m]
1457 6709741928 : < ira_reg_class_min_nregs[cl][m])
1458 53866024 : ira_reg_class_min_nregs[cl][m] = ira_reg_class_min_nregs[cl2][m];
1459 : }
1460 222149 : }
1461 :
1462 :
1463 :
1464 : /* Set up IRA_PROHIBITED_CLASS_MODE_REGS, IRA_EXCLUDE_CLASS_MODE_REGS, and
1465 : IRA_CLASS_SINGLETON. This function is called once IRA_CLASS_HARD_REGS has
1466 : been initialized. */
1467 : static void
1468 222149 : setup_prohibited_and_exclude_class_mode_regs (void)
1469 : {
1470 222149 : int j, k, hard_regno, cl, last_hard_regno, count;
1471 :
1472 7775215 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
1473 : {
1474 7553066 : temp_hard_regset = reg_class_contents[cl] & ~no_unit_alloc_regs;
1475 944133250 : for (j = 0; j < NUM_MACHINE_MODES; j++)
1476 : {
1477 936580184 : count = 0;
1478 936580184 : last_hard_regno = -1;
1479 3746320736 : CLEAR_HARD_REG_SET (ira_prohibited_class_mode_regs[cl][j]);
1480 936580184 : CLEAR_HARD_REG_SET (ira_exclude_class_mode_regs[cl][j]);
1481 10431810992 : for (k = ira_class_hard_regs_num[cl] - 1; k >= 0; k--)
1482 : {
1483 9495230808 : hard_regno = ira_class_hard_regs[cl][k];
1484 9495230808 : if (!targetm.hard_regno_mode_ok (hard_regno, (machine_mode) j))
1485 7158084542 : SET_HARD_REG_BIT (ira_prohibited_class_mode_regs[cl][j],
1486 : hard_regno);
1487 2337146266 : else if (in_hard_reg_set_p (temp_hard_regset,
1488 : (machine_mode) j, hard_regno))
1489 : {
1490 2238338089 : last_hard_regno = hard_regno;
1491 2238338089 : count++;
1492 : }
1493 : else
1494 : {
1495 98808177 : SET_HARD_REG_BIT (ira_exclude_class_mode_regs[cl][j], hard_regno);
1496 : }
1497 : }
1498 985172061 : ira_class_singleton[cl][j] = (count == 1 ? last_hard_regno : -1);
1499 : }
1500 : }
1501 222149 : }
1502 :
1503 : /* Clarify IRA_PROHIBITED_CLASS_MODE_REGS by excluding hard registers
1504 : spanning from one register pressure class to another one. It is
1505 : called after defining the pressure classes. */
1506 : static void
1507 222149 : clarify_prohibited_class_mode_regs (void)
1508 : {
1509 222149 : int j, k, hard_regno, cl, pclass, nregs;
1510 :
1511 7775215 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
1512 944133250 : for (j = 0; j < NUM_MACHINE_MODES; j++)
1513 : {
1514 936580184 : CLEAR_HARD_REG_SET (ira_useful_class_mode_regs[cl][j]);
1515 10431810992 : for (k = ira_class_hard_regs_num[cl] - 1; k >= 0; k--)
1516 : {
1517 9495230808 : hard_regno = ira_class_hard_regs[cl][k];
1518 9495230808 : if (TEST_HARD_REG_BIT (ira_prohibited_class_mode_regs[cl][j], hard_regno))
1519 7158084542 : continue;
1520 2337146266 : nregs = hard_regno_nregs (hard_regno, (machine_mode) j);
1521 2337146266 : if (hard_regno + nregs > FIRST_PSEUDO_REGISTER)
1522 : {
1523 2256 : SET_HARD_REG_BIT (ira_prohibited_class_mode_regs[cl][j],
1524 : hard_regno);
1525 2256 : continue;
1526 : }
1527 2337144010 : pclass = ira_pressure_class_translate[REGNO_REG_CLASS (hard_regno)];
1528 5134728201 : for (nregs-- ;nregs >= 0; nregs--)
1529 2849566527 : if (((enum reg_class) pclass
1530 2849566527 : != ira_pressure_class_translate[REGNO_REG_CLASS
1531 2849566527 : (hard_regno + nregs)]))
1532 : {
1533 51982336 : SET_HARD_REG_BIT (ira_prohibited_class_mode_regs[cl][j],
1534 : hard_regno);
1535 51982336 : break;
1536 : }
1537 2337144010 : if (!TEST_HARD_REG_BIT (ira_prohibited_class_mode_regs[cl][j],
1538 : hard_regno))
1539 2285161674 : add_to_hard_reg_set (&ira_useful_class_mode_regs[cl][j],
1540 : (machine_mode) j, hard_regno);
1541 : }
1542 : }
1543 222149 : }
1544 :
1545 : /* Allocate and initialize IRA_REGISTER_MOVE_COST, IRA_MAY_MOVE_IN_COST
1546 : and IRA_MAY_MOVE_OUT_COST for MODE. */
1547 : void
1548 10362094 : ira_init_register_move_cost (machine_mode mode)
1549 : {
1550 10362094 : static unsigned short last_move_cost[N_REG_CLASSES][N_REG_CLASSES];
1551 10362094 : bool all_match = true;
1552 10362094 : unsigned int i, cl1, cl2;
1553 10362094 : HARD_REG_SET ok_regs;
1554 :
1555 10362094 : ira_assert (ira_register_move_cost[mode] == NULL
1556 : && ira_may_move_in_cost[mode] == NULL
1557 : && ira_may_move_out_cost[mode] == NULL);
1558 984398930 : CLEAR_HARD_REG_SET (ok_regs);
1559 984398930 : for (i = 0; i < FIRST_PSEUDO_REGISTER; i++)
1560 974036836 : if (targetm.hard_regno_mode_ok (i, mode))
1561 449848760 : SET_HARD_REG_BIT (ok_regs, i);
1562 :
1563 : /* Note that we might be asked about the move costs of modes that
1564 : cannot be stored in any hard register, for example if an inline
1565 : asm tries to create a register operand with an impossible mode.
1566 : We therefore can't assert have_regs_of_mode[mode] here. */
1567 362673290 : for (cl1 = 0; cl1 < N_REG_CLASSES; cl1++)
1568 12330891860 : for (cl2 = 0; cl2 < N_REG_CLASSES; cl2++)
1569 : {
1570 11978580664 : int cost;
1571 11978580664 : if (!hard_reg_set_intersect_p (ok_regs, reg_class_contents[cl1])
1572 19802313354 : || !hard_reg_set_intersect_p (ok_regs, reg_class_contents[cl2]))
1573 : {
1574 6257008441 : if ((ira_reg_class_max_nregs[cl1][mode]
1575 6257008441 : > ira_class_hard_regs_num[cl1])
1576 4440895325 : || (ira_reg_class_max_nregs[cl2][mode]
1577 4440895325 : > ira_class_hard_regs_num[cl2]))
1578 : cost = 65535;
1579 : else
1580 3055630005 : cost = (ira_memory_move_cost[mode][cl1][0]
1581 3055630005 : + ira_memory_move_cost[mode][cl2][1]) * 2;
1582 : }
1583 : else
1584 : {
1585 5721572223 : cost = register_move_cost (mode, (enum reg_class) cl1,
1586 : (enum reg_class) cl2);
1587 5721572223 : ira_assert (cost < 65535);
1588 : }
1589 11978580664 : all_match &= (last_move_cost[cl1][cl2] == cost);
1590 11978580664 : last_move_cost[cl1][cl2] = cost;
1591 : }
1592 10362094 : if (all_match && last_mode_for_init_move_cost != -1)
1593 : {
1594 4302625 : ira_register_move_cost[mode]
1595 4302625 : = ira_register_move_cost[last_mode_for_init_move_cost];
1596 4302625 : ira_may_move_in_cost[mode]
1597 4302625 : = ira_may_move_in_cost[last_mode_for_init_move_cost];
1598 4302625 : ira_may_move_out_cost[mode]
1599 4302625 : = ira_may_move_out_cost[last_mode_for_init_move_cost];
1600 4302625 : return;
1601 : }
1602 6059469 : last_mode_for_init_move_cost = mode;
1603 6059469 : ira_register_move_cost[mode] = XNEWVEC (move_table, N_REG_CLASSES);
1604 6059469 : ira_may_move_in_cost[mode] = XNEWVEC (move_table, N_REG_CLASSES);
1605 6059469 : ira_may_move_out_cost[mode] = XNEWVEC (move_table, N_REG_CLASSES);
1606 212081415 : for (cl1 = 0; cl1 < N_REG_CLASSES; cl1++)
1607 7210768110 : for (cl2 = 0; cl2 < N_REG_CLASSES; cl2++)
1608 : {
1609 7004746164 : int cost;
1610 7004746164 : enum reg_class *p1, *p2;
1611 :
1612 7004746164 : if (last_move_cost[cl1][cl2] == 65535)
1613 : {
1614 1748934004 : ira_register_move_cost[mode][cl1][cl2] = 65535;
1615 1748934004 : ira_may_move_in_cost[mode][cl1][cl2] = 65535;
1616 1748934004 : ira_may_move_out_cost[mode][cl1][cl2] = 65535;
1617 : }
1618 : else
1619 : {
1620 5255812160 : cost = last_move_cost[cl1][cl2];
1621 :
1622 44040728609 : for (p2 = ®_class_subclasses[cl2][0];
1623 44040728609 : *p2 != LIM_REG_CLASSES; p2++)
1624 38784916449 : if (ira_class_hard_regs_num[*p2] > 0
1625 38073592613 : && (ira_reg_class_max_nregs[*p2][mode]
1626 : <= ira_class_hard_regs_num[*p2]))
1627 31227009854 : cost = MAX (cost, ira_register_move_cost[mode][cl1][*p2]);
1628 :
1629 44040728609 : for (p1 = ®_class_subclasses[cl1][0];
1630 44040728609 : *p1 != LIM_REG_CLASSES; p1++)
1631 38784916449 : if (ira_class_hard_regs_num[*p1] > 0
1632 38073592613 : && (ira_reg_class_max_nregs[*p1][mode]
1633 : <= ira_class_hard_regs_num[*p1]))
1634 31227009854 : cost = MAX (cost, ira_register_move_cost[mode][*p1][cl2]);
1635 :
1636 5255812160 : ira_assert (cost <= 65535);
1637 5255812160 : ira_register_move_cost[mode][cl1][cl2] = cost;
1638 :
1639 5255812160 : if (ira_class_subset_p[cl1][cl2])
1640 1586008525 : ira_may_move_in_cost[mode][cl1][cl2] = 0;
1641 : else
1642 3669803635 : ira_may_move_in_cost[mode][cl1][cl2] = cost;
1643 :
1644 5255812160 : if (ira_class_subset_p[cl2][cl1])
1645 1586008525 : ira_may_move_out_cost[mode][cl1][cl2] = 0;
1646 : else
1647 3669803635 : ira_may_move_out_cost[mode][cl1][cl2] = cost;
1648 : }
1649 : }
1650 : }
1651 :
1652 :
1653 :
1654 : /* This is called once during compiler work. It sets up
1655 : different arrays whose values don't depend on the compiled
1656 : function. */
1657 : void
1658 217848 : ira_init_once (void)
1659 : {
1660 217848 : ira_init_costs_once ();
1661 217848 : lra_init_once ();
1662 :
1663 217848 : ira_use_lra_p = targetm.lra_p ();
1664 217848 : }
1665 :
1666 : /* Free ira_max_register_move_cost, ira_may_move_in_cost and
1667 : ira_may_move_out_cost for each mode. */
1668 : void
1669 547809 : target_ira_int::free_register_move_costs (void)
1670 : {
1671 547809 : int mode, i;
1672 :
1673 : /* Reset move_cost and friends, making sure we only free shared
1674 : table entries once. */
1675 68476125 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
1676 67928316 : if (x_ira_register_move_cost[mode])
1677 : {
1678 621919128 : for (i = 0;
1679 631474286 : i < mode && (x_ira_register_move_cost[i]
1680 : != x_ira_register_move_cost[mode]);
1681 : i++)
1682 : ;
1683 9555158 : if (i == mode)
1684 : {
1685 5590971 : free (x_ira_register_move_cost[mode]);
1686 5590971 : free (x_ira_may_move_in_cost[mode]);
1687 5590971 : free (x_ira_may_move_out_cost[mode]);
1688 : }
1689 : }
1690 547809 : memset (x_ira_register_move_cost, 0, sizeof x_ira_register_move_cost);
1691 547809 : memset (x_ira_may_move_in_cost, 0, sizeof x_ira_may_move_in_cost);
1692 547809 : memset (x_ira_may_move_out_cost, 0, sizeof x_ira_may_move_out_cost);
1693 547809 : last_mode_for_init_move_cost = -1;
1694 547809 : }
1695 :
1696 325660 : target_ira_int::~target_ira_int ()
1697 : {
1698 325660 : free_ira_costs ();
1699 325660 : free_register_move_costs ();
1700 325660 : }
1701 :
1702 : /* This is called every time when register related information is
1703 : changed. */
1704 : void
1705 222149 : ira_init (void)
1706 : {
1707 222149 : this_target_ira_int->free_register_move_costs ();
1708 222149 : setup_reg_mode_hard_regset ();
1709 222149 : setup_alloc_regs (flag_omit_frame_pointer != 0);
1710 222149 : setup_class_subset_and_memory_move_costs ();
1711 222149 : setup_reg_class_nregs ();
1712 222149 : setup_prohibited_and_exclude_class_mode_regs ();
1713 222149 : find_reg_classes ();
1714 222149 : clarify_prohibited_class_mode_regs ();
1715 222149 : setup_hard_regno_aclass ();
1716 222149 : ira_init_costs ();
1717 222149 : }
1718 :
1719 :
1720 : #define ira_prohibited_mode_move_regs_initialized_p \
1721 : (this_target_ira_int->x_ira_prohibited_mode_move_regs_initialized_p)
1722 :
1723 : /* Set up IRA_PROHIBITED_MODE_MOVE_REGS. */
1724 : static void
1725 1517127 : setup_prohibited_mode_move_regs (void)
1726 : {
1727 1517127 : int i, j;
1728 1517127 : rtx test_reg1, test_reg2, move_pat;
1729 1517127 : rtx_insn *move_insn;
1730 :
1731 1517127 : if (ira_prohibited_mode_move_regs_initialized_p)
1732 : return;
1733 220872 : ira_prohibited_mode_move_regs_initialized_p = true;
1734 220872 : test_reg1 = gen_rtx_REG (word_mode, LAST_VIRTUAL_REGISTER + 1);
1735 220872 : test_reg2 = gen_rtx_REG (word_mode, LAST_VIRTUAL_REGISTER + 2);
1736 220872 : move_pat = gen_rtx_SET (test_reg1, test_reg2);
1737 220872 : move_insn = gen_rtx_INSN (VOIDmode, 0, 0, 0, move_pat, 0, -1, 0);
1738 27829872 : for (i = 0; i < NUM_MACHINE_MODES; i++)
1739 : {
1740 27388128 : SET_HARD_REG_SET (ira_prohibited_mode_move_regs[i]);
1741 2601872160 : for (j = 0; j < FIRST_PSEUDO_REGISTER; j++)
1742 : {
1743 2574484032 : if (!targetm.hard_regno_mode_ok (j, (machine_mode) i))
1744 2122745731 : continue;
1745 451738301 : set_mode_and_regno (test_reg1, (machine_mode) i, j);
1746 451738301 : set_mode_and_regno (test_reg2, (machine_mode) i, j);
1747 451738301 : INSN_CODE (move_insn) = -1;
1748 451738301 : recog_memoized (move_insn);
1749 451738301 : if (INSN_CODE (move_insn) < 0)
1750 222239372 : continue;
1751 229498929 : extract_insn (move_insn);
1752 : /* We don't know whether the move will be in code that is optimized
1753 : for size or speed, so consider all enabled alternatives. */
1754 229498929 : if (! constrain_operands (1, get_enabled_alternatives (move_insn)))
1755 1350111 : continue;
1756 228148818 : CLEAR_HARD_REG_BIT (ira_prohibited_mode_move_regs[i], j);
1757 : }
1758 : }
1759 : }
1760 :
1761 :
1762 :
1763 : /* Extract INSN and return the set of alternatives that we should consider.
1764 : This excludes any alternatives whose constraints are obviously impossible
1765 : to meet (e.g. because the constraint requires a constant and the operand
1766 : is nonconstant). It also excludes alternatives that are bound to need
1767 : a spill or reload, as long as we have other alternatives that match
1768 : exactly. */
1769 : alternative_mask
1770 106118646 : ira_setup_alts (rtx_insn *insn)
1771 : {
1772 106118646 : int nop, nalt;
1773 106118646 : bool curr_swapped;
1774 106118646 : const char *p;
1775 106118646 : int commutative = -1;
1776 :
1777 106118646 : extract_insn (insn);
1778 106118646 : preprocess_constraints (insn);
1779 106118646 : alternative_mask preferred = get_preferred_alternatives (insn);
1780 106118646 : alternative_mask alts = 0;
1781 106118646 : alternative_mask exact_alts = 0;
1782 : /* Check that the hard reg set is enough for holding all
1783 : alternatives. It is hard to imagine the situation when the
1784 : assertion is wrong. */
1785 106118646 : ira_assert (recog_data.n_alternatives
1786 : <= (int) MAX (sizeof (HARD_REG_ELT_TYPE) * CHAR_BIT,
1787 : FIRST_PSEUDO_REGISTER));
1788 312047328 : for (nop = 0; nop < recog_data.n_operands; nop++)
1789 218307206 : if (recog_data.constraints[nop][0] == '%')
1790 : {
1791 : commutative = nop;
1792 : break;
1793 : }
1794 106118646 : for (curr_swapped = false;; curr_swapped = true)
1795 : {
1796 1431594529 : for (nalt = 0; nalt < recog_data.n_alternatives; nalt++)
1797 : {
1798 1313097359 : if (!TEST_BIT (preferred, nalt) || TEST_BIT (exact_alts, nalt))
1799 454461341 : continue;
1800 :
1801 858636018 : const operand_alternative *op_alt
1802 858636018 : = &recog_op_alt[nalt * recog_data.n_operands];
1803 858636018 : int this_reject = 0;
1804 2454300012 : for (nop = 0; nop < recog_data.n_operands; nop++)
1805 : {
1806 1790168694 : int c, len;
1807 :
1808 1790168694 : this_reject += op_alt[nop].reject;
1809 :
1810 1790168694 : rtx op = recog_data.operand[nop];
1811 1790168694 : p = op_alt[nop].constraint;
1812 1790168694 : if (*p == 0 || *p == ',')
1813 25182385 : continue;
1814 :
1815 : bool win_p = false;
1816 3581416306 : do
1817 3581416306 : switch (c = *p, len = CONSTRAINT_LEN (c, p), c)
1818 : {
1819 : case '#':
1820 : case ',':
1821 : c = '\0';
1822 : /* FALLTHRU */
1823 751741770 : case '\0':
1824 751741770 : len = 0;
1825 751741770 : break;
1826 :
1827 : case '%':
1828 : /* The commutative modifier is handled above. */
1829 : break;
1830 :
1831 75995299 : case '0': case '1': case '2': case '3': case '4':
1832 75995299 : case '5': case '6': case '7': case '8': case '9':
1833 75995299 : {
1834 75995299 : char *end;
1835 75995299 : unsigned long dup = strtoul (p, &end, 10);
1836 75995299 : rtx other = recog_data.operand[dup];
1837 75995299 : len = end - p;
1838 1891832 : if (MEM_P (other)
1839 75995299 : ? rtx_equal_p (other, op)
1840 74103467 : : REG_P (op) || SUBREG_P (op))
1841 52971150 : goto op_success;
1842 23024149 : win_p = true;
1843 : }
1844 23024149 : break;
1845 :
1846 10874955 : case 'g':
1847 10874955 : goto op_success;
1848 145 : break;
1849 :
1850 145 : case '{':
1851 145 : if (REG_P (op) || SUBREG_P (op))
1852 143 : goto op_success;
1853 : win_p = true;
1854 : break;
1855 :
1856 2723344191 : default:
1857 2723344191 : {
1858 2723344191 : enum constraint_num cn = lookup_constraint (p);
1859 2723344191 : rtx mem = NULL;
1860 2723344191 : switch (get_constraint_type (cn))
1861 : {
1862 2221075458 : case CT_REGISTER:
1863 3497065423 : if (reg_class_for_constraint (cn) != NO_REGS)
1864 : {
1865 1230091377 : if (REG_P (op) || SUBREG_P (op))
1866 803714551 : goto op_success;
1867 : win_p = true;
1868 : }
1869 : break;
1870 :
1871 4217114 : case CT_CONST_INT:
1872 4217114 : if (CONST_INT_P (op)
1873 6911403 : && (insn_const_int_ok_for_constraint
1874 2694289 : (INTVAL (op), cn)))
1875 1954567 : goto op_success;
1876 : break;
1877 :
1878 812186 : case CT_ADDRESS:
1879 812186 : goto op_success;
1880 :
1881 167901301 : case CT_MEMORY:
1882 167901301 : case CT_RELAXED_MEMORY:
1883 167901301 : mem = op;
1884 : /* Fall through. */
1885 167901301 : case CT_SPECIAL_MEMORY:
1886 167901301 : if (!mem)
1887 69216116 : mem = extract_mem_from_operand (op);
1888 237117417 : if (MEM_P (mem))
1889 71383380 : goto op_success;
1890 : win_p = true;
1891 : break;
1892 :
1893 260122016 : case CT_FIXED_FORM:
1894 260122016 : if (constraint_satisfied_p (op, cn))
1895 71533607 : goto op_success;
1896 : break;
1897 : }
1898 : break;
1899 : }
1900 : }
1901 2568171767 : while (p += len, c);
1902 751741770 : if (!win_p)
1903 : break;
1904 : /* We can make the alternative match by spilling a register
1905 : to memory or loading something into a register. Count a
1906 : cost of one reload (the equivalent of the '?' constraint). */
1907 557237070 : this_reject += 6;
1908 1595663994 : op_success:
1909 1595663994 : ;
1910 : }
1911 :
1912 858636018 : if (nop >= recog_data.n_operands)
1913 : {
1914 664131318 : alts |= ALTERNATIVE_BIT (nalt);
1915 664131318 : if (this_reject == 0)
1916 139585732 : exact_alts |= ALTERNATIVE_BIT (nalt);
1917 : }
1918 : }
1919 118497170 : if (commutative < 0)
1920 : break;
1921 : /* Swap forth and back to avoid changing recog_data. */
1922 24757048 : std::swap (recog_data.operand[commutative],
1923 24757048 : recog_data.operand[commutative + 1]);
1924 24757048 : if (curr_swapped)
1925 : break;
1926 : }
1927 106118646 : return exact_alts ? exact_alts : alts;
1928 : }
1929 :
1930 : /* Return the number of the output non-early clobber operand which
1931 : should be the same in any case as operand with number OP_NUM (or
1932 : negative value if there is no such operand). ALTS is the mask
1933 : of alternatives that we should consider. SINGLE_INPUT_OP_HAS_CSTR_P
1934 : should be set in this function, it indicates whether there is only
1935 : a single input operand which has the matching constraint on the
1936 : output operand at the position specified in return value. If the
1937 : pattern allows any one of several input operands holds the matching
1938 : constraint, it's set as false, one typical case is destructive FMA
1939 : instruction on target rs6000. Note that for a non-NO_REG preferred
1940 : register class with no free register move copy, if the parameter
1941 : PARAM_IRA_CONSIDER_DUP_IN_ALL_ALTS is set to one, this function
1942 : will check all available alternatives for matching constraints,
1943 : even if it has found or will find one alternative with non-NO_REG
1944 : regclass, it can respect more cases with matching constraints. If
1945 : PARAM_IRA_CONSIDER_DUP_IN_ALL_ALTS is set to zero,
1946 : SINGLE_INPUT_OP_HAS_CSTR_P is always true, it will stop to find
1947 : matching constraint relationship once it hits some alternative with
1948 : some non-NO_REG regclass. */
1949 : int
1950 21183263 : ira_get_dup_out_num (int op_num, alternative_mask alts,
1951 : bool &single_input_op_has_cstr_p)
1952 : {
1953 21183263 : int curr_alt, c, original;
1954 21183263 : bool ignore_p, use_commut_op_p;
1955 21183263 : const char *str;
1956 :
1957 21183263 : if (op_num < 0 || recog_data.n_alternatives == 0)
1958 : return -1;
1959 : /* We should find duplications only for input operands. */
1960 21183263 : if (recog_data.operand_type[op_num] != OP_IN)
1961 : return -1;
1962 15273183 : str = recog_data.constraints[op_num];
1963 15273183 : use_commut_op_p = false;
1964 15273183 : single_input_op_has_cstr_p = true;
1965 :
1966 15273183 : rtx op = recog_data.operand[op_num];
1967 15273183 : int op_regno = reg_or_subregno (op);
1968 15273183 : enum reg_class op_pref_cl = reg_preferred_class (op_regno);
1969 15273183 : machine_mode op_mode = GET_MODE (op);
1970 :
1971 15273183 : ira_init_register_move_cost_if_necessary (op_mode);
1972 : /* If the preferred regclass isn't NO_REG, continue to find the matching
1973 : constraint in all available alternatives with preferred regclass, even
1974 : if we have found or will find one alternative whose constraint stands
1975 : for a REG (non-NO_REG) regclass. Note that it would be fine not to
1976 : respect matching constraint if the register copy is free, so exclude
1977 : it. */
1978 15273183 : bool respect_dup_despite_reg_cstr
1979 15273183 : = param_ira_consider_dup_in_all_alts
1980 479 : && op_pref_cl != NO_REGS
1981 15273658 : && ira_register_move_cost[op_mode][op_pref_cl][op_pref_cl] > 0;
1982 :
1983 : /* Record the alternative whose constraint uses the same regclass as the
1984 : preferred regclass, later if we find one matching constraint for this
1985 : operand with preferred reclass, we will visit these recorded
1986 : alternatives to check whether if there is one alternative in which no
1987 : any INPUT operands have one matching constraint same as our candidate.
1988 : If yes, it means there is one alternative which is perfectly fine
1989 : without satisfying this matching constraint. If no, it means in any
1990 : alternatives there is one other INPUT operand holding this matching
1991 : constraint, it's fine to respect this matching constraint and further
1992 : create this constraint copy since it would become harmless once some
1993 : other takes preference and it's interfered. */
1994 17815463 : alternative_mask pref_cl_alts;
1995 :
1996 17815463 : for (;;)
1997 : {
1998 17815463 : pref_cl_alts = 0;
1999 :
2000 17815463 : for (curr_alt = 0, ignore_p = !TEST_BIT (alts, curr_alt),
2001 17815463 : original = -1;;)
2002 : {
2003 105553572 : c = *str;
2004 105553572 : if (c == '\0')
2005 : break;
2006 100517719 : if (c == '#')
2007 : ignore_p = true;
2008 100517719 : else if (c == ',')
2009 : {
2010 34001558 : curr_alt++;
2011 34001558 : ignore_p = !TEST_BIT (alts, curr_alt);
2012 : }
2013 66516161 : else if (! ignore_p)
2014 21386446 : switch (c)
2015 : {
2016 869 : case 'g':
2017 869 : goto fail;
2018 16097341 : default:
2019 16097341 : {
2020 16097341 : enum constraint_num cn = lookup_constraint (str);
2021 16097341 : enum reg_class cl = reg_class_for_constraint (cn);
2022 13074269 : if (cl != NO_REGS && !targetm.class_likely_spilled_p (cl))
2023 : {
2024 12777282 : if (respect_dup_despite_reg_cstr)
2025 : {
2026 : /* If it's free to move from one preferred class to
2027 : the one without matching constraint, it doesn't
2028 : have to respect this constraint with costs. */
2029 694 : if (cl != op_pref_cl
2030 117 : && (ira_reg_class_intersect[cl][op_pref_cl]
2031 : != NO_REGS)
2032 105 : && (ira_may_move_in_cost[op_mode][op_pref_cl][cl]
2033 : == 0))
2034 89 : goto fail;
2035 605 : else if (cl == op_pref_cl)
2036 577 : pref_cl_alts |= ALTERNATIVE_BIT (curr_alt);
2037 : }
2038 : else
2039 12776588 : goto fail;
2040 : }
2041 3320664 : if (constraint_satisfied_p (op, cn))
2042 2064 : goto fail;
2043 : break;
2044 : }
2045 :
2046 5288236 : case '0': case '1': case '2': case '3': case '4':
2047 5288236 : case '5': case '6': case '7': case '8': case '9':
2048 5288236 : {
2049 5288236 : char *end;
2050 5288236 : int n = (int) strtoul (str, &end, 10);
2051 5288236 : str = end;
2052 5288236 : if (original != -1 && original != n)
2053 0 : goto fail;
2054 5288236 : gcc_assert (n < recog_data.n_operands);
2055 5288236 : if (respect_dup_despite_reg_cstr)
2056 : {
2057 235 : const operand_alternative *op_alt
2058 235 : = &recog_op_alt[curr_alt * recog_data.n_operands];
2059 : /* Only respect the one with preferred rclass, without
2060 : respect_dup_despite_reg_cstr it's possible to get
2061 : one whose regclass isn't preferred first before,
2062 : but it would fail since there should be other
2063 : alternatives with preferred regclass. */
2064 235 : if (op_alt[n].cl == op_pref_cl)
2065 5288167 : original = n;
2066 : }
2067 : else
2068 : original = n;
2069 5288236 : continue;
2070 5288236 : }
2071 : }
2072 82449873 : str += CONSTRAINT_LEN (c, str);
2073 : }
2074 5035853 : if (original == -1)
2075 1922954 : goto fail;
2076 3112899 : if (recog_data.operand_type[original] == OP_OUT)
2077 : {
2078 3112580 : if (pref_cl_alts == 0)
2079 : return original;
2080 : /* Visit these recorded alternatives to check whether
2081 : there is one alternative in which no any INPUT operands
2082 : have one matching constraint same as our candidate.
2083 : Give up this candidate if so. */
2084 : int nop, nalt;
2085 382 : for (nalt = 0; nalt < recog_data.n_alternatives; nalt++)
2086 : {
2087 359 : if (!TEST_BIT (pref_cl_alts, nalt))
2088 256 : continue;
2089 103 : const operand_alternative *op_alt
2090 103 : = &recog_op_alt[nalt * recog_data.n_operands];
2091 103 : bool dup_in_other = false;
2092 379 : for (nop = 0; nop < recog_data.n_operands; nop++)
2093 : {
2094 321 : if (recog_data.operand_type[nop] != OP_IN)
2095 103 : continue;
2096 218 : if (nop == op_num)
2097 92 : continue;
2098 126 : if (op_alt[nop].matches == original)
2099 : {
2100 : dup_in_other = true;
2101 : break;
2102 : }
2103 : }
2104 103 : if (!dup_in_other)
2105 : return -1;
2106 : }
2107 23 : single_input_op_has_cstr_p = false;
2108 23 : return original;
2109 : }
2110 319 : fail:
2111 14702883 : if (use_commut_op_p)
2112 : break;
2113 12968943 : use_commut_op_p = true;
2114 12968943 : if (recog_data.constraints[op_num][0] == '%')
2115 1119633 : str = recog_data.constraints[op_num + 1];
2116 11849310 : else if (op_num > 0 && recog_data.constraints[op_num - 1][0] == '%')
2117 : str = recog_data.constraints[op_num - 1];
2118 : else
2119 : break;
2120 : }
2121 : return -1;
2122 : }
2123 :
2124 :
2125 :
2126 : /* Return true if a replacement of SRC by DEST does not lead to unsatisfiable
2127 : asm. A replacement is valid if SRC or DEST are not constrained in asm
2128 : inputs of a single asm statement. See match_asm_constraints_2() for more
2129 : details. TODO: As in match_asm_constraints_2() consider alternatives more
2130 : precisely. */
2131 :
2132 : static bool
2133 8003 : valid_replacement_for_asm_input_p_1 (const_rtx asmops, const_rtx src, const_rtx dest)
2134 : {
2135 8003 : int ninputs = ASM_OPERANDS_INPUT_LENGTH (asmops);
2136 8003 : rtvec inputs = ASM_OPERANDS_INPUT_VEC (asmops);
2137 41492 : for (int i = 0; i < ninputs; ++i)
2138 : {
2139 33489 : rtx input_src = RTVEC_ELT (inputs, i);
2140 33489 : const char *constraint_src
2141 33489 : = ASM_OPERANDS_INPUT_CONSTRAINT (asmops, i);
2142 33489 : if (rtx_equal_p (input_src, src)
2143 33489 : && strchr (constraint_src, '{') != nullptr)
2144 0 : for (int j = 0; j < ninputs; ++j)
2145 : {
2146 0 : rtx input_dest = RTVEC_ELT (inputs, j);
2147 0 : const char *constraint_dest
2148 0 : = ASM_OPERANDS_INPUT_CONSTRAINT (asmops, j);
2149 0 : if (rtx_equal_p (input_dest, dest)
2150 0 : && strchr (constraint_dest, '{') != nullptr)
2151 : return false;
2152 : }
2153 : }
2154 : return true;
2155 : }
2156 :
2157 : /* Return true if a replacement of SRC by DEST does not lead to unsatisfiable
2158 : asm. A replacement is valid if SRC or DEST are not constrained in asm
2159 : inputs of a single asm statement. The final check is done in function
2160 : valid_replacement_for_asm_input_p_1. */
2161 :
2162 : static bool
2163 562400 : valid_replacement_for_asm_input_p (const_rtx src, const_rtx dest)
2164 : {
2165 : /* Bail out early if there is no asm statement. */
2166 562400 : if (!crtl->has_asm_statement)
2167 : return true;
2168 28216 : for (df_ref use = DF_REG_USE_CHAIN (REGNO (src));
2169 795892 : use;
2170 767676 : use = DF_REF_NEXT_REG (use))
2171 : {
2172 767676 : struct df_insn_info *use_info = DF_REF_INSN_INFO (use);
2173 : /* Only check real uses, not artificial ones. */
2174 767676 : if (use_info)
2175 : {
2176 767676 : rtx_insn *insn = DF_REF_INSN (use);
2177 767676 : rtx pat = PATTERN (insn);
2178 767676 : if (asm_noperands (pat) <= 0)
2179 764269 : continue;
2180 3407 : if (GET_CODE (pat) == SET)
2181 : {
2182 0 : if (!valid_replacement_for_asm_input_p_1 (SET_SRC (pat), src, dest))
2183 : return false;
2184 : }
2185 3407 : else if (GET_CODE (pat) == PARALLEL)
2186 14871 : for (int i = 0, len = XVECLEN (pat, 0); i < len; ++i)
2187 : {
2188 11464 : rtx asmops = XVECEXP (pat, 0, i);
2189 11464 : if (GET_CODE (asmops) == SET)
2190 7964 : asmops = SET_SRC (asmops);
2191 11464 : if (GET_CODE (asmops) == ASM_OPERANDS
2192 11464 : && !valid_replacement_for_asm_input_p_1 (asmops, src, dest))
2193 : return false;
2194 : }
2195 0 : else if (GET_CODE (pat) == ASM_OPERANDS)
2196 : {
2197 0 : if (!valid_replacement_for_asm_input_p_1 (pat, src, dest))
2198 : return false;
2199 : }
2200 : else
2201 0 : gcc_unreachable ();
2202 : }
2203 : }
2204 : return true;
2205 : }
2206 :
2207 : /* Search forward to see if the source register of a copy insn dies
2208 : before either it or the destination register is modified, but don't
2209 : scan past the end of the basic block. If so, we can replace the
2210 : source with the destination and let the source die in the copy
2211 : insn.
2212 :
2213 : This will reduce the number of registers live in that range and may
2214 : enable the destination and the source coalescing, thus often saving
2215 : one register in addition to a register-register copy. */
2216 :
2217 : static void
2218 1517127 : decrease_live_ranges_number (void)
2219 : {
2220 1517127 : basic_block bb;
2221 1517127 : rtx_insn *insn;
2222 1517127 : rtx set, src, dest, dest_death, note;
2223 1517127 : rtx_insn *p, *q;
2224 1517127 : int sregno, dregno;
2225 :
2226 1517127 : if (! flag_expensive_optimizations)
2227 : return;
2228 :
2229 982698 : if (ira_dump_file)
2230 32 : fprintf (ira_dump_file, "Starting decreasing number of live ranges...\n");
2231 :
2232 11594818 : FOR_EACH_BB_FN (bb, cfun)
2233 140056939 : FOR_BB_INSNS (bb, insn)
2234 : {
2235 129444819 : set = single_set (insn);
2236 129444819 : if (! set)
2237 76570724 : continue;
2238 52874095 : src = SET_SRC (set);
2239 52874095 : dest = SET_DEST (set);
2240 14528774 : if (! REG_P (src) || ! REG_P (dest)
2241 62046832 : || find_reg_note (insn, REG_DEAD, src))
2242 49906396 : continue;
2243 2967699 : sregno = REGNO (src);
2244 2967699 : dregno = REGNO (dest);
2245 :
2246 : /* We don't want to mess with hard regs if register classes
2247 : are small. */
2248 5372998 : if (sregno == dregno
2249 2967665 : || (targetm.small_register_classes_for_mode_p (GET_MODE (src))
2250 2967665 : && (sregno < FIRST_PSEUDO_REGISTER
2251 2967665 : || dregno < FIRST_PSEUDO_REGISTER))
2252 : /* We don't see all updates to SP if they are in an
2253 : auto-inc memory reference, so we must disallow this
2254 : optimization on them. */
2255 562400 : || sregno == STACK_POINTER_REGNUM
2256 562400 : || dregno == STACK_POINTER_REGNUM
2257 3530099 : || !valid_replacement_for_asm_input_p (src, dest))
2258 2405299 : continue;
2259 :
2260 562400 : dest_death = NULL_RTX;
2261 :
2262 7066533 : for (p = NEXT_INSN (insn); p; p = NEXT_INSN (p))
2263 : {
2264 6498914 : if (! INSN_P (p))
2265 1187844 : continue;
2266 5311070 : if (BLOCK_FOR_INSN (p) != bb)
2267 : break;
2268 :
2269 9609212 : if (reg_set_p (src, p) || reg_set_p (dest, p)
2270 : /* If SRC is an asm-declared register, it must not be
2271 : replaced in any asm. Unfortunately, the REG_EXPR
2272 : tree for the asm variable may be absent in the SRC
2273 : rtx, so we can't check the actual register
2274 : declaration easily (the asm operand will have it,
2275 : though). To avoid complicating the test for a rare
2276 : case, we just don't perform register replacement
2277 : for a hard reg mentioned in an asm. */
2278 4761460 : || (sregno < FIRST_PSEUDO_REGISTER
2279 0 : && asm_noperands (PATTERN (p)) >= 0
2280 0 : && reg_overlap_mentioned_p (src, PATTERN (p)))
2281 : /* Don't change hard registers used by a call. */
2282 4761460 : || (CALL_P (p) && sregno < FIRST_PSEUDO_REGISTER
2283 0 : && find_reg_fusage (p, USE, src))
2284 : /* Don't change a USE of a register. */
2285 9580587 : || (GET_CODE (PATTERN (p)) == USE
2286 1536 : && reg_overlap_mentioned_p (src, XEXP (PATTERN (p), 0))))
2287 : break;
2288 :
2289 : /* See if all of SRC dies in P. This test is slightly
2290 : more conservative than it needs to be. */
2291 4761460 : if ((note = find_regno_note (p, REG_DEAD, sregno))
2292 4761460 : && GET_MODE (XEXP (note, 0)) == GET_MODE (src))
2293 : {
2294 7571 : int failed = 0;
2295 :
2296 : /* We can do the optimization. Scan forward from INSN
2297 : again, replacing regs as we go. Set FAILED if a
2298 : replacement can't be done. In that case, we can't
2299 : move the death note for SRC. This should be
2300 : rare. */
2301 :
2302 : /* Set to stop at next insn. */
2303 7571 : for (q = next_real_insn (insn);
2304 70290 : q != next_real_insn (p);
2305 62719 : q = next_real_insn (q))
2306 : {
2307 62719 : if (reg_overlap_mentioned_p (src, PATTERN (q)))
2308 : {
2309 : /* If SRC is a hard register, we might miss
2310 : some overlapping registers with
2311 : validate_replace_rtx, so we would have to
2312 : undo it. We can't if DEST is present in
2313 : the insn, so fail in that combination of
2314 : cases. */
2315 8857 : if (sregno < FIRST_PSEUDO_REGISTER
2316 8857 : && reg_mentioned_p (dest, PATTERN (q)))
2317 : failed = 1;
2318 :
2319 : /* Attempt to replace all uses. */
2320 8857 : else if (!validate_replace_rtx (src, dest, q))
2321 : failed = 1;
2322 :
2323 : /* If this succeeded, but some part of the
2324 : register is still present, undo the
2325 : replacement. */
2326 8857 : else if (sregno < FIRST_PSEUDO_REGISTER
2327 8857 : && reg_overlap_mentioned_p (src, PATTERN (q)))
2328 : {
2329 0 : validate_replace_rtx (dest, src, q);
2330 0 : failed = 1;
2331 : }
2332 : }
2333 :
2334 : /* If DEST dies here, remove the death note and
2335 : save it for later. Make sure ALL of DEST dies
2336 : here; again, this is overly conservative. */
2337 62719 : if (! dest_death
2338 62719 : && (dest_death = find_regno_note (q, REG_DEAD, dregno)))
2339 : {
2340 2 : if (GET_MODE (XEXP (dest_death, 0)) == GET_MODE (dest))
2341 2 : remove_note (q, dest_death);
2342 : else
2343 : {
2344 : failed = 1;
2345 : dest_death = 0;
2346 : }
2347 : }
2348 : }
2349 :
2350 7571 : if (! failed)
2351 : {
2352 : /* Move death note of SRC from P to INSN. */
2353 7571 : remove_note (p, note);
2354 7571 : XEXP (note, 1) = REG_NOTES (insn);
2355 7571 : REG_NOTES (insn) = note;
2356 : }
2357 :
2358 : /* DEST is also dead if INSN has a REG_UNUSED note for
2359 : DEST. */
2360 7571 : if (! dest_death
2361 7571 : && (dest_death
2362 7569 : = find_regno_note (insn, REG_UNUSED, dregno)))
2363 : {
2364 0 : PUT_REG_NOTE_KIND (dest_death, REG_DEAD);
2365 0 : remove_note (insn, dest_death);
2366 : }
2367 :
2368 : /* Put death note of DEST on P if we saw it die. */
2369 7571 : if (dest_death)
2370 : {
2371 2 : XEXP (dest_death, 1) = REG_NOTES (p);
2372 2 : REG_NOTES (p) = dest_death;
2373 : }
2374 : break;
2375 : }
2376 :
2377 : /* If SRC is a hard register which is set or killed in
2378 : some other way, we can't do this optimization. */
2379 4753889 : else if (sregno < FIRST_PSEUDO_REGISTER && dead_or_set_p (p, src))
2380 : break;
2381 : }
2382 : }
2383 : }
2384 :
2385 :
2386 :
2387 : /* Return nonzero if REGNO is a particularly bad choice for reloading X. */
2388 : static bool
2389 0 : ira_bad_reload_regno_1 (int regno, rtx x)
2390 : {
2391 0 : int x_regno, n, i;
2392 0 : ira_allocno_t a;
2393 0 : enum reg_class pref;
2394 :
2395 : /* We only deal with pseudo regs. */
2396 0 : if (! x || GET_CODE (x) != REG)
2397 : return false;
2398 :
2399 0 : x_regno = REGNO (x);
2400 0 : if (x_regno < FIRST_PSEUDO_REGISTER)
2401 : return false;
2402 :
2403 : /* If the pseudo prefers REGNO explicitly, then do not consider
2404 : REGNO a bad spill choice. */
2405 0 : pref = reg_preferred_class (x_regno);
2406 0 : if (reg_class_size[pref] == 1)
2407 0 : return !TEST_HARD_REG_BIT (reg_class_contents[pref], regno);
2408 :
2409 : /* If the pseudo conflicts with REGNO, then we consider REGNO a
2410 : poor choice for a reload regno. */
2411 0 : a = ira_regno_allocno_map[x_regno];
2412 0 : n = ALLOCNO_NUM_OBJECTS (a);
2413 0 : for (i = 0; i < n; i++)
2414 : {
2415 0 : ira_object_t obj = ALLOCNO_OBJECT (a, i);
2416 0 : if (TEST_HARD_REG_BIT (OBJECT_TOTAL_CONFLICT_HARD_REGS (obj), regno))
2417 : return true;
2418 : }
2419 : return false;
2420 : }
2421 :
2422 : /* Return nonzero if REGNO is a particularly bad choice for reloading
2423 : IN or OUT. */
2424 : bool
2425 0 : ira_bad_reload_regno (int regno, rtx in, rtx out)
2426 : {
2427 0 : return (ira_bad_reload_regno_1 (regno, in)
2428 0 : || ira_bad_reload_regno_1 (regno, out));
2429 : }
2430 :
2431 : /* Add register clobbers from asm statements. */
2432 : static void
2433 1547921 : compute_regs_asm_clobbered (void)
2434 : {
2435 1547921 : basic_block bb;
2436 :
2437 16620168 : FOR_EACH_BB_FN (bb, cfun)
2438 : {
2439 15072247 : rtx_insn *insn;
2440 183108293 : FOR_BB_INSNS_REVERSE (bb, insn)
2441 : {
2442 168036046 : df_ref def;
2443 :
2444 168036046 : if (NONDEBUG_INSN_P (insn) && asm_noperands (PATTERN (insn)) >= 0)
2445 307427 : FOR_EACH_INSN_DEF (def, insn)
2446 : {
2447 210929 : unsigned int dregno = DF_REF_REGNO (def);
2448 210929 : if (HARD_REGISTER_NUM_P (dregno))
2449 279052 : add_to_hard_reg_set (&crtl->asm_clobbers,
2450 139526 : GET_MODE (DF_REF_REAL_REG (def)),
2451 : dregno);
2452 : }
2453 : }
2454 : }
2455 1547921 : }
2456 :
2457 :
2458 : /* Set up ELIMINABLE_REGSET, IRA_NO_ALLOC_REGS, and
2459 : REGS_EVER_LIVE. */
2460 : void
2461 1547921 : ira_setup_eliminable_regset (void)
2462 : {
2463 1547921 : int i;
2464 1547921 : static const struct {const int from, to; } eliminables[] = ELIMINABLE_REGS;
2465 1678847 : int fp_reg_count = hard_regno_nregs (HARD_FRAME_POINTER_REGNUM, Pmode);
2466 :
2467 : /* Setup is_leaf as frame_pointer_required may use it. This function
2468 : is called by sched_init before ira if scheduling is enabled. */
2469 1547921 : crtl->is_leaf = leaf_function_p ();
2470 :
2471 : /* FIXME: If EXIT_IGNORE_STACK is set, we will not save and restore
2472 : sp for alloca. So we can't eliminate the frame pointer in that
2473 : case. At some point, we should improve this by emitting the
2474 : sp-adjusting insns for this case. */
2475 1547921 : frame_pointer_needed
2476 3095842 : = (! flag_omit_frame_pointer
2477 1081568 : || (cfun->calls_alloca && EXIT_IGNORE_STACK)
2478 : /* We need the frame pointer to catch stack overflow exceptions if
2479 : the stack pointer is moving (as for the alloca case just above). */
2480 1072032 : || (STACK_CHECK_MOVING_SP
2481 1072032 : && flag_stack_check
2482 62 : && flag_exceptions
2483 26 : && cfun->can_throw_non_call_exceptions)
2484 1072028 : || crtl->accesses_prior_frames
2485 1069092 : || (SUPPORTS_STACK_ALIGNMENT && crtl->stack_realign_needed)
2486 2570257 : || targetm.frame_pointer_required ());
2487 :
2488 : /* The chance that FRAME_POINTER_NEEDED is changed from inspecting
2489 : RTL is very small. So if we use frame pointer for RA and RTL
2490 : actually prevents this, we will spill pseudos assigned to the
2491 : frame pointer in LRA. */
2492 :
2493 1547921 : if (frame_pointer_needed)
2494 1051260 : for (i = 0; i < fp_reg_count; i++)
2495 525630 : df_set_regs_ever_live (HARD_FRAME_POINTER_REGNUM + i, true);
2496 :
2497 1547921 : ira_no_alloc_regs = no_unit_alloc_regs;
2498 1547921 : CLEAR_HARD_REG_SET (eliminable_regset);
2499 :
2500 1547921 : compute_regs_asm_clobbered ();
2501 :
2502 : /* Build the regset of all eliminable registers and show we can't
2503 : use those that we already know won't be eliminated. */
2504 9287526 : for (i = 0; i < (int) ARRAY_SIZE (eliminables); i++)
2505 : {
2506 6191684 : bool cannot_elim
2507 6191684 : = (! targetm.can_eliminate (eliminables[i].from, eliminables[i].to)
2508 6191684 : || (eliminables[i].to == STACK_POINTER_REGNUM && frame_pointer_needed));
2509 :
2510 6191684 : if (!TEST_HARD_REG_BIT (crtl->asm_clobbers, eliminables[i].from))
2511 : {
2512 6191684 : SET_HARD_REG_BIT (eliminable_regset, eliminables[i].from);
2513 :
2514 6191684 : if (cannot_elim)
2515 1112555 : SET_HARD_REG_BIT (ira_no_alloc_regs, eliminables[i].from);
2516 : }
2517 0 : else if (cannot_elim)
2518 0 : error ("%s cannot be used in %<asm%> here",
2519 : reg_names[eliminables[i].from]);
2520 : else
2521 0 : df_set_regs_ever_live (eliminables[i].from, true);
2522 : }
2523 : if (!HARD_FRAME_POINTER_IS_FRAME_POINTER)
2524 : {
2525 3095842 : for (i = 0; i < fp_reg_count; i++)
2526 1547921 : if (global_regs[HARD_FRAME_POINTER_REGNUM + i])
2527 : /* Nothing to do: the register is already treated as live
2528 : where appropriate, and cannot be eliminated. */
2529 : ;
2530 1547900 : else if (!TEST_HARD_REG_BIT (crtl->asm_clobbers,
2531 : HARD_FRAME_POINTER_REGNUM + i))
2532 : {
2533 1546609 : SET_HARD_REG_BIT (eliminable_regset,
2534 : HARD_FRAME_POINTER_REGNUM + i);
2535 1546609 : if (frame_pointer_needed)
2536 525628 : SET_HARD_REG_BIT (ira_no_alloc_regs,
2537 : HARD_FRAME_POINTER_REGNUM + i);
2538 : }
2539 1291 : else if (frame_pointer_needed)
2540 0 : error ("%s cannot be used in %<asm%> here",
2541 : reg_names[HARD_FRAME_POINTER_REGNUM + i]);
2542 : else
2543 1291 : df_set_regs_ever_live (HARD_FRAME_POINTER_REGNUM + i, true);
2544 : }
2545 1547921 : }
2546 :
2547 :
2548 :
2549 : /* Vector of substitutions of register numbers,
2550 : used to map pseudo regs into hardware regs.
2551 : This is set up as a result of register allocation.
2552 : Element N is the hard reg assigned to pseudo reg N,
2553 : or is -1 if no hard reg was assigned.
2554 : If N is a hard reg number, element N is N. */
2555 : short *reg_renumber;
2556 :
2557 : /* Set up REG_RENUMBER and CALLER_SAVE_NEEDED (used by reload) from
2558 : the allocation found by IRA. */
2559 : static void
2560 1517127 : setup_reg_renumber (void)
2561 : {
2562 1517127 : int regno, hard_regno;
2563 1517127 : ira_allocno_t a;
2564 1517127 : ira_allocno_iterator ai;
2565 :
2566 1517127 : caller_save_needed = 0;
2567 38961363 : FOR_EACH_ALLOCNO (a, ai)
2568 : {
2569 37444236 : if (ira_use_lra_p && ALLOCNO_CAP_MEMBER (a) != NULL)
2570 3680218 : continue;
2571 : /* There are no caps at this point. */
2572 33764018 : ira_assert (ALLOCNO_CAP_MEMBER (a) == NULL);
2573 33764018 : if (! ALLOCNO_ASSIGNED_P (a))
2574 : /* It can happen if A is not referenced but partially anticipated
2575 : somewhere in a region. */
2576 0 : ALLOCNO_ASSIGNED_P (a) = true;
2577 33764018 : ira_free_allocno_updated_costs (a);
2578 33764018 : hard_regno = ALLOCNO_HARD_REGNO (a);
2579 33764018 : regno = ALLOCNO_REGNO (a);
2580 33764018 : reg_renumber[regno] = (hard_regno < 0 ? -1 : hard_regno);
2581 33764018 : if (hard_regno >= 0)
2582 : {
2583 30358948 : int i, nwords;
2584 30358948 : enum reg_class pclass;
2585 30358948 : ira_object_t obj;
2586 :
2587 30358948 : pclass = ira_pressure_class_translate[REGNO_REG_CLASS (hard_regno)];
2588 30358948 : nwords = ALLOCNO_NUM_OBJECTS (a);
2589 61868310 : for (i = 0; i < nwords; i++)
2590 : {
2591 31509362 : obj = ALLOCNO_OBJECT (a, i);
2592 31509362 : OBJECT_TOTAL_CONFLICT_HARD_REGS (obj)
2593 63018724 : |= ~reg_class_contents[pclass];
2594 : }
2595 30358948 : if (ira_need_caller_save_p (a, hard_regno))
2596 : {
2597 439159 : ira_assert (!optimize || flag_caller_saves
2598 : || (ALLOCNO_CALLS_CROSSED_NUM (a)
2599 : == ALLOCNO_CHEAP_CALLS_CROSSED_NUM (a))
2600 : || regno >= ira_reg_equiv_len
2601 : || ira_equiv_no_lvalue_p (regno));
2602 439159 : caller_save_needed = 1;
2603 : }
2604 : }
2605 : }
2606 1517127 : }
2607 :
2608 : /* Set up allocno assignment flags for further allocation
2609 : improvements. */
2610 : static void
2611 0 : setup_allocno_assignment_flags (void)
2612 : {
2613 0 : int hard_regno;
2614 0 : ira_allocno_t a;
2615 0 : ira_allocno_iterator ai;
2616 :
2617 0 : FOR_EACH_ALLOCNO (a, ai)
2618 : {
2619 0 : if (! ALLOCNO_ASSIGNED_P (a))
2620 : /* It can happen if A is not referenced but partially anticipated
2621 : somewhere in a region. */
2622 0 : ira_free_allocno_updated_costs (a);
2623 0 : hard_regno = ALLOCNO_HARD_REGNO (a);
2624 : /* Don't assign hard registers to allocnos which are destination
2625 : of removed store at the end of loop. It has no sense to keep
2626 : the same value in different hard registers. It is also
2627 : impossible to assign hard registers correctly to such
2628 : allocnos because the cost info and info about intersected
2629 : calls are incorrect for them. */
2630 0 : ALLOCNO_ASSIGNED_P (a) = (hard_regno >= 0
2631 0 : || ALLOCNO_EMIT_DATA (a)->mem_optimized_dest_p
2632 0 : || (ALLOCNO_MEMORY_COST (a)
2633 0 : - ALLOCNO_CLASS_COST (a)) < 0);
2634 0 : ira_assert
2635 : (hard_regno < 0
2636 : || ira_hard_reg_in_set_p (hard_regno, ALLOCNO_MODE (a),
2637 : reg_class_contents[ALLOCNO_CLASS (a)]));
2638 : }
2639 0 : }
2640 :
2641 : /* Evaluate overall allocation cost and the costs for using hard
2642 : registers and memory for allocnos. */
2643 : static void
2644 1517127 : calculate_allocation_cost (void)
2645 : {
2646 1517127 : int hard_regno, cost;
2647 1517127 : ira_allocno_t a;
2648 1517127 : ira_allocno_iterator ai;
2649 :
2650 1517127 : ira_overall_cost = ira_reg_cost = ira_mem_cost = 0;
2651 38961363 : FOR_EACH_ALLOCNO (a, ai)
2652 : {
2653 37444236 : hard_regno = ALLOCNO_HARD_REGNO (a);
2654 37444236 : ira_assert (hard_regno < 0
2655 : || (ira_hard_reg_in_set_p
2656 : (hard_regno, ALLOCNO_MODE (a),
2657 : reg_class_contents[ALLOCNO_CLASS (a)])));
2658 33633010 : if (hard_regno < 0)
2659 : {
2660 3811226 : cost = ALLOCNO_MEMORY_COST (a);
2661 3811226 : ira_mem_cost += cost;
2662 : }
2663 33633010 : else if (ALLOCNO_HARD_REG_COSTS (a) != NULL)
2664 : {
2665 8844616 : cost = (ALLOCNO_HARD_REG_COSTS (a)
2666 : [ira_class_hard_reg_index
2667 8844616 : [ALLOCNO_CLASS (a)][hard_regno]]);
2668 8844616 : ira_reg_cost += cost;
2669 : }
2670 : else
2671 : {
2672 24788394 : cost = ALLOCNO_CLASS_COST (a);
2673 24788394 : ira_reg_cost += cost;
2674 : }
2675 37444236 : ira_overall_cost += cost;
2676 : }
2677 :
2678 1517127 : if (internal_flag_ira_verbose > 0 && ira_dump_file != NULL)
2679 : {
2680 95 : fprintf (ira_dump_file,
2681 : "+++Costs: overall %" PRId64
2682 : ", reg %" PRId64
2683 : ", mem %" PRId64
2684 : ", ld %" PRId64
2685 : ", st %" PRId64
2686 : ", move %" PRId64,
2687 : ira_overall_cost, ira_reg_cost, ira_mem_cost,
2688 : ira_load_cost, ira_store_cost, ira_shuffle_cost);
2689 95 : fprintf (ira_dump_file, "\n+++ move loops %d, new jumps %d\n",
2690 : ira_move_loops_num, ira_additional_jumps_num);
2691 : }
2692 :
2693 1517127 : }
2694 :
2695 : #ifdef ENABLE_IRA_CHECKING
2696 : /* Check the correctness of the allocation. We do need this because
2697 : of complicated code to transform more one region internal
2698 : representation into one region representation. */
2699 : static void
2700 0 : check_allocation (void)
2701 : {
2702 0 : ira_allocno_t a;
2703 0 : int hard_regno, nregs, conflict_nregs;
2704 0 : ira_allocno_iterator ai;
2705 :
2706 0 : FOR_EACH_ALLOCNO (a, ai)
2707 : {
2708 0 : int n = ALLOCNO_NUM_OBJECTS (a);
2709 0 : int i;
2710 :
2711 0 : if (ALLOCNO_CAP_MEMBER (a) != NULL
2712 0 : || (hard_regno = ALLOCNO_HARD_REGNO (a)) < 0)
2713 0 : continue;
2714 0 : nregs = hard_regno_nregs (hard_regno, ALLOCNO_MODE (a));
2715 0 : if (nregs == 1)
2716 : /* We allocated a single hard register. */
2717 : n = 1;
2718 0 : else if (n > 1)
2719 : /* We allocated multiple hard registers, and we will test
2720 : conflicts in a granularity of single hard regs. */
2721 0 : nregs = 1;
2722 :
2723 0 : for (i = 0; i < n; i++)
2724 : {
2725 0 : ira_object_t obj = ALLOCNO_OBJECT (a, i);
2726 0 : ira_object_t conflict_obj;
2727 0 : ira_object_conflict_iterator oci;
2728 0 : int this_regno = hard_regno;
2729 0 : if (n > 1)
2730 : {
2731 0 : if (REG_WORDS_BIG_ENDIAN)
2732 : this_regno += n - i - 1;
2733 : else
2734 0 : this_regno += i;
2735 : }
2736 0 : FOR_EACH_OBJECT_CONFLICT (obj, conflict_obj, oci)
2737 : {
2738 0 : ira_allocno_t conflict_a = OBJECT_ALLOCNO (conflict_obj);
2739 0 : int conflict_hard_regno = ALLOCNO_HARD_REGNO (conflict_a);
2740 0 : if (conflict_hard_regno < 0)
2741 0 : continue;
2742 0 : if (ira_soft_conflict (a, conflict_a))
2743 0 : continue;
2744 :
2745 0 : conflict_nregs = hard_regno_nregs (conflict_hard_regno,
2746 0 : ALLOCNO_MODE (conflict_a));
2747 :
2748 0 : if (ALLOCNO_NUM_OBJECTS (conflict_a) > 1
2749 0 : && conflict_nregs == ALLOCNO_NUM_OBJECTS (conflict_a))
2750 : {
2751 0 : if (REG_WORDS_BIG_ENDIAN)
2752 : conflict_hard_regno += (ALLOCNO_NUM_OBJECTS (conflict_a)
2753 : - OBJECT_SUBWORD (conflict_obj) - 1);
2754 : else
2755 0 : conflict_hard_regno += OBJECT_SUBWORD (conflict_obj);
2756 0 : conflict_nregs = 1;
2757 : }
2758 :
2759 0 : if ((conflict_hard_regno <= this_regno
2760 0 : && this_regno < conflict_hard_regno + conflict_nregs)
2761 0 : || (this_regno <= conflict_hard_regno
2762 0 : && conflict_hard_regno < this_regno + nregs))
2763 : {
2764 0 : fprintf (stderr, "bad allocation for %d and %d\n",
2765 : ALLOCNO_REGNO (a), ALLOCNO_REGNO (conflict_a));
2766 0 : gcc_unreachable ();
2767 : }
2768 : }
2769 : }
2770 : }
2771 0 : }
2772 : #endif
2773 :
2774 : /* Allocate REG_EQUIV_INIT. Set up it from IRA_REG_EQUIV which should
2775 : be already calculated. */
2776 : static void
2777 1517127 : setup_reg_equiv_init (void)
2778 : {
2779 1517127 : int i;
2780 1517127 : int max_regno = max_reg_num ();
2781 :
2782 213563366 : for (i = 0; i < max_regno; i++)
2783 210529112 : reg_equiv_init (i) = ira_reg_equiv[i].init_insns;
2784 1517127 : }
2785 :
2786 : /* Update equiv regno from movement of FROM_REGNO to TO_REGNO. INSNS
2787 : are insns which were generated for such movement. It is assumed
2788 : that FROM_REGNO and TO_REGNO always have the same value at the
2789 : point of any move containing such registers. This function is used
2790 : to update equiv info for register shuffles on the region borders
2791 : and for caller save/restore insns. */
2792 : void
2793 2187936 : ira_update_equiv_info_by_shuffle_insn (int to_regno, int from_regno, rtx_insn *insns)
2794 : {
2795 2187936 : rtx_insn *insn;
2796 2187936 : rtx x, note;
2797 :
2798 2187936 : if (! ira_reg_equiv[from_regno].defined_p
2799 2187936 : && (! ira_reg_equiv[to_regno].defined_p
2800 812 : || ((x = ira_reg_equiv[to_regno].memory) != NULL_RTX
2801 812 : && ! MEM_READONLY_P (x))))
2802 : return;
2803 45917 : insn = insns;
2804 45917 : if (NEXT_INSN (insn) != NULL_RTX)
2805 : {
2806 0 : if (! ira_reg_equiv[to_regno].defined_p)
2807 : {
2808 0 : ira_assert (ira_reg_equiv[to_regno].init_insns == NULL_RTX);
2809 : return;
2810 : }
2811 0 : ira_reg_equiv[to_regno].defined_p = false;
2812 0 : ira_reg_equiv[to_regno].caller_save_p = false;
2813 0 : ira_reg_equiv[to_regno].memory
2814 0 : = ira_reg_equiv[to_regno].constant
2815 0 : = ira_reg_equiv[to_regno].invariant
2816 0 : = ira_reg_equiv[to_regno].init_insns = NULL;
2817 0 : if (internal_flag_ira_verbose > 3 && ira_dump_file != NULL)
2818 0 : fprintf (ira_dump_file,
2819 : " Invalidating equiv info for reg %d\n", to_regno);
2820 : return;
2821 : }
2822 : /* It is possible that FROM_REGNO still has no equivalence because
2823 : in shuffles to_regno<-from_regno and from_regno<-to_regno the 2nd
2824 : insn was not processed yet. */
2825 45917 : if (ira_reg_equiv[from_regno].defined_p)
2826 : {
2827 45917 : ira_reg_equiv[to_regno].defined_p = true;
2828 45917 : if ((x = ira_reg_equiv[from_regno].memory) != NULL_RTX)
2829 : {
2830 45727 : ira_assert (ira_reg_equiv[from_regno].invariant == NULL_RTX
2831 : && ira_reg_equiv[from_regno].constant == NULL_RTX);
2832 45727 : ira_assert (ira_reg_equiv[to_regno].memory == NULL_RTX
2833 : || rtx_equal_p (ira_reg_equiv[to_regno].memory, x));
2834 45727 : ira_reg_equiv[to_regno].memory = x;
2835 45727 : if (! MEM_READONLY_P (x))
2836 : /* We don't add the insn to insn init list because memory
2837 : equivalence is just to say what memory is better to use
2838 : when the pseudo is spilled. */
2839 : return;
2840 : }
2841 190 : else if ((x = ira_reg_equiv[from_regno].constant) != NULL_RTX)
2842 : {
2843 93 : ira_assert (ira_reg_equiv[from_regno].invariant == NULL_RTX);
2844 93 : ira_assert (ira_reg_equiv[to_regno].constant == NULL_RTX
2845 : || rtx_equal_p (ira_reg_equiv[to_regno].constant, x));
2846 93 : ira_reg_equiv[to_regno].constant = x;
2847 : }
2848 : else
2849 : {
2850 97 : x = ira_reg_equiv[from_regno].invariant;
2851 97 : ira_assert (x != NULL_RTX);
2852 97 : ira_assert (ira_reg_equiv[to_regno].invariant == NULL_RTX
2853 : || rtx_equal_p (ira_reg_equiv[to_regno].invariant, x));
2854 97 : ira_reg_equiv[to_regno].invariant = x;
2855 : }
2856 203 : if (find_reg_note (insn, REG_EQUIV, x) == NULL_RTX)
2857 : {
2858 203 : note = set_unique_reg_note (insn, REG_EQUIV, copy_rtx (x));
2859 203 : gcc_assert (note != NULL_RTX);
2860 203 : if (internal_flag_ira_verbose > 3 && ira_dump_file != NULL)
2861 : {
2862 0 : fprintf (ira_dump_file,
2863 : " Adding equiv note to insn %u for reg %d ",
2864 0 : INSN_UID (insn), to_regno);
2865 0 : dump_value_slim (ira_dump_file, x, 1);
2866 0 : fprintf (ira_dump_file, "\n");
2867 : }
2868 : }
2869 : }
2870 203 : ira_reg_equiv[to_regno].init_insns
2871 406 : = gen_rtx_INSN_LIST (VOIDmode, insn,
2872 203 : ira_reg_equiv[to_regno].init_insns);
2873 203 : if (internal_flag_ira_verbose > 3 && ira_dump_file != NULL)
2874 0 : fprintf (ira_dump_file,
2875 : " Adding equiv init move insn %u to reg %d\n",
2876 0 : INSN_UID (insn), to_regno);
2877 : }
2878 :
2879 : /* Fix values of array REG_EQUIV_INIT after live range splitting done
2880 : by IRA. */
2881 : static void
2882 2129926 : fix_reg_equiv_init (void)
2883 : {
2884 2129926 : int max_regno = max_reg_num ();
2885 2129926 : int i, new_regno, max;
2886 2129926 : rtx set;
2887 2129926 : rtx_insn_list *x, *next, *prev;
2888 2129926 : rtx_insn *insn;
2889 :
2890 2129926 : if (max_regno_before_ira < max_regno)
2891 : {
2892 526454 : max = vec_safe_length (reg_equivs);
2893 526454 : grow_reg_equivs ();
2894 49860198 : for (i = FIRST_PSEUDO_REGISTER; i < max; i++)
2895 48807290 : for (prev = NULL, x = reg_equiv_init (i);
2896 53438353 : x != NULL_RTX;
2897 4631063 : x = next)
2898 : {
2899 4631063 : next = x->next ();
2900 4631063 : insn = x->insn ();
2901 4631063 : set = single_set (insn);
2902 4631063 : ira_assert (set != NULL_RTX
2903 : && (REG_P (SET_DEST (set)) || REG_P (SET_SRC (set))));
2904 4631063 : if (REG_P (SET_DEST (set))
2905 4631063 : && ((int) REGNO (SET_DEST (set)) == i
2906 0 : || (int) ORIGINAL_REGNO (SET_DEST (set)) == i))
2907 : new_regno = REGNO (SET_DEST (set));
2908 451111 : else if (REG_P (SET_SRC (set))
2909 451111 : && ((int) REGNO (SET_SRC (set)) == i
2910 0 : || (int) ORIGINAL_REGNO (SET_SRC (set)) == i))
2911 : new_regno = REGNO (SET_SRC (set));
2912 : else
2913 0 : gcc_unreachable ();
2914 4631063 : if (new_regno == i)
2915 : prev = x;
2916 : else
2917 : {
2918 : /* Remove the wrong list element. */
2919 0 : if (prev == NULL_RTX)
2920 0 : reg_equiv_init (i) = next;
2921 : else
2922 0 : XEXP (prev, 1) = next;
2923 0 : XEXP (x, 1) = reg_equiv_init (new_regno);
2924 0 : reg_equiv_init (new_regno) = x;
2925 : }
2926 : }
2927 : }
2928 2129926 : }
2929 :
2930 : #ifdef ENABLE_IRA_CHECKING
2931 : /* Print redundant memory-memory copies. */
2932 : static void
2933 1064963 : print_redundant_copies (void)
2934 : {
2935 1064963 : int hard_regno;
2936 1064963 : ira_allocno_t a;
2937 1064963 : ira_copy_t cp, next_cp;
2938 1064963 : ira_allocno_iterator ai;
2939 :
2940 26735318 : FOR_EACH_ALLOCNO (a, ai)
2941 : {
2942 25670355 : if (ALLOCNO_CAP_MEMBER (a) != NULL)
2943 : /* It is a cap. */
2944 3680218 : continue;
2945 21990137 : hard_regno = ALLOCNO_HARD_REGNO (a);
2946 21990137 : if (hard_regno >= 0)
2947 18675267 : continue;
2948 4287276 : for (cp = ALLOCNO_COPIES (a); cp != NULL; cp = next_cp)
2949 972406 : if (cp->first == a)
2950 375746 : next_cp = cp->next_first_allocno_copy;
2951 : else
2952 : {
2953 596660 : next_cp = cp->next_second_allocno_copy;
2954 596660 : if (internal_flag_ira_verbose > 4 && ira_dump_file != NULL
2955 1 : && cp->insn != NULL_RTX
2956 0 : && ALLOCNO_HARD_REGNO (cp->first) == hard_regno)
2957 0 : fprintf (ira_dump_file,
2958 : " Redundant move from %d(freq %d):%d\n",
2959 0 : INSN_UID (cp->insn), cp->freq, hard_regno);
2960 : }
2961 : }
2962 1064963 : }
2963 : #endif
2964 :
2965 : /* Setup preferred and alternative classes for new pseudo-registers
2966 : created by IRA starting with START. */
2967 : static void
2968 1101131 : setup_preferred_alternate_classes_for_new_pseudos (int start)
2969 : {
2970 1101131 : int i, old_regno;
2971 1101131 : int max_regno = max_reg_num ();
2972 :
2973 3399164 : for (i = start; i < max_regno; i++)
2974 : {
2975 1196902 : old_regno = ORIGINAL_REGNO (regno_reg_rtx[i]);
2976 1196902 : ira_assert (i != old_regno);
2977 1196902 : setup_reg_classes (i, reg_preferred_class (old_regno),
2978 : reg_alternate_class (old_regno),
2979 : reg_allocno_class (old_regno));
2980 1196902 : if (internal_flag_ira_verbose > 2 && ira_dump_file != NULL)
2981 0 : fprintf (ira_dump_file,
2982 : " New r%d: setting preferred %s, alternative %s\n",
2983 0 : i, reg_class_names[reg_preferred_class (old_regno)],
2984 0 : reg_class_names[reg_alternate_class (old_regno)]);
2985 : }
2986 1101131 : }
2987 :
2988 :
2989 : /* The number of entries allocated in reg_info. */
2990 : static int allocated_reg_info_size;
2991 :
2992 : /* Regional allocation can create new pseudo-registers. This function
2993 : expands some arrays for pseudo-registers. */
2994 : static void
2995 1101131 : expand_reg_info (void)
2996 : {
2997 1101131 : int i;
2998 1101131 : int size = max_reg_num ();
2999 :
3000 1101131 : resize_reg_info ();
3001 2298033 : for (i = allocated_reg_info_size; i < size; i++)
3002 1196902 : setup_reg_classes (i, GENERAL_REGS, ALL_REGS, GENERAL_REGS);
3003 1101131 : setup_preferred_alternate_classes_for_new_pseudos (allocated_reg_info_size);
3004 1101131 : allocated_reg_info_size = size;
3005 1101131 : }
3006 :
3007 : /* Return TRUE if there is too high register pressure in the function.
3008 : It is used to decide when stack slot sharing is worth to do. */
3009 : static bool
3010 1517127 : too_high_register_pressure_p (void)
3011 : {
3012 1517127 : int i;
3013 1517127 : enum reg_class pclass;
3014 :
3015 7625212 : for (i = 0; i < ira_pressure_classes_num; i++)
3016 : {
3017 6108087 : pclass = ira_pressure_classes[i];
3018 6108087 : if (ira_loop_tree_root->reg_pressure[pclass] > 10000)
3019 : return true;
3020 : }
3021 : return false;
3022 : }
3023 :
3024 :
3025 :
3026 : /* Indicate that hard register number FROM was eliminated and replaced with
3027 : an offset from hard register number TO. The status of hard registers live
3028 : at the start of a basic block is updated by replacing a use of FROM with
3029 : a use of TO. */
3030 :
3031 : void
3032 0 : mark_elimination (int from, int to)
3033 : {
3034 0 : basic_block bb;
3035 0 : bitmap r;
3036 :
3037 0 : FOR_EACH_BB_FN (bb, cfun)
3038 : {
3039 0 : r = DF_LR_IN (bb);
3040 0 : if (bitmap_bit_p (r, from))
3041 : {
3042 0 : bitmap_clear_bit (r, from);
3043 0 : bitmap_set_bit (r, to);
3044 : }
3045 0 : if (! df_live)
3046 0 : continue;
3047 0 : r = DF_LIVE_IN (bb);
3048 0 : if (bitmap_bit_p (r, from))
3049 : {
3050 0 : bitmap_clear_bit (r, from);
3051 0 : bitmap_set_bit (r, to);
3052 : }
3053 : }
3054 0 : }
3055 :
3056 :
3057 :
3058 : /* The length of the following array. */
3059 : int ira_reg_equiv_len;
3060 :
3061 : /* Info about equiv. info for each register. */
3062 : struct ira_reg_equiv_s *ira_reg_equiv;
3063 :
3064 : /* Expand ira_reg_equiv if necessary. */
3065 : void
3066 15318761 : ira_expand_reg_equiv (void)
3067 : {
3068 15318761 : int old = ira_reg_equiv_len;
3069 :
3070 15318761 : if (ira_reg_equiv_len > max_reg_num ())
3071 : return;
3072 1520348 : ira_reg_equiv_len = max_reg_num () * 3 / 2 + 1;
3073 1520348 : ira_reg_equiv
3074 3040696 : = (struct ira_reg_equiv_s *) xrealloc (ira_reg_equiv,
3075 1520348 : ira_reg_equiv_len
3076 : * sizeof (struct ira_reg_equiv_s));
3077 1520348 : gcc_assert (old < ira_reg_equiv_len);
3078 1520348 : memset (ira_reg_equiv + old, 0,
3079 1520348 : sizeof (struct ira_reg_equiv_s) * (ira_reg_equiv_len - old));
3080 : }
3081 :
3082 : static void
3083 1517127 : init_reg_equiv (void)
3084 : {
3085 1517127 : ira_reg_equiv_len = 0;
3086 1517127 : ira_reg_equiv = NULL;
3087 0 : ira_expand_reg_equiv ();
3088 0 : }
3089 :
3090 : static void
3091 1517127 : finish_reg_equiv (void)
3092 : {
3093 1517127 : free (ira_reg_equiv);
3094 0 : }
3095 :
3096 :
3097 :
3098 : struct equivalence
3099 : {
3100 : /* Set when a REG_EQUIV note is found or created. Use to
3101 : keep track of what memory accesses might be created later,
3102 : e.g. by reload. */
3103 : rtx replacement;
3104 : rtx *src_p;
3105 :
3106 : /* The list of each instruction which initializes this register.
3107 :
3108 : NULL indicates we know nothing about this register's equivalence
3109 : properties.
3110 :
3111 : An INSN_LIST with a NULL insn indicates this pseudo is already
3112 : known to not have a valid equivalence. */
3113 : rtx_insn_list *init_insns;
3114 :
3115 : /* Loop depth is used to recognize equivalences which appear
3116 : to be present within the same loop (or in an inner loop). */
3117 : short loop_depth;
3118 : /* Nonzero if this had a preexisting REG_EQUIV note. */
3119 : unsigned char is_arg_equivalence : 1;
3120 : /* Set when an attempt should be made to replace a register
3121 : with the associated src_p entry. */
3122 : unsigned char replace : 1;
3123 : /* Set if this register has no known equivalence. */
3124 : unsigned char no_equiv : 1;
3125 : /* Set if this register is mentioned in a paradoxical subreg. */
3126 : unsigned char pdx_subregs : 1;
3127 : };
3128 :
3129 : /* reg_equiv[N] (where N is a pseudo reg number) is the equivalence
3130 : structure for that register. */
3131 : static struct equivalence *reg_equiv;
3132 :
3133 : /* Used for communication between the following two functions. */
3134 : struct equiv_mem_data
3135 : {
3136 : /* A MEM that we wish to ensure remains unchanged. */
3137 : rtx equiv_mem;
3138 :
3139 : /* Set true if EQUIV_MEM is modified. */
3140 : bool equiv_mem_modified;
3141 : };
3142 :
3143 : /* If EQUIV_MEM is modified by modifying DEST, indicate that it is modified.
3144 : Called via note_stores. */
3145 : static void
3146 14786286 : validate_equiv_mem_from_store (rtx dest, const_rtx set ATTRIBUTE_UNUSED,
3147 : void *data)
3148 : {
3149 14786286 : struct equiv_mem_data *info = (struct equiv_mem_data *) data;
3150 :
3151 14786286 : if ((REG_P (dest)
3152 11192934 : && reg_overlap_mentioned_p (dest, info->equiv_mem))
3153 25964172 : || (MEM_P (dest)
3154 3560626 : && anti_dependence (info->equiv_mem, dest)))
3155 303631 : info->equiv_mem_modified = true;
3156 14786286 : }
3157 :
3158 : static bool equiv_init_varies_p (rtx x);
3159 :
3160 : enum valid_equiv { valid_none, valid_combine, valid_reload };
3161 :
3162 : /* Verify that no store between START and the death of REG invalidates
3163 : MEMREF. MEMREF is invalidated by modifying a register used in MEMREF,
3164 : by storing into an overlapping memory location, or with a non-const
3165 : CALL_INSN.
3166 :
3167 : Return VALID_RELOAD if MEMREF remains valid for both reload and
3168 : combine_and_move insns, VALID_COMBINE if only valid for
3169 : combine_and_move_insns, and VALID_NONE otherwise. */
3170 : static enum valid_equiv
3171 3965958 : validate_equiv_mem (rtx_insn *start, rtx reg, rtx memref)
3172 : {
3173 3965958 : rtx_insn *insn;
3174 3965958 : rtx note;
3175 3965958 : struct equiv_mem_data info = { memref, false };
3176 3965958 : enum valid_equiv ret = valid_reload;
3177 :
3178 : /* If the memory reference has side effects or is volatile, it isn't a
3179 : valid equivalence. */
3180 3965958 : if (side_effects_p (memref))
3181 : return valid_none;
3182 :
3183 21648796 : for (insn = start; insn; insn = NEXT_INSN (insn))
3184 : {
3185 21648601 : if (!INSN_P (insn))
3186 1431348 : continue;
3187 :
3188 20217253 : if (find_reg_note (insn, REG_DEAD, reg))
3189 : return ret;
3190 :
3191 17446358 : if (auto call = dyn_cast<rtx_call_insn *> (insn))
3192 : {
3193 : /* We can combine a reg def from one insn into a reg use in
3194 : another over a call if the memory is readonly or the call
3195 : const/pure. However, we can't set reg_equiv notes up for
3196 : reload over any call. The problem is the equivalent form
3197 : may reference a pseudo which gets assigned a call
3198 : clobbered hard reg. When we later replace REG with its
3199 : equivalent form, the value in the call-clobbered reg has
3200 : been changed and all hell breaks loose. */
3201 88732 : ret = valid_combine;
3202 88732 : if (!MEM_READONLY_P (memref)
3203 88732 : && (!RTL_CONST_OR_PURE_CALL_P (call)
3204 8407 : || equiv_init_varies_p (XEXP (memref, 0))))
3205 : return valid_none;
3206 : }
3207 :
3208 17363495 : note_stores (insn, validate_equiv_mem_from_store, &info);
3209 17363495 : if (info.equiv_mem_modified)
3210 : return valid_none;
3211 :
3212 : /* If a register mentioned in MEMREF is modified via an
3213 : auto-increment, we lose the equivalence. Do the same if one
3214 : dies; although we could extend the life, it doesn't seem worth
3215 : the trouble. */
3216 :
3217 24236996 : for (note = REG_NOTES (insn); note; note = XEXP (note, 1))
3218 7811943 : if ((REG_NOTE_KIND (note) == REG_INC
3219 7811943 : || REG_NOTE_KIND (note) == REG_DEAD)
3220 5937396 : && REG_P (XEXP (note, 0))
3221 13749339 : && reg_overlap_mentioned_p (XEXP (note, 0), memref))
3222 : return valid_none;
3223 : }
3224 :
3225 : return valid_none;
3226 : }
3227 :
3228 : /* Returns false if X is known to be invariant. */
3229 : static bool
3230 911194 : equiv_init_varies_p (rtx x)
3231 : {
3232 911194 : RTX_CODE code = GET_CODE (x);
3233 911194 : int i;
3234 911194 : const char *fmt;
3235 :
3236 911194 : switch (code)
3237 : {
3238 229476 : case MEM:
3239 229476 : return !MEM_READONLY_P (x) || equiv_init_varies_p (XEXP (x, 0));
3240 :
3241 : case CONST:
3242 : CASE_CONST_ANY:
3243 : case SYMBOL_REF:
3244 : case LABEL_REF:
3245 : return false;
3246 :
3247 231321 : case REG:
3248 231321 : return reg_equiv[REGNO (x)].replace == 0 && rtx_varies_p (x, 0);
3249 :
3250 0 : case ASM_OPERANDS:
3251 0 : if (MEM_VOLATILE_P (x))
3252 : return true;
3253 :
3254 : /* Fall through. */
3255 :
3256 157545 : default:
3257 157545 : break;
3258 : }
3259 :
3260 157545 : fmt = GET_RTX_FORMAT (code);
3261 377928 : for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
3262 269652 : if (fmt[i] == 'e')
3263 : {
3264 242699 : if (equiv_init_varies_p (XEXP (x, i)))
3265 : return true;
3266 : }
3267 26953 : else if (fmt[i] == 'E')
3268 : {
3269 : int j;
3270 3333 : for (j = 0; j < XVECLEN (x, i); j++)
3271 2978 : if (equiv_init_varies_p (XVECEXP (x, i, j)))
3272 : return true;
3273 : }
3274 :
3275 : return false;
3276 : }
3277 :
3278 : /* Returns true if X (used to initialize register REGNO) is movable.
3279 : X is only movable if the registers it uses have equivalent initializations
3280 : which appear to be within the same loop (or in an inner loop) and movable
3281 : or if they are not candidates for local_alloc and don't vary. */
3282 : static bool
3283 10547657 : equiv_init_movable_p (rtx x, int regno)
3284 : {
3285 13565849 : int i, j;
3286 13565849 : const char *fmt;
3287 13565849 : enum rtx_code code = GET_CODE (x);
3288 :
3289 13565849 : switch (code)
3290 : {
3291 3018192 : case SET:
3292 3018192 : return equiv_init_movable_p (SET_SRC (x), regno);
3293 :
3294 : case CLOBBER:
3295 : return false;
3296 :
3297 : case PRE_INC:
3298 : case PRE_DEC:
3299 : case POST_INC:
3300 : case POST_DEC:
3301 : case PRE_MODIFY:
3302 : case POST_MODIFY:
3303 : return false;
3304 :
3305 1759188 : case REG:
3306 1759188 : return ((reg_equiv[REGNO (x)].loop_depth >= reg_equiv[regno].loop_depth
3307 1305882 : && reg_equiv[REGNO (x)].replace)
3308 2993347 : || (REG_BASIC_BLOCK (REGNO (x)) < NUM_FIXED_BLOCKS
3309 1580509 : && ! rtx_varies_p (x, 0)));
3310 :
3311 : case UNSPEC_VOLATILE:
3312 : return false;
3313 :
3314 0 : case ASM_OPERANDS:
3315 0 : if (MEM_VOLATILE_P (x))
3316 : return false;
3317 :
3318 : /* Fall through. */
3319 :
3320 8125769 : default:
3321 8125769 : break;
3322 : }
3323 :
3324 8125769 : fmt = GET_RTX_FORMAT (code);
3325 19305780 : for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
3326 13159238 : switch (fmt[i])
3327 : {
3328 5959536 : case 'e':
3329 5959536 : if (! equiv_init_movable_p (XEXP (x, i), regno))
3330 : return false;
3331 : break;
3332 783654 : case 'E':
3333 1028177 : for (j = XVECLEN (x, i) - 1; j >= 0; j--)
3334 907229 : if (! equiv_init_movable_p (XVECEXP (x, i, j), regno))
3335 : return false;
3336 : break;
3337 : }
3338 :
3339 : return true;
3340 : }
3341 :
3342 : static bool memref_referenced_p (rtx memref, rtx x, bool read_p);
3343 :
3344 : /* Auxiliary function for memref_referenced_p. Process setting X for
3345 : MEMREF store. */
3346 : static bool
3347 753776 : process_set_for_memref_referenced_p (rtx memref, rtx x)
3348 : {
3349 : /* If we are setting a MEM, it doesn't count (its address does), but any
3350 : other SET_DEST that has a MEM in it is referencing the MEM. */
3351 753776 : if (MEM_P (x))
3352 : {
3353 582196 : if (memref_referenced_p (memref, XEXP (x, 0), true))
3354 : return true;
3355 : }
3356 171580 : else if (memref_referenced_p (memref, x, false))
3357 : return true;
3358 :
3359 : return false;
3360 : }
3361 :
3362 : /* TRUE if X references a memory location (as a read if READ_P) that
3363 : would be affected by a store to MEMREF. */
3364 : static bool
3365 2816184 : memref_referenced_p (rtx memref, rtx x, bool read_p)
3366 : {
3367 3537117 : int i, j;
3368 3537117 : const char *fmt;
3369 3537117 : enum rtx_code code = GET_CODE (x);
3370 :
3371 3537117 : switch (code)
3372 : {
3373 : case CONST:
3374 : case LABEL_REF:
3375 : case SYMBOL_REF:
3376 : CASE_CONST_ANY:
3377 : case PC:
3378 : case HIGH:
3379 : case LO_SUM:
3380 : return false;
3381 :
3382 1416480 : case REG:
3383 1416480 : return (reg_equiv[REGNO (x)].replacement
3384 1493963 : && memref_referenced_p (memref,
3385 77483 : reg_equiv[REGNO (x)].replacement, read_p));
3386 :
3387 125864 : case MEM:
3388 : /* Memory X might have another effective type than MEMREF. */
3389 125864 : if (read_p || true_dependence (memref, VOIDmode, x))
3390 : return true;
3391 : break;
3392 :
3393 735753 : case SET:
3394 735753 : if (process_set_for_memref_referenced_p (memref, SET_DEST (x)))
3395 : return true;
3396 :
3397 720933 : return memref_referenced_p (memref, SET_SRC (x), true);
3398 :
3399 18023 : case CLOBBER:
3400 18023 : if (process_set_for_memref_referenced_p (memref, XEXP (x, 0)))
3401 : return true;
3402 :
3403 : return false;
3404 :
3405 0 : case PRE_DEC:
3406 0 : case POST_DEC:
3407 0 : case PRE_INC:
3408 0 : case POST_INC:
3409 0 : if (process_set_for_memref_referenced_p (memref, XEXP (x, 0)))
3410 : return true;
3411 :
3412 0 : return memref_referenced_p (memref, XEXP (x, 0), true);
3413 :
3414 0 : case POST_MODIFY:
3415 0 : case PRE_MODIFY:
3416 : /* op0 = op0 + op1 */
3417 0 : if (process_set_for_memref_referenced_p (memref, XEXP (x, 0)))
3418 : return true;
3419 :
3420 0 : if (memref_referenced_p (memref, XEXP (x, 0), true))
3421 : return true;
3422 :
3423 0 : return memref_referenced_p (memref, XEXP (x, 1), true);
3424 :
3425 : default:
3426 : break;
3427 : }
3428 :
3429 652091 : fmt = GET_RTX_FORMAT (code);
3430 1873471 : for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
3431 1256415 : switch (fmt[i])
3432 : {
3433 1206027 : case 'e':
3434 1206027 : if (memref_referenced_p (memref, XEXP (x, i), read_p))
3435 : return true;
3436 : break;
3437 21493 : case 'E':
3438 60607 : for (j = XVECLEN (x, i) - 1; j >= 0; j--)
3439 43112 : if (memref_referenced_p (memref, XVECEXP (x, i, j), read_p))
3440 : return true;
3441 : break;
3442 : }
3443 :
3444 : return false;
3445 : }
3446 :
3447 : /* TRUE if some insn in the range (START, END] references a memory location
3448 : that would be affected by a store to MEMREF.
3449 :
3450 : Callers should not call this routine if START is after END in the
3451 : RTL chain. */
3452 :
3453 : static bool
3454 584468 : memref_used_between_p (rtx memref, rtx_insn *start, rtx_insn *end)
3455 : {
3456 584468 : rtx_insn *insn;
3457 :
3458 584468 : for (insn = NEXT_INSN (start);
3459 3866386 : insn && insn != NEXT_INSN (end);
3460 1357048 : insn = NEXT_INSN (insn))
3461 : {
3462 1470942 : if (!NONDEBUG_INSN_P (insn))
3463 735156 : continue;
3464 :
3465 735786 : if (memref_referenced_p (memref, PATTERN (insn), false))
3466 : return true;
3467 :
3468 : /* Nonconst functions may access memory. */
3469 1357048 : auto call = dyn_cast<rtx_call_insn *> (insn);
3470 0 : if (call && ! RTL_CONST_CALL_P (call))
3471 : return true;
3472 : }
3473 :
3474 470574 : gcc_assert (insn == NEXT_INSN (end));
3475 : return false;
3476 : }
3477 :
3478 : /* Mark REG as having no known equivalence.
3479 : Some instructions might have been processed before and furnished
3480 : with REG_EQUIV notes for this register; these notes will have to be
3481 : removed.
3482 : STORE is the piece of RTL that does the non-constant / conflicting
3483 : assignment - a SET, CLOBBER or REG_INC note. It is currently not used,
3484 : but needs to be there because this function is called from note_stores. */
3485 : static void
3486 52371447 : no_equiv (rtx reg, const_rtx store ATTRIBUTE_UNUSED,
3487 : void *data ATTRIBUTE_UNUSED)
3488 : {
3489 52371447 : int regno;
3490 52371447 : rtx_insn_list *list;
3491 :
3492 52371447 : if (!REG_P (reg))
3493 : return;
3494 36266401 : regno = REGNO (reg);
3495 36266401 : reg_equiv[regno].no_equiv = 1;
3496 36266401 : list = reg_equiv[regno].init_insns;
3497 66020749 : if (list && list->insn () == NULL)
3498 : return;
3499 7319849 : reg_equiv[regno].init_insns = gen_rtx_INSN_LIST (VOIDmode, NULL_RTX, NULL);
3500 7319849 : reg_equiv[regno].replacement = NULL_RTX;
3501 : /* This doesn't matter for equivalences made for argument registers, we
3502 : should keep their initialization insns. */
3503 7319849 : if (reg_equiv[regno].is_arg_equivalence)
3504 : return;
3505 7314581 : ira_reg_equiv[regno].defined_p = false;
3506 7314581 : ira_reg_equiv[regno].caller_save_p = false;
3507 7314581 : ira_reg_equiv[regno].init_insns = NULL;
3508 8162625 : for (; list; list = list->next ())
3509 : {
3510 848044 : rtx_insn *insn = list->insn ();
3511 848044 : remove_note (insn, find_reg_note (insn, REG_EQUIV, NULL_RTX));
3512 : }
3513 : }
3514 :
3515 : /* Check whether the SUBREG is a paradoxical subreg and set the result
3516 : in PDX_SUBREGS. */
3517 :
3518 : static void
3519 86062421 : set_paradoxical_subreg (rtx_insn *insn)
3520 : {
3521 86062421 : subrtx_iterator::array_type array;
3522 544362237 : FOR_EACH_SUBRTX (iter, array, PATTERN (insn), NONCONST)
3523 : {
3524 458299816 : const_rtx subreg = *iter;
3525 458299816 : if (GET_CODE (subreg) == SUBREG)
3526 : {
3527 3095980 : const_rtx reg = SUBREG_REG (subreg);
3528 459123173 : if (REG_P (reg) && paradoxical_subreg_p (subreg))
3529 823357 : reg_equiv[REGNO (reg)].pdx_subregs = true;
3530 : }
3531 : }
3532 86062421 : }
3533 :
3534 : /* In DEBUG_INSN location adjust REGs from CLEARED_REGS bitmap to the
3535 : equivalent replacement. */
3536 :
3537 : static rtx
3538 46352799 : adjust_cleared_regs (rtx loc, const_rtx old_rtx ATTRIBUTE_UNUSED, void *data)
3539 : {
3540 46352799 : if (REG_P (loc))
3541 : {
3542 6935350 : bitmap cleared_regs = (bitmap) data;
3543 6935350 : if (bitmap_bit_p (cleared_regs, REGNO (loc)))
3544 17644 : return simplify_replace_fn_rtx (copy_rtx (*reg_equiv[REGNO (loc)].src_p),
3545 17644 : NULL_RTX, adjust_cleared_regs, data);
3546 : }
3547 : return NULL_RTX;
3548 : }
3549 :
3550 : /* Given register REGNO is set only once, return true if the defining
3551 : insn dominates all uses. */
3552 :
3553 : static bool
3554 50028 : def_dominates_uses (int regno)
3555 : {
3556 50028 : df_ref def = DF_REG_DEF_CHAIN (regno);
3557 :
3558 50028 : struct df_insn_info *def_info = DF_REF_INSN_INFO (def);
3559 : /* If this is an artificial def (eh handler regs, hard frame pointer
3560 : for non-local goto, regs defined on function entry) then def_info
3561 : is NULL and the reg is always live before any use. We might
3562 : reasonably return true in that case, but since the only call
3563 : of this function is currently here in ira.cc when we are looking
3564 : at a defining insn we can't have an artificial def as that would
3565 : bump DF_REG_DEF_COUNT. */
3566 50028 : gcc_assert (DF_REG_DEF_COUNT (regno) == 1 && def_info != NULL);
3567 :
3568 50028 : rtx_insn *def_insn = DF_REF_INSN (def);
3569 50028 : basic_block def_bb = BLOCK_FOR_INSN (def_insn);
3570 :
3571 50028 : for (df_ref use = DF_REG_USE_CHAIN (regno);
3572 168166 : use;
3573 118138 : use = DF_REF_NEXT_REG (use))
3574 : {
3575 118139 : struct df_insn_info *use_info = DF_REF_INSN_INFO (use);
3576 : /* Only check real uses, not artificial ones. */
3577 118139 : if (use_info)
3578 : {
3579 118139 : rtx_insn *use_insn = DF_REF_INSN (use);
3580 118139 : if (!DEBUG_INSN_P (use_insn))
3581 : {
3582 117872 : basic_block use_bb = BLOCK_FOR_INSN (use_insn);
3583 117872 : if (use_bb != def_bb
3584 117872 : ? !dominated_by_p (CDI_DOMINATORS, use_bb, def_bb)
3585 79574 : : DF_INSN_INFO_LUID (use_info) < DF_INSN_INFO_LUID (def_info))
3586 : return false;
3587 : }
3588 : }
3589 : }
3590 : return true;
3591 : }
3592 :
3593 : /* Scan the instructions before update_equiv_regs. Record which registers
3594 : are referenced as paradoxical subregs. Also check for cases in which
3595 : the current function needs to save a register that one of its call
3596 : instructions clobbers.
3597 :
3598 : These things are logically unrelated, but it's more efficient to do
3599 : them together. */
3600 :
3601 : static void
3602 1517127 : update_equiv_regs_prescan (void)
3603 : {
3604 1517127 : basic_block bb;
3605 1517127 : rtx_insn *insn;
3606 1517127 : function_abi_aggregator callee_abis;
3607 :
3608 16239609 : FOR_EACH_BB_FN (bb, cfun)
3609 180522350 : FOR_BB_INSNS (bb, insn)
3610 165799868 : if (NONDEBUG_INSN_P (insn))
3611 : {
3612 86062421 : set_paradoxical_subreg (insn);
3613 171973166 : if (auto call = dyn_cast<rtx_call_insn *> (insn))
3614 6173298 : callee_abis.note_callee_abi (insn_callee_abi (call));
3615 : }
3616 :
3617 1517127 : HARD_REG_SET extra_caller_saves = callee_abis.caller_save_regs (*crtl->abi);
3618 :
3619 1517127 : hard_reg_set_iterator hrsi;
3620 1517127 : unsigned int regno = 0;
3621 3034254 : if (!hard_reg_set_empty_p (extra_caller_saves))
3622 : {
3623 526093 : EXECUTE_IF_SET_IN_HARD_REG_SET (extra_caller_saves, 0, regno, hrsi)
3624 484959 : df_set_regs_ever_live (regno, true);
3625 : }
3626 1517127 : }
3627 :
3628 : /* Find registers that are equivalent to a single value throughout the
3629 : compilation (either because they can be referenced in memory or are
3630 : set once from a single constant). Lower their priority for a
3631 : register.
3632 :
3633 : If such a register is only referenced once, try substituting its
3634 : value into the using insn. If it succeeds, we can eliminate the
3635 : register completely.
3636 :
3637 : Initialize init_insns in ira_reg_equiv array. */
3638 : static void
3639 1517127 : update_equiv_regs (void)
3640 : {
3641 1517127 : rtx_insn *insn;
3642 1517127 : basic_block bb;
3643 :
3644 : /* Scan the insns and find which registers have equivalences. Do this
3645 : in a separate scan of the insns because (due to -fcse-follow-jumps)
3646 : a register can be set below its use. */
3647 1517127 : bitmap setjmp_crosses = regstat_get_setjmp_crosses ();
3648 16239609 : FOR_EACH_BB_FN (bb, cfun)
3649 : {
3650 14722482 : int loop_depth = bb_loop_depth (bb);
3651 :
3652 180522350 : for (insn = BB_HEAD (bb);
3653 180522350 : insn != NEXT_INSN (BB_END (bb));
3654 165799868 : insn = NEXT_INSN (insn))
3655 : {
3656 165799868 : rtx note;
3657 165799868 : rtx set;
3658 165799868 : rtx dest, src;
3659 165799868 : int regno;
3660 :
3661 165799868 : if (! INSN_P (insn))
3662 27185782 : continue;
3663 :
3664 226641589 : for (note = REG_NOTES (insn); note; note = XEXP (note, 1))
3665 88027503 : if (REG_NOTE_KIND (note) == REG_INC)
3666 0 : no_equiv (XEXP (note, 0), note, NULL);
3667 :
3668 138614086 : set = single_set (insn);
3669 :
3670 : /* If this insn contains more (or less) than a single SET,
3671 : only mark all destinations as having no known equivalence. */
3672 199678239 : if (set == NULL_RTX
3673 138614086 : || side_effects_p (SET_SRC (set)))
3674 : {
3675 61064153 : note_pattern_stores (PATTERN (insn), no_equiv, NULL);
3676 61064153 : continue;
3677 : }
3678 77549933 : else if (GET_CODE (PATTERN (insn)) == PARALLEL)
3679 : {
3680 10829853 : int i;
3681 :
3682 32715857 : for (i = XVECLEN (PATTERN (insn), 0) - 1; i >= 0; i--)
3683 : {
3684 21886004 : rtx part = XVECEXP (PATTERN (insn), 0, i);
3685 21886004 : if (part != set)
3686 11056151 : note_pattern_stores (part, no_equiv, NULL);
3687 : }
3688 : }
3689 :
3690 77549933 : dest = SET_DEST (set);
3691 77549933 : src = SET_SRC (set);
3692 :
3693 : /* See if this is setting up the equivalence between an argument
3694 : register and its stack slot. */
3695 77549933 : note = find_reg_note (insn, REG_EQUIV, NULL_RTX);
3696 77549933 : if (note)
3697 : {
3698 231084 : gcc_assert (REG_P (dest));
3699 231084 : regno = REGNO (dest);
3700 :
3701 : /* Note that we don't want to clear init_insns in
3702 : ira_reg_equiv even if there are multiple sets of this
3703 : register. */
3704 231084 : reg_equiv[regno].is_arg_equivalence = 1;
3705 :
3706 : /* The insn result can have equivalence memory although
3707 : the equivalence is not set up by the insn. We add
3708 : this insn to init insns as it is a flag for now that
3709 : regno has an equivalence. We will remove the insn
3710 : from init insn list later. */
3711 231084 : if (rtx_equal_p (src, XEXP (note, 0)) || MEM_P (XEXP (note, 0)))
3712 231084 : ira_reg_equiv[regno].init_insns
3713 231084 : = gen_rtx_INSN_LIST (VOIDmode, insn,
3714 231084 : ira_reg_equiv[regno].init_insns);
3715 :
3716 : /* Continue normally in case this is a candidate for
3717 : replacements. */
3718 : }
3719 :
3720 77549933 : if (!optimize)
3721 23473771 : continue;
3722 :
3723 : /* We only handle the case of a pseudo register being set
3724 : once, or always to the same value. */
3725 : /* ??? The mn10200 port breaks if we add equivalences for
3726 : values that need an ADDRESS_REGS register and set them equivalent
3727 : to a MEM of a pseudo. The actual problem is in the over-conservative
3728 : handling of INPADDR_ADDRESS / INPUT_ADDRESS / INPUT triples in
3729 : calculate_needs, but we traditionally work around this problem
3730 : here by rejecting equivalences when the destination is in a register
3731 : that's likely spilled. This is fragile, of course, since the
3732 : preferred class of a pseudo depends on all instructions that set
3733 : or use it. */
3734 :
3735 89294682 : if (!REG_P (dest)
3736 37591790 : || (regno = REGNO (dest)) < FIRST_PSEUDO_REGISTER
3737 21552584 : || (reg_equiv[regno].init_insns
3738 3552988 : && reg_equiv[regno].init_insns->insn () == NULL)
3739 72933867 : || (targetm.class_likely_spilled_p (reg_preferred_class (regno))
3740 350 : && MEM_P (src) && ! reg_equiv[regno].is_arg_equivalence))
3741 : {
3742 : /* This might be setting a SUBREG of a pseudo, a pseudo that is
3743 : also set somewhere else to a constant. */
3744 35218520 : note_pattern_stores (set, no_equiv, NULL);
3745 35218520 : continue;
3746 : }
3747 :
3748 : /* Don't set reg mentioned in a paradoxical subreg
3749 : equivalent to a mem. */
3750 18857642 : if (MEM_P (src) && reg_equiv[regno].pdx_subregs)
3751 : {
3752 17696 : note_pattern_stores (set, no_equiv, NULL);
3753 17696 : continue;
3754 : }
3755 :
3756 18839946 : note = find_reg_note (insn, REG_EQUAL, NULL_RTX);
3757 :
3758 : /* cse sometimes generates function invariants, but doesn't put a
3759 : REG_EQUAL note on the insn. Since this note would be redundant,
3760 : there's no point creating it earlier than here. */
3761 18839946 : if (! note && ! rtx_varies_p (src, 0))
3762 2869311 : note = set_unique_reg_note (insn, REG_EQUAL, copy_rtx (src));
3763 :
3764 : /* Don't bother considering a REG_EQUAL note containing an EXPR_LIST
3765 : since it represents a function call. */
3766 18839946 : if (note && GET_CODE (XEXP (note, 0)) == EXPR_LIST)
3767 14841015 : note = NULL_RTX;
3768 :
3769 18839946 : if (DF_REG_DEF_COUNT (regno) != 1)
3770 : {
3771 2979550 : bool equal_p = true;
3772 2979550 : rtx_insn_list *list;
3773 :
3774 : /* If we have already processed this pseudo and determined it
3775 : cannot have an equivalence, then honor that decision. */
3776 2979550 : if (reg_equiv[regno].no_equiv)
3777 0 : continue;
3778 :
3779 4848731 : if (! note
3780 1152025 : || rtx_varies_p (XEXP (note, 0), 0)
3781 4089919 : || (reg_equiv[regno].replacement
3782 0 : && ! rtx_equal_p (XEXP (note, 0),
3783 : reg_equiv[regno].replacement)))
3784 : {
3785 1869181 : no_equiv (dest, set, NULL);
3786 1869181 : continue;
3787 : }
3788 :
3789 1110369 : list = reg_equiv[regno].init_insns;
3790 2083628 : for (; list; list = list->next ())
3791 : {
3792 1055021 : rtx note_tmp;
3793 1055021 : rtx_insn *insn_tmp;
3794 :
3795 1055021 : insn_tmp = list->insn ();
3796 1055021 : note_tmp = find_reg_note (insn_tmp, REG_EQUAL, NULL_RTX);
3797 1055021 : gcc_assert (note_tmp);
3798 1055021 : if (! rtx_equal_p (XEXP (note, 0), XEXP (note_tmp, 0)))
3799 : {
3800 : equal_p = false;
3801 : break;
3802 : }
3803 : }
3804 :
3805 1110369 : if (! equal_p)
3806 : {
3807 81762 : no_equiv (dest, set, NULL);
3808 81762 : continue;
3809 : }
3810 : }
3811 :
3812 : /* Record this insn as initializing this register. */
3813 16889003 : reg_equiv[regno].init_insns
3814 16889003 : = gen_rtx_INSN_LIST (VOIDmode, insn, reg_equiv[regno].init_insns);
3815 :
3816 : /* If this register is known to be equal to a constant, record that
3817 : it is always equivalent to the constant.
3818 : Note that it is possible to have a register use before
3819 : the def in loops (see gcc.c-torture/execute/pr79286.c)
3820 : where the reg is undefined on first use. If the def insn
3821 : won't trap we can use it as an equivalence, effectively
3822 : choosing the "undefined" value for the reg to be the
3823 : same as the value set by the def. */
3824 16889003 : if (DF_REG_DEF_COUNT (regno) == 1
3825 15860396 : && note
3826 2846906 : && !rtx_varies_p (XEXP (note, 0), 0)
3827 19302119 : && (!may_trap_or_fault_p (XEXP (note, 0))
3828 50028 : || def_dominates_uses (regno)))
3829 : {
3830 2413115 : rtx note_value = XEXP (note, 0);
3831 2413115 : remove_note (insn, note);
3832 2413115 : set_unique_reg_note (insn, REG_EQUIV, note_value);
3833 : }
3834 :
3835 : /* If this insn introduces a "constant" register, decrease the priority
3836 : of that register. Record this insn if the register is only used once
3837 : more and the equivalence value is the same as our source.
3838 :
3839 : The latter condition is checked for two reasons: First, it is an
3840 : indication that it may be more efficient to actually emit the insn
3841 : as written (if no registers are available, reload will substitute
3842 : the equivalence). Secondly, it avoids problems with any registers
3843 : dying in this insn whose death notes would be missed.
3844 :
3845 : If we don't have a REG_EQUIV note, see if this insn is loading
3846 : a register used only in one basic block from a MEM. If so, and the
3847 : MEM remains unchanged for the life of the register, add a REG_EQUIV
3848 : note. */
3849 16889003 : note = find_reg_note (insn, REG_EQUIV, NULL_RTX);
3850 :
3851 16889003 : rtx replacement = NULL_RTX;
3852 16889003 : if (note)
3853 2636773 : replacement = XEXP (note, 0);
3854 14252230 : else if (REG_BASIC_BLOCK (regno) >= NUM_FIXED_BLOCKS
3855 10199110 : && MEM_P (SET_SRC (set)))
3856 : {
3857 2973791 : enum valid_equiv validity;
3858 2973791 : validity = validate_equiv_mem (insn, dest, SET_SRC (set));
3859 2973791 : if (validity != valid_none)
3860 : {
3861 2186314 : replacement = copy_rtx (SET_SRC (set));
3862 2186314 : if (validity == valid_reload)
3863 : {
3864 2185461 : note = set_unique_reg_note (insn, REG_EQUIV, replacement);
3865 : }
3866 853 : else if (ira_use_lra_p)
3867 : {
3868 : /* We still can use this equivalence for caller save
3869 : optimization in LRA. Mark this. */
3870 853 : ira_reg_equiv[regno].caller_save_p = true;
3871 853 : ira_reg_equiv[regno].init_insns
3872 853 : = gen_rtx_INSN_LIST (VOIDmode, insn,
3873 853 : ira_reg_equiv[regno].init_insns);
3874 : }
3875 : }
3876 : }
3877 :
3878 : /* If we haven't done so, record for reload that this is an
3879 : equivalencing insn. */
3880 16889003 : if (note && !reg_equiv[regno].is_arg_equivalence)
3881 4598576 : ira_reg_equiv[regno].init_insns
3882 4598576 : = gen_rtx_INSN_LIST (VOIDmode, insn,
3883 4598576 : ira_reg_equiv[regno].init_insns);
3884 :
3885 16889003 : if (replacement)
3886 : {
3887 4823087 : reg_equiv[regno].replacement = replacement;
3888 4823087 : reg_equiv[regno].src_p = &SET_SRC (set);
3889 4823087 : reg_equiv[regno].loop_depth = (short) loop_depth;
3890 :
3891 : /* Don't mess with things live during setjmp. */
3892 4823087 : if (optimize && !bitmap_bit_p (setjmp_crosses, regno))
3893 : {
3894 : /* If the register is referenced exactly twice, meaning it is
3895 : set once and used once, indicate that the reference may be
3896 : replaced by the equivalence we computed above. Do this
3897 : even if the register is only used in one block so that
3898 : dependencies can be handled where the last register is
3899 : used in a different block (i.e. HIGH / LO_SUM sequences)
3900 : and to reduce the number of registers alive across
3901 : calls. */
3902 :
3903 4823036 : if (REG_N_REFS (regno) == 2
3904 3787469 : && (rtx_equal_p (replacement, src)
3905 427638 : || ! equiv_init_varies_p (src))
3906 3680892 : && NONJUMP_INSN_P (insn)
3907 8503928 : && equiv_init_movable_p (PATTERN (insn), regno))
3908 2270596 : reg_equiv[regno].replace = 1;
3909 : }
3910 : }
3911 : }
3912 : }
3913 1517127 : }
3914 :
3915 : /* For insns that set a MEM to the contents of a REG that is only used
3916 : in a single basic block, see if the register is always equivalent
3917 : to that memory location and if moving the store from INSN to the
3918 : insn that sets REG is safe. If so, put a REG_EQUIV note on the
3919 : initializing insn. */
3920 : static void
3921 982697 : add_store_equivs (void)
3922 : {
3923 982697 : auto_sbitmap seen_insns (get_max_uid () + 1);
3924 982697 : bitmap_clear (seen_insns);
3925 :
3926 133931272 : for (rtx_insn *insn = get_insns (); insn; insn = NEXT_INSN (insn))
3927 : {
3928 132948575 : rtx set, src, dest;
3929 132948575 : unsigned regno;
3930 132948575 : rtx_insn *init_insn;
3931 :
3932 132948575 : bitmap_set_bit (seen_insns, INSN_UID (insn));
3933 :
3934 132948575 : if (! INSN_P (insn))
3935 24269695 : continue;
3936 :
3937 108678880 : set = single_set (insn);
3938 108678880 : if (! set)
3939 56090727 : continue;
3940 :
3941 52588153 : dest = SET_DEST (set);
3942 52588153 : src = SET_SRC (set);
3943 :
3944 : /* Don't add a REG_EQUIV note if the insn already has one. The existing
3945 : REG_EQUIV is likely more useful than the one we are adding. */
3946 7962936 : if (MEM_P (dest) && REG_P (src)
3947 5290433 : && (regno = REGNO (src)) >= FIRST_PSEUDO_REGISTER
3948 5223693 : && REG_BASIC_BLOCK (regno) >= NUM_FIXED_BLOCKS
3949 2982756 : && DF_REG_DEF_COUNT (regno) == 1
3950 2916563 : && ! reg_equiv[regno].pdx_subregs
3951 2773406 : && reg_equiv[regno].init_insns != NULL
3952 2773406 : && (init_insn = reg_equiv[regno].init_insns->insn ()) != 0
3953 2718606 : && bitmap_bit_p (seen_insns, INSN_UID (init_insn))
3954 2718606 : && ! find_reg_note (init_insn, REG_EQUIV, NULL_RTX)
3955 992167 : && validate_equiv_mem (init_insn, src, dest) == valid_reload
3956 584468 : && ! memref_used_between_p (dest, init_insn, insn)
3957 : /* Attaching a REG_EQUIV note will fail if INIT_INSN has
3958 : multiple sets. */
3959 53058727 : && set_unique_reg_note (init_insn, REG_EQUIV, copy_rtx (dest)))
3960 : {
3961 : /* This insn makes the equivalence, not the one initializing
3962 : the register. */
3963 470071 : ira_reg_equiv[regno].init_insns
3964 470071 : = gen_rtx_INSN_LIST (VOIDmode, insn, NULL_RTX);
3965 470071 : df_notes_rescan (init_insn);
3966 470071 : if (dump_file)
3967 88 : fprintf (dump_file,
3968 : "Adding REG_EQUIV to insn %d for source of insn %d\n",
3969 88 : INSN_UID (init_insn),
3970 88 : INSN_UID (insn));
3971 : }
3972 : }
3973 982697 : }
3974 :
3975 : /* Scan all regs killed in an insn to see if any of them are registers
3976 : only used that once. If so, see if we can replace the reference
3977 : with the equivalent form. If we can, delete the initializing
3978 : reference and this register will go away. If we can't replace the
3979 : reference, and the initializing reference is within the same loop
3980 : (or in an inner loop), then move the register initialization just
3981 : before the use, so that they are in the same basic block. */
3982 : static void
3983 1064910 : combine_and_move_insns (void)
3984 : {
3985 1064910 : auto_bitmap cleared_regs;
3986 1064910 : int max = max_reg_num ();
3987 :
3988 54668181 : for (int regno = FIRST_PSEUDO_REGISTER; regno < max; regno++)
3989 : {
3990 52538361 : if (!reg_equiv[regno].replace)
3991 50267978 : continue;
3992 :
3993 2270383 : rtx_insn *use_insn = 0;
3994 2270383 : bool multiple_insns = false;
3995 2270383 : for (df_ref use = DF_REG_USE_CHAIN (regno);
3996 4561845 : use;
3997 2291462 : use = DF_REF_NEXT_REG (use))
3998 2291462 : if (DF_REF_INSN_INFO (use))
3999 : {
4000 2291462 : if (DEBUG_INSN_P (DF_REF_INSN (use)))
4001 21077 : continue;
4002 2270385 : if (use_insn && DF_REF_INSN (use) != use_insn)
4003 : {
4004 : multiple_insns = true;
4005 : break;
4006 : }
4007 : use_insn = DF_REF_INSN (use);
4008 : }
4009 2270383 : gcc_assert (use_insn);
4010 :
4011 : /* If a register is used by more than one insn, we cannot trivially move
4012 : or delete the definition anymore. */
4013 2270383 : if (multiple_insns)
4014 0 : continue;
4015 :
4016 : /* Don't substitute into jumps. indirect_jump_optimize does
4017 : this for anything we are prepared to handle. */
4018 2270383 : if (JUMP_P (use_insn))
4019 411 : continue;
4020 :
4021 : /* Also don't substitute into a conditional trap insn -- it can become
4022 : an unconditional trap, and that is a flow control insn. */
4023 2269972 : if (GET_CODE (PATTERN (use_insn)) == TRAP_IF)
4024 0 : continue;
4025 :
4026 2269972 : df_ref def = DF_REG_DEF_CHAIN (regno);
4027 2269972 : gcc_assert (DF_REG_DEF_COUNT (regno) == 1 && DF_REF_INSN_INFO (def));
4028 2269972 : rtx_insn *def_insn = DF_REF_INSN (def);
4029 :
4030 : /* We may not move instructions that can throw, since that
4031 : changes basic block boundaries and we are not prepared to
4032 : adjust the CFG to match. */
4033 2269972 : if (can_throw_internal (def_insn))
4034 0 : continue;
4035 :
4036 : /* Instructions with multiple sets can only be moved if DF analysis is
4037 : performed for all of the registers set. See PR91052. */
4038 2269972 : if (multiple_sets (def_insn))
4039 0 : continue;
4040 :
4041 2269972 : basic_block use_bb = BLOCK_FOR_INSN (use_insn);
4042 2269972 : basic_block def_bb = BLOCK_FOR_INSN (def_insn);
4043 2269972 : if (bb_loop_depth (use_bb) > bb_loop_depth (def_bb))
4044 138993 : continue;
4045 :
4046 2130979 : if (asm_noperands (PATTERN (def_insn)) < 0
4047 4261958 : && validate_replace_rtx (regno_reg_rtx[regno],
4048 2130979 : *reg_equiv[regno].src_p, use_insn))
4049 : {
4050 446950 : rtx link;
4051 : /* Append the REG_DEAD notes from def_insn. */
4052 894798 : for (rtx *p = ®_NOTES (def_insn); (link = *p) != 0; )
4053 : {
4054 447848 : if (REG_NOTE_KIND (link) == REG_DEAD)
4055 : {
4056 343 : *p = XEXP (link, 1);
4057 343 : XEXP (link, 1) = REG_NOTES (use_insn);
4058 343 : REG_NOTES (use_insn) = link;
4059 : }
4060 : else
4061 447505 : p = &XEXP (link, 1);
4062 : }
4063 :
4064 446950 : remove_death (regno, use_insn);
4065 446950 : SET_REG_N_REFS (regno, 0);
4066 446950 : REG_FREQ (regno) = 0;
4067 446950 : df_ref use;
4068 529032 : FOR_EACH_INSN_USE (use, def_insn)
4069 : {
4070 82082 : unsigned int use_regno = DF_REF_REGNO (use);
4071 82082 : if (!HARD_REGISTER_NUM_P (use_regno))
4072 1197 : reg_equiv[use_regno].replace = 0;
4073 : }
4074 :
4075 446950 : delete_insn (def_insn);
4076 :
4077 446950 : reg_equiv[regno].init_insns = NULL;
4078 446950 : ira_reg_equiv[regno].init_insns = NULL;
4079 446950 : bitmap_set_bit (cleared_regs, regno);
4080 : }
4081 :
4082 : /* Move the initialization of the register to just before
4083 : USE_INSN. Update the flow information. */
4084 1684029 : else if (prev_nondebug_insn (use_insn) != def_insn)
4085 : {
4086 347087 : rtx_insn *new_insn;
4087 :
4088 347087 : new_insn = emit_insn_before (PATTERN (def_insn), use_insn);
4089 347087 : REG_NOTES (new_insn) = REG_NOTES (def_insn);
4090 347087 : REG_NOTES (def_insn) = 0;
4091 : /* Rescan it to process the notes. */
4092 347087 : df_insn_rescan (new_insn);
4093 :
4094 : /* Make sure this insn is recognized before reload begins,
4095 : otherwise eliminate_regs_in_insn will die. */
4096 347087 : INSN_CODE (new_insn) = INSN_CODE (def_insn);
4097 :
4098 347087 : delete_insn (def_insn);
4099 :
4100 347087 : XEXP (reg_equiv[regno].init_insns, 0) = new_insn;
4101 :
4102 347087 : REG_BASIC_BLOCK (regno) = use_bb->index;
4103 347087 : REG_N_CALLS_CROSSED (regno) = 0;
4104 :
4105 347087 : if (use_insn == BB_HEAD (use_bb))
4106 0 : BB_HEAD (use_bb) = new_insn;
4107 :
4108 : /* We know regno dies in use_insn, but inside a loop
4109 : REG_DEAD notes might be missing when def_insn was in
4110 : another basic block. However, when we move def_insn into
4111 : this bb we'll definitely get a REG_DEAD note and reload
4112 : will see the death. It's possible that update_equiv_regs
4113 : set up an equivalence referencing regno for a reg set by
4114 : use_insn, when regno was seen as non-local. Now that
4115 : regno is local to this block, and dies, such an
4116 : equivalence is invalid. */
4117 347087 : if (find_reg_note (use_insn, REG_EQUIV, regno_reg_rtx[regno]))
4118 : {
4119 0 : rtx set = single_set (use_insn);
4120 0 : if (set && REG_P (SET_DEST (set)))
4121 0 : no_equiv (SET_DEST (set), set, NULL);
4122 : }
4123 :
4124 347087 : ira_reg_equiv[regno].init_insns
4125 347087 : = gen_rtx_INSN_LIST (VOIDmode, new_insn, NULL_RTX);
4126 347087 : bitmap_set_bit (cleared_regs, regno);
4127 : }
4128 : }
4129 :
4130 1064910 : if (!bitmap_empty_p (cleared_regs))
4131 : {
4132 245071 : basic_block bb;
4133 :
4134 6085455 : FOR_EACH_BB_FN (bb, cfun)
4135 : {
4136 11680768 : bitmap_and_compl_into (DF_LR_IN (bb), cleared_regs);
4137 11680768 : bitmap_and_compl_into (DF_LR_OUT (bb), cleared_regs);
4138 5840384 : if (!df_live)
4139 5840384 : continue;
4140 0 : bitmap_and_compl_into (DF_LIVE_IN (bb), cleared_regs);
4141 0 : bitmap_and_compl_into (DF_LIVE_OUT (bb), cleared_regs);
4142 : }
4143 :
4144 : /* Last pass - adjust debug insns referencing cleared regs. */
4145 245071 : if (MAY_HAVE_DEBUG_BIND_INSNS)
4146 66189940 : for (rtx_insn *insn = get_insns (); insn; insn = NEXT_INSN (insn))
4147 99109588 : if (DEBUG_BIND_INSN_P (insn))
4148 : {
4149 25475463 : rtx old_loc = INSN_VAR_LOCATION_LOC (insn);
4150 25475463 : INSN_VAR_LOCATION_LOC (insn)
4151 50950926 : = simplify_replace_fn_rtx (old_loc, NULL_RTX,
4152 : adjust_cleared_regs,
4153 25475463 : (void *) cleared_regs);
4154 25475463 : if (old_loc != INSN_VAR_LOCATION_LOC (insn))
4155 17264 : df_insn_rescan (insn);
4156 : }
4157 : }
4158 1064910 : }
4159 :
4160 : /* A pass over indirect jumps, converting simple cases to direct jumps.
4161 : Combine does this optimization too, but only within a basic block. */
4162 : static void
4163 1517127 : indirect_jump_optimize (void)
4164 : {
4165 1517127 : basic_block bb;
4166 1517127 : bool rebuild_p = false;
4167 :
4168 16239611 : FOR_EACH_BB_REVERSE_FN (bb, cfun)
4169 : {
4170 14722484 : rtx_insn *insn = BB_END (bb);
4171 20530754 : if (!JUMP_P (insn)
4172 14722484 : || find_reg_note (insn, REG_NON_LOCAL_GOTO, NULL_RTX))
4173 5808270 : continue;
4174 :
4175 8914214 : rtx x = pc_set (insn);
4176 8914214 : if (!x || !REG_P (SET_SRC (x)))
4177 8912764 : continue;
4178 :
4179 1450 : int regno = REGNO (SET_SRC (x));
4180 1450 : if (DF_REG_DEF_COUNT (regno) == 1)
4181 : {
4182 1339 : df_ref def = DF_REG_DEF_CHAIN (regno);
4183 1339 : if (!DF_REF_IS_ARTIFICIAL (def))
4184 : {
4185 1339 : rtx_insn *def_insn = DF_REF_INSN (def);
4186 1339 : rtx lab = NULL_RTX;
4187 1339 : rtx set = single_set (def_insn);
4188 1339 : if (set && GET_CODE (SET_SRC (set)) == LABEL_REF)
4189 : lab = SET_SRC (set);
4190 : else
4191 : {
4192 1338 : rtx eqnote = find_reg_note (def_insn, REG_EQUAL, NULL_RTX);
4193 1338 : if (eqnote && GET_CODE (XEXP (eqnote, 0)) == LABEL_REF)
4194 : lab = XEXP (eqnote, 0);
4195 : }
4196 1 : if (lab && validate_replace_rtx (SET_SRC (x), lab, insn))
4197 : rebuild_p = true;
4198 : }
4199 : }
4200 : }
4201 :
4202 1517127 : if (rebuild_p)
4203 : {
4204 1 : timevar_push (TV_JUMP);
4205 1 : rebuild_jump_labels (get_insns ());
4206 1 : if (purge_all_dead_edges ())
4207 1 : delete_unreachable_blocks ();
4208 1 : timevar_pop (TV_JUMP);
4209 : }
4210 1517127 : }
4211 :
4212 : /* Set up fields memory, constant, and invariant from init_insns in
4213 : the structures of array ira_reg_equiv. */
4214 : static void
4215 1517127 : setup_reg_equiv (void)
4216 : {
4217 1517127 : int i;
4218 1517127 : rtx_insn_list *elem, *prev_elem, *next_elem;
4219 1517127 : rtx_insn *insn;
4220 1517127 : rtx set, x;
4221 :
4222 175641487 : for (i = FIRST_PSEUDO_REGISTER; i < ira_reg_equiv_len; i++)
4223 174124360 : for (prev_elem = NULL, elem = ira_reg_equiv[i].init_insns;
4224 178852205 : elem;
4225 4727845 : prev_elem = elem, elem = next_elem)
4226 : {
4227 4853489 : next_elem = elem->next ();
4228 4853489 : insn = elem->insn ();
4229 4853489 : set = single_set (insn);
4230 :
4231 : /* Init insns can set up equivalence when the reg is a destination or
4232 : a source (in this case the destination is memory). */
4233 4853489 : if (set != 0 && (REG_P (SET_DEST (set)) || REG_P (SET_SRC (set))))
4234 : {
4235 4853489 : if ((x = find_reg_note (insn, REG_EQUIV, NULL_RTX)) != NULL)
4236 : {
4237 4382638 : x = XEXP (x, 0);
4238 4382638 : if (REG_P (SET_DEST (set))
4239 4382638 : && REGNO (SET_DEST (set)) == (unsigned int) i
4240 8765276 : && ! rtx_equal_p (SET_SRC (set), x) && MEM_P (x))
4241 : {
4242 : /* This insn reporting the equivalence but
4243 : actually not setting it. Remove it from the
4244 : list. */
4245 29943 : if (prev_elem == NULL)
4246 29943 : ira_reg_equiv[i].init_insns = next_elem;
4247 : else
4248 0 : XEXP (prev_elem, 1) = next_elem;
4249 : elem = prev_elem;
4250 : }
4251 : }
4252 470851 : else if (REG_P (SET_DEST (set))
4253 470851 : && REGNO (SET_DEST (set)) == (unsigned int) i)
4254 780 : x = SET_SRC (set);
4255 : else
4256 : {
4257 470071 : gcc_assert (REG_P (SET_SRC (set))
4258 : && REGNO (SET_SRC (set)) == (unsigned int) i);
4259 : x = SET_DEST (set);
4260 : }
4261 : /* If PIC is enabled and the equiv is not a LEGITIMATE_PIC_OPERAND,
4262 : we can't use it. */
4263 4853489 : if (! CONSTANT_P (x)
4264 1122621 : || ! flag_pic
4265 : /* A function invariant is often CONSTANT_P but may
4266 : include a register. We promise to only pass
4267 : CONSTANT_P objects to LEGITIMATE_PIC_OPERAND_P. */
4268 5008814 : || LEGITIMATE_PIC_OPERAND_P (x))
4269 : {
4270 : /* It can happen that a REG_EQUIV note contains a MEM
4271 : that is not a legitimate memory operand. As later
4272 : stages of reload assume that all addresses found in
4273 : the lra_regno_equiv_* arrays were originally
4274 : legitimate, we ignore such REG_EQUIV notes. */
4275 4808872 : if (memory_operand (x, VOIDmode))
4276 : {
4277 2740959 : ira_reg_equiv[i].defined_p = !ira_reg_equiv[i].caller_save_p;
4278 2740959 : ira_reg_equiv[i].memory = x;
4279 2740959 : continue;
4280 : }
4281 2067913 : else if (function_invariant_p (x))
4282 : {
4283 1987424 : machine_mode mode;
4284 :
4285 1987424 : mode = GET_MODE (SET_DEST (set));
4286 1987424 : if (GET_CODE (x) == PLUS
4287 1079216 : || x == frame_pointer_rtx || x == arg_pointer_rtx)
4288 : /* This is PLUS of frame pointer and a constant,
4289 : or fp, or argp. */
4290 909420 : ira_reg_equiv[i].invariant = x;
4291 1078004 : else if (targetm.legitimate_constant_p (mode, x))
4292 824163 : ira_reg_equiv[i].constant = x;
4293 : else
4294 : {
4295 253841 : ira_reg_equiv[i].memory = force_const_mem (mode, x);
4296 253841 : if (ira_reg_equiv[i].memory == NULL_RTX)
4297 : {
4298 538 : ira_reg_equiv[i].defined_p = false;
4299 538 : ira_reg_equiv[i].caller_save_p = false;
4300 538 : ira_reg_equiv[i].init_insns = NULL;
4301 538 : break;
4302 : }
4303 : }
4304 1986886 : ira_reg_equiv[i].defined_p = true;
4305 1986886 : continue;
4306 1986886 : }
4307 : }
4308 : }
4309 125106 : ira_reg_equiv[i].defined_p = false;
4310 125106 : ira_reg_equiv[i].caller_save_p = false;
4311 125106 : ira_reg_equiv[i].init_insns = NULL;
4312 125106 : break;
4313 : }
4314 1517127 : }
4315 :
4316 :
4317 :
4318 : /* Print chain C to FILE. */
4319 : static void
4320 0 : print_insn_chain (FILE *file, class insn_chain *c)
4321 : {
4322 0 : fprintf (file, "insn=%d, ", INSN_UID (c->insn));
4323 0 : bitmap_print (file, &c->live_throughout, "live_throughout: ", ", ");
4324 0 : bitmap_print (file, &c->dead_or_set, "dead_or_set: ", "\n");
4325 0 : }
4326 :
4327 :
4328 : /* Print all reload_insn_chains to FILE. */
4329 : static void
4330 0 : print_insn_chains (FILE *file)
4331 : {
4332 0 : class insn_chain *c;
4333 0 : for (c = reload_insn_chain; c ; c = c->next)
4334 0 : print_insn_chain (file, c);
4335 0 : }
4336 :
4337 : /* Return true if pseudo REGNO should be added to set live_throughout
4338 : or dead_or_set of the insn chains for reload consideration. */
4339 : static bool
4340 0 : pseudo_for_reload_consideration_p (int regno)
4341 : {
4342 : /* Consider spilled pseudos too for IRA because they still have a
4343 : chance to get hard-registers in the reload when IRA is used. */
4344 0 : return (reg_renumber[regno] >= 0 || ira_conflicts_p);
4345 : }
4346 :
4347 : /* Return true if we can track the individual bytes of subreg X.
4348 : When returning true, set *OUTER_SIZE to the number of bytes in
4349 : X itself, *INNER_SIZE to the number of bytes in the inner register
4350 : and *START to the offset of the first byte. */
4351 : static bool
4352 0 : get_subreg_tracking_sizes (rtx x, HOST_WIDE_INT *outer_size,
4353 : HOST_WIDE_INT *inner_size, HOST_WIDE_INT *start)
4354 : {
4355 0 : rtx reg = regno_reg_rtx[REGNO (SUBREG_REG (x))];
4356 0 : return (GET_MODE_SIZE (GET_MODE (x)).is_constant (outer_size)
4357 0 : && GET_MODE_SIZE (GET_MODE (reg)).is_constant (inner_size)
4358 0 : && SUBREG_BYTE (x).is_constant (start));
4359 : }
4360 :
4361 : /* Init LIVE_SUBREGS[ALLOCNUM] and LIVE_SUBREGS_USED[ALLOCNUM] for
4362 : a register with SIZE bytes, making the register live if INIT_VALUE. */
4363 : static void
4364 0 : init_live_subregs (bool init_value, sbitmap *live_subregs,
4365 : bitmap live_subregs_used, int allocnum, int size)
4366 : {
4367 0 : gcc_assert (size > 0);
4368 :
4369 : /* Been there, done that. */
4370 0 : if (bitmap_bit_p (live_subregs_used, allocnum))
4371 : return;
4372 :
4373 : /* Create a new one. */
4374 0 : if (live_subregs[allocnum] == NULL)
4375 0 : live_subregs[allocnum] = sbitmap_alloc (size);
4376 :
4377 : /* If the entire reg was live before blasting into subregs, we need
4378 : to init all of the subregs to ones else init to 0. */
4379 0 : if (init_value)
4380 0 : bitmap_ones (live_subregs[allocnum]);
4381 : else
4382 0 : bitmap_clear (live_subregs[allocnum]);
4383 :
4384 0 : bitmap_set_bit (live_subregs_used, allocnum);
4385 : }
4386 :
4387 : /* Walk the insns of the current function and build reload_insn_chain,
4388 : and record register life information. */
4389 : static void
4390 0 : build_insn_chain (void)
4391 : {
4392 0 : unsigned int i;
4393 0 : class insn_chain **p = &reload_insn_chain;
4394 0 : basic_block bb;
4395 0 : class insn_chain *c = NULL;
4396 0 : class insn_chain *next = NULL;
4397 0 : auto_bitmap live_relevant_regs;
4398 0 : auto_bitmap elim_regset;
4399 : /* live_subregs is a vector used to keep accurate information about
4400 : which hardregs are live in multiword pseudos. live_subregs and
4401 : live_subregs_used are indexed by pseudo number. The live_subreg
4402 : entry for a particular pseudo is only used if the corresponding
4403 : element is non zero in live_subregs_used. The sbitmap size of
4404 : live_subreg[allocno] is number of bytes that the pseudo can
4405 : occupy. */
4406 0 : sbitmap *live_subregs = XCNEWVEC (sbitmap, max_regno);
4407 0 : auto_bitmap live_subregs_used;
4408 :
4409 0 : hard_reg_set_iterator hrsi;
4410 0 : EXECUTE_IF_SET_IN_HARD_REG_SET (eliminable_regset, 0, i, hrsi)
4411 0 : bitmap_set_bit (elim_regset, i);
4412 0 : FOR_EACH_BB_REVERSE_FN (bb, cfun)
4413 : {
4414 0 : bitmap_iterator bi;
4415 0 : rtx_insn *insn;
4416 :
4417 0 : CLEAR_REG_SET (live_relevant_regs);
4418 0 : bitmap_clear (live_subregs_used);
4419 :
4420 0 : EXECUTE_IF_SET_IN_BITMAP (df_get_live_out (bb), 0, i, bi)
4421 : {
4422 0 : if (i >= FIRST_PSEUDO_REGISTER)
4423 : break;
4424 0 : bitmap_set_bit (live_relevant_regs, i);
4425 : }
4426 :
4427 0 : EXECUTE_IF_SET_IN_BITMAP (df_get_live_out (bb),
4428 : FIRST_PSEUDO_REGISTER, i, bi)
4429 : {
4430 0 : if (pseudo_for_reload_consideration_p (i))
4431 0 : bitmap_set_bit (live_relevant_regs, i);
4432 : }
4433 :
4434 0 : FOR_BB_INSNS_REVERSE (bb, insn)
4435 : {
4436 0 : if (!NOTE_P (insn) && !BARRIER_P (insn))
4437 : {
4438 0 : struct df_insn_info *insn_info = DF_INSN_INFO_GET (insn);
4439 0 : df_ref def, use;
4440 :
4441 0 : c = new_insn_chain ();
4442 0 : c->next = next;
4443 0 : next = c;
4444 0 : *p = c;
4445 0 : p = &c->prev;
4446 :
4447 0 : c->insn = insn;
4448 0 : c->block = bb->index;
4449 :
4450 0 : if (NONDEBUG_INSN_P (insn))
4451 0 : FOR_EACH_INSN_INFO_DEF (def, insn_info)
4452 : {
4453 0 : unsigned int regno = DF_REF_REGNO (def);
4454 :
4455 : /* Ignore may clobbers because these are generated
4456 : from calls. However, every other kind of def is
4457 : added to dead_or_set. */
4458 0 : if (!DF_REF_FLAGS_IS_SET (def, DF_REF_MAY_CLOBBER))
4459 : {
4460 0 : if (regno < FIRST_PSEUDO_REGISTER)
4461 : {
4462 0 : if (!fixed_regs[regno])
4463 0 : bitmap_set_bit (&c->dead_or_set, regno);
4464 : }
4465 0 : else if (pseudo_for_reload_consideration_p (regno))
4466 0 : bitmap_set_bit (&c->dead_or_set, regno);
4467 : }
4468 :
4469 0 : if ((regno < FIRST_PSEUDO_REGISTER
4470 0 : || reg_renumber[regno] >= 0
4471 0 : || ira_conflicts_p)
4472 0 : && (!DF_REF_FLAGS_IS_SET (def, DF_REF_CONDITIONAL)))
4473 : {
4474 0 : rtx reg = DF_REF_REG (def);
4475 0 : HOST_WIDE_INT outer_size, inner_size, start;
4476 :
4477 : /* We can usually track the liveness of individual
4478 : bytes within a subreg. The only exceptions are
4479 : subregs wrapped in ZERO_EXTRACTs and subregs whose
4480 : size is not known; in those cases we need to be
4481 : conservative and treat the definition as a partial
4482 : definition of the full register rather than a full
4483 : definition of a specific part of the register. */
4484 0 : if (GET_CODE (reg) == SUBREG
4485 0 : && !DF_REF_FLAGS_IS_SET (def, DF_REF_ZERO_EXTRACT)
4486 0 : && get_subreg_tracking_sizes (reg, &outer_size,
4487 : &inner_size, &start))
4488 : {
4489 0 : HOST_WIDE_INT last = start + outer_size;
4490 :
4491 0 : init_live_subregs
4492 0 : (bitmap_bit_p (live_relevant_regs, regno),
4493 : live_subregs, live_subregs_used, regno,
4494 : inner_size);
4495 :
4496 0 : if (!DF_REF_FLAGS_IS_SET
4497 : (def, DF_REF_STRICT_LOW_PART))
4498 : {
4499 : /* Expand the range to cover entire words.
4500 : Bytes added here are "don't care". */
4501 0 : start
4502 0 : = start / UNITS_PER_WORD * UNITS_PER_WORD;
4503 0 : last = ((last + UNITS_PER_WORD - 1)
4504 0 : / UNITS_PER_WORD * UNITS_PER_WORD);
4505 : }
4506 :
4507 : /* Ignore the paradoxical bits. */
4508 0 : if (last > SBITMAP_SIZE (live_subregs[regno]))
4509 : last = SBITMAP_SIZE (live_subregs[regno]);
4510 :
4511 0 : while (start < last)
4512 : {
4513 0 : bitmap_clear_bit (live_subregs[regno], start);
4514 0 : start++;
4515 : }
4516 :
4517 0 : if (bitmap_empty_p (live_subregs[regno]))
4518 : {
4519 0 : bitmap_clear_bit (live_subregs_used, regno);
4520 0 : bitmap_clear_bit (live_relevant_regs, regno);
4521 : }
4522 : else
4523 : /* Set live_relevant_regs here because
4524 : that bit has to be true to get us to
4525 : look at the live_subregs fields. */
4526 0 : bitmap_set_bit (live_relevant_regs, regno);
4527 : }
4528 : else
4529 : {
4530 : /* DF_REF_PARTIAL is generated for
4531 : subregs, STRICT_LOW_PART, and
4532 : ZERO_EXTRACT. We handle the subreg
4533 : case above so here we have to keep from
4534 : modeling the def as a killing def. */
4535 0 : if (!DF_REF_FLAGS_IS_SET (def, DF_REF_PARTIAL))
4536 : {
4537 0 : bitmap_clear_bit (live_subregs_used, regno);
4538 0 : bitmap_clear_bit (live_relevant_regs, regno);
4539 : }
4540 : }
4541 : }
4542 : }
4543 :
4544 0 : bitmap_and_compl_into (live_relevant_regs, elim_regset);
4545 0 : bitmap_copy (&c->live_throughout, live_relevant_regs);
4546 :
4547 0 : if (NONDEBUG_INSN_P (insn))
4548 0 : FOR_EACH_INSN_INFO_USE (use, insn_info)
4549 : {
4550 0 : unsigned int regno = DF_REF_REGNO (use);
4551 0 : rtx reg = DF_REF_REG (use);
4552 :
4553 : /* DF_REF_READ_WRITE on a use means that this use
4554 : is fabricated from a def that is a partial set
4555 : to a multiword reg. Here, we only model the
4556 : subreg case that is not wrapped in ZERO_EXTRACT
4557 : precisely so we do not need to look at the
4558 : fabricated use. */
4559 0 : if (DF_REF_FLAGS_IS_SET (use, DF_REF_READ_WRITE)
4560 0 : && !DF_REF_FLAGS_IS_SET (use, DF_REF_ZERO_EXTRACT)
4561 0 : && DF_REF_FLAGS_IS_SET (use, DF_REF_SUBREG))
4562 0 : continue;
4563 :
4564 : /* Add the last use of each var to dead_or_set. */
4565 0 : if (!bitmap_bit_p (live_relevant_regs, regno))
4566 : {
4567 0 : if (regno < FIRST_PSEUDO_REGISTER)
4568 : {
4569 0 : if (!fixed_regs[regno])
4570 0 : bitmap_set_bit (&c->dead_or_set, regno);
4571 : }
4572 0 : else if (pseudo_for_reload_consideration_p (regno))
4573 0 : bitmap_set_bit (&c->dead_or_set, regno);
4574 : }
4575 :
4576 0 : if (regno < FIRST_PSEUDO_REGISTER
4577 0 : || pseudo_for_reload_consideration_p (regno))
4578 : {
4579 0 : HOST_WIDE_INT outer_size, inner_size, start;
4580 0 : if (GET_CODE (reg) == SUBREG
4581 0 : && !DF_REF_FLAGS_IS_SET (use,
4582 : DF_REF_SIGN_EXTRACT
4583 : | DF_REF_ZERO_EXTRACT)
4584 0 : && get_subreg_tracking_sizes (reg, &outer_size,
4585 : &inner_size, &start))
4586 : {
4587 0 : HOST_WIDE_INT last = start + outer_size;
4588 :
4589 0 : init_live_subregs
4590 0 : (bitmap_bit_p (live_relevant_regs, regno),
4591 : live_subregs, live_subregs_used, regno,
4592 : inner_size);
4593 :
4594 : /* Ignore the paradoxical bits. */
4595 0 : if (last > SBITMAP_SIZE (live_subregs[regno]))
4596 : last = SBITMAP_SIZE (live_subregs[regno]);
4597 :
4598 0 : while (start < last)
4599 : {
4600 0 : bitmap_set_bit (live_subregs[regno], start);
4601 0 : start++;
4602 : }
4603 : }
4604 : else
4605 : /* Resetting the live_subregs_used is
4606 : effectively saying do not use the subregs
4607 : because we are reading the whole
4608 : pseudo. */
4609 0 : bitmap_clear_bit (live_subregs_used, regno);
4610 0 : bitmap_set_bit (live_relevant_regs, regno);
4611 : }
4612 : }
4613 : }
4614 : }
4615 :
4616 : /* FIXME!! The following code is a disaster. Reload needs to see the
4617 : labels and jump tables that are just hanging out in between
4618 : the basic blocks. See pr33676. */
4619 0 : insn = BB_HEAD (bb);
4620 :
4621 : /* Skip over the barriers and cruft. */
4622 0 : while (insn && (BARRIER_P (insn) || NOTE_P (insn)
4623 0 : || BLOCK_FOR_INSN (insn) == bb))
4624 0 : insn = PREV_INSN (insn);
4625 :
4626 : /* While we add anything except barriers and notes, the focus is
4627 : to get the labels and jump tables into the
4628 : reload_insn_chain. */
4629 0 : while (insn)
4630 : {
4631 0 : if (!NOTE_P (insn) && !BARRIER_P (insn))
4632 : {
4633 0 : if (BLOCK_FOR_INSN (insn))
4634 : break;
4635 :
4636 0 : c = new_insn_chain ();
4637 0 : c->next = next;
4638 0 : next = c;
4639 0 : *p = c;
4640 0 : p = &c->prev;
4641 :
4642 : /* The block makes no sense here, but it is what the old
4643 : code did. */
4644 0 : c->block = bb->index;
4645 0 : c->insn = insn;
4646 0 : bitmap_copy (&c->live_throughout, live_relevant_regs);
4647 : }
4648 0 : insn = PREV_INSN (insn);
4649 : }
4650 : }
4651 :
4652 0 : reload_insn_chain = c;
4653 0 : *p = NULL;
4654 :
4655 0 : for (i = 0; i < (unsigned int) max_regno; i++)
4656 0 : if (live_subregs[i] != NULL)
4657 0 : sbitmap_free (live_subregs[i]);
4658 0 : free (live_subregs);
4659 :
4660 0 : if (dump_file)
4661 0 : print_insn_chains (dump_file);
4662 0 : }
4663 :
4664 : /* Examine the rtx found in *LOC, which is read or written to as determined
4665 : by TYPE. Return false if we find a reason why an insn containing this
4666 : rtx should not be moved (such as accesses to non-constant memory), true
4667 : otherwise. */
4668 : static bool
4669 6965952 : rtx_moveable_p (rtx *loc, enum op_type type)
4670 : {
4671 6975521 : const char *fmt;
4672 6975521 : rtx x = *loc;
4673 6975521 : int i, j;
4674 :
4675 6975521 : enum rtx_code code = GET_CODE (x);
4676 6975521 : switch (code)
4677 : {
4678 : case CONST:
4679 : CASE_CONST_ANY:
4680 : case SYMBOL_REF:
4681 : case LABEL_REF:
4682 : return true;
4683 :
4684 0 : case PC:
4685 0 : return type == OP_IN;
4686 :
4687 2421352 : case REG:
4688 2421352 : if (x == frame_pointer_rtx)
4689 : return true;
4690 2420963 : if (HARD_REGISTER_P (x))
4691 : return false;
4692 :
4693 : return true;
4694 :
4695 662155 : case MEM:
4696 662155 : if (type == OP_IN && MEM_READONLY_P (x))
4697 9494 : return rtx_moveable_p (&XEXP (x, 0), OP_IN);
4698 : return false;
4699 :
4700 2136003 : case SET:
4701 2136003 : return (rtx_moveable_p (&SET_SRC (x), OP_IN)
4702 2136003 : && rtx_moveable_p (&SET_DEST (x), OP_OUT));
4703 :
4704 10 : case STRICT_LOW_PART:
4705 10 : return rtx_moveable_p (&XEXP (x, 0), OP_OUT);
4706 :
4707 491 : case ZERO_EXTRACT:
4708 491 : case SIGN_EXTRACT:
4709 491 : return (rtx_moveable_p (&XEXP (x, 0), type)
4710 491 : && rtx_moveable_p (&XEXP (x, 1), OP_IN)
4711 982 : && rtx_moveable_p (&XEXP (x, 2), OP_IN));
4712 :
4713 65 : case CLOBBER:
4714 65 : return rtx_moveable_p (&SET_DEST (x), OP_OUT);
4715 :
4716 : case UNSPEC_VOLATILE:
4717 : /* It is a bad idea to consider insns with such rtl
4718 : as moveable ones. The insn scheduler also considers them as barrier
4719 : for a reason. */
4720 : return false;
4721 :
4722 0 : case ASM_OPERANDS:
4723 : /* The same is true for volatile asm: it has unknown side effects, it
4724 : cannot be moved at will. */
4725 0 : if (MEM_VOLATILE_P (x))
4726 : return false;
4727 :
4728 1150808 : default:
4729 1150808 : break;
4730 : }
4731 :
4732 1150808 : fmt = GET_RTX_FORMAT (code);
4733 2964058 : for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
4734 : {
4735 2004003 : if (fmt[i] == 'e')
4736 : {
4737 1597173 : if (!rtx_moveable_p (&XEXP (x, i), type))
4738 : return false;
4739 : }
4740 406830 : else if (fmt[i] == 'E')
4741 560527 : for (j = XVECLEN (x, i) - 1; j >= 0; j--)
4742 : {
4743 430685 : if (!rtx_moveable_p (&XVECEXP (x, i, j), type))
4744 : return false;
4745 : }
4746 : }
4747 : return true;
4748 : }
4749 :
4750 : /* A wrapper around dominated_by_p, which uses the information in UID_LUID
4751 : to give dominance relationships between two insns I1 and I2. */
4752 : static bool
4753 21988001 : insn_dominated_by_p (rtx i1, rtx i2, int *uid_luid)
4754 : {
4755 21988001 : basic_block bb1 = BLOCK_FOR_INSN (i1);
4756 21988001 : basic_block bb2 = BLOCK_FOR_INSN (i2);
4757 :
4758 21988001 : if (bb1 == bb2)
4759 11806141 : return uid_luid[INSN_UID (i2)] < uid_luid[INSN_UID (i1)];
4760 10181860 : return dominated_by_p (CDI_DOMINATORS, bb1, bb2);
4761 : }
4762 :
4763 : /* Record the range of register numbers added by find_moveable_pseudos. */
4764 : int first_moveable_pseudo, last_moveable_pseudo;
4765 :
4766 : /* These two vectors hold data for every register added by
4767 : find_movable_pseudos, with index 0 holding data for the
4768 : first_moveable_pseudo. */
4769 : /* The original home register. */
4770 : static vec<rtx> pseudo_replaced_reg;
4771 :
4772 : /* Look for instances where we have an instruction that is known to increase
4773 : register pressure, and whose result is not used immediately. If it is
4774 : possible to move the instruction downwards to just before its first use,
4775 : split its lifetime into two ranges. We create a new pseudo to compute the
4776 : value, and emit a move instruction just before the first use. If, after
4777 : register allocation, the new pseudo remains unallocated, the function
4778 : move_unallocated_pseudos then deletes the move instruction and places
4779 : the computation just before the first use.
4780 :
4781 : Such a move is safe and profitable if all the input registers remain live
4782 : and unchanged between the original computation and its first use. In such
4783 : a situation, the computation is known to increase register pressure, and
4784 : moving it is known to at least not worsen it.
4785 :
4786 : We restrict moves to only those cases where a register remains unallocated,
4787 : in order to avoid interfering too much with the instruction schedule. As
4788 : an exception, we may move insns which only modify their input register
4789 : (typically induction variables), as this increases the freedom for our
4790 : intended transformation, and does not limit the second instruction
4791 : scheduler pass. */
4792 :
4793 : static void
4794 1064963 : find_moveable_pseudos (void)
4795 : {
4796 1064963 : unsigned i;
4797 1064963 : int max_regs = max_reg_num ();
4798 1064963 : int max_uid = get_max_uid ();
4799 1064963 : basic_block bb;
4800 1064963 : int *uid_luid = XNEWVEC (int, max_uid);
4801 1064963 : rtx_insn **closest_uses = XNEWVEC (rtx_insn *, max_regs);
4802 : /* A set of registers which are live but not modified throughout a block. */
4803 1064963 : bitmap_head *bb_transp_live = XNEWVEC (bitmap_head,
4804 : last_basic_block_for_fn (cfun));
4805 : /* A set of registers which only exist in a given basic block. */
4806 1064963 : bitmap_head *bb_local = XNEWVEC (bitmap_head,
4807 : last_basic_block_for_fn (cfun));
4808 : /* A set of registers which are set once, in an instruction that can be
4809 : moved freely downwards, but are otherwise transparent to a block. */
4810 1064963 : bitmap_head *bb_moveable_reg_sets = XNEWVEC (bitmap_head,
4811 : last_basic_block_for_fn (cfun));
4812 1064963 : auto_bitmap live, used, set, interesting, unusable_as_input;
4813 1064963 : bitmap_iterator bi;
4814 :
4815 1064963 : first_moveable_pseudo = max_regs;
4816 1064963 : pseudo_replaced_reg.release ();
4817 1064963 : pseudo_replaced_reg.safe_grow_cleared (max_regs, true);
4818 :
4819 1064963 : df_analyze ();
4820 1064963 : calculate_dominance_info (CDI_DOMINATORS);
4821 :
4822 1064963 : i = 0;
4823 12272738 : FOR_EACH_BB_FN (bb, cfun)
4824 : {
4825 11207775 : rtx_insn *insn;
4826 11207775 : bitmap transp = bb_transp_live + bb->index;
4827 11207775 : bitmap moveable = bb_moveable_reg_sets + bb->index;
4828 11207775 : bitmap local = bb_local + bb->index;
4829 :
4830 11207775 : bitmap_initialize (local, 0);
4831 11207775 : bitmap_initialize (transp, 0);
4832 11207775 : bitmap_initialize (moveable, 0);
4833 11207775 : bitmap_copy (live, df_get_live_out (bb));
4834 11207775 : bitmap_and_into (live, df_get_live_in (bb));
4835 11207775 : bitmap_copy (transp, live);
4836 11207775 : bitmap_clear (moveable);
4837 11207775 : bitmap_clear (live);
4838 11207775 : bitmap_clear (used);
4839 11207775 : bitmap_clear (set);
4840 145543079 : FOR_BB_INSNS (bb, insn)
4841 134335304 : if (NONDEBUG_INSN_P (insn))
4842 : {
4843 60108553 : df_insn_info *insn_info = DF_INSN_INFO_GET (insn);
4844 60108553 : df_ref def, use;
4845 :
4846 60108553 : uid_luid[INSN_UID (insn)] = i++;
4847 :
4848 60108553 : def = df_single_def (insn_info);
4849 60108553 : use = df_single_use (insn_info);
4850 60108553 : if (use
4851 60108553 : && def
4852 19836624 : && DF_REF_REGNO (use) == DF_REF_REGNO (def)
4853 632017 : && !bitmap_bit_p (set, DF_REF_REGNO (use))
4854 60195418 : && rtx_moveable_p (&PATTERN (insn), OP_IN))
4855 : {
4856 36521 : unsigned regno = DF_REF_REGNO (use);
4857 36521 : bitmap_set_bit (moveable, regno);
4858 36521 : bitmap_set_bit (set, regno);
4859 36521 : bitmap_set_bit (used, regno);
4860 36521 : bitmap_clear_bit (transp, regno);
4861 36521 : continue;
4862 36521 : }
4863 134575730 : FOR_EACH_INSN_INFO_USE (use, insn_info)
4864 : {
4865 74503698 : unsigned regno = DF_REF_REGNO (use);
4866 74503698 : bitmap_set_bit (used, regno);
4867 74503698 : if (bitmap_clear_bit (moveable, regno))
4868 15766 : bitmap_clear_bit (transp, regno);
4869 : }
4870 :
4871 509585647 : FOR_EACH_INSN_INFO_DEF (def, insn_info)
4872 : {
4873 449513615 : unsigned regno = DF_REF_REGNO (def);
4874 449513615 : bitmap_set_bit (set, regno);
4875 449513615 : bitmap_clear_bit (transp, regno);
4876 449513615 : bitmap_clear_bit (moveable, regno);
4877 : }
4878 : }
4879 : }
4880 :
4881 12272738 : FOR_EACH_BB_FN (bb, cfun)
4882 : {
4883 11207775 : bitmap local = bb_local + bb->index;
4884 11207775 : rtx_insn *insn;
4885 :
4886 145543079 : FOR_BB_INSNS (bb, insn)
4887 134335304 : if (NONDEBUG_INSN_P (insn))
4888 : {
4889 60108553 : df_insn_info *insn_info = DF_INSN_INFO_GET (insn);
4890 60108553 : rtx_insn *def_insn;
4891 60108553 : rtx closest_use, note;
4892 60108553 : df_ref def, use;
4893 60108553 : unsigned regno;
4894 60108553 : bool all_dominated, all_local;
4895 60108553 : machine_mode mode;
4896 :
4897 60108553 : def = df_single_def (insn_info);
4898 : /* There must be exactly one def in this insn. */
4899 32807670 : if (!def || !single_set (insn))
4900 27378337 : continue;
4901 : /* This must be the only definition of the reg. We also limit
4902 : which modes we deal with so that we can assume we can generate
4903 : move instructions. */
4904 32730216 : regno = DF_REF_REGNO (def);
4905 32730216 : mode = GET_MODE (DF_REF_REG (def));
4906 32730216 : if (DF_REG_DEF_COUNT (regno) != 1
4907 12428621 : || !DF_REF_INSN_INFO (def)
4908 12428621 : || HARD_REGISTER_NUM_P (regno)
4909 12394581 : || DF_REG_EQ_USE_COUNT (regno) > 0
4910 11881784 : || (!INTEGRAL_MODE_P (mode)
4911 : && !FLOAT_MODE_P (mode)
4912 : && !OPAQUE_MODE_P (mode)))
4913 20848432 : continue;
4914 11881784 : def_insn = DF_REF_INSN (def);
4915 :
4916 20206165 : for (note = REG_NOTES (def_insn); note; note = XEXP (note, 1))
4917 10869216 : if (REG_NOTE_KIND (note) == REG_EQUIV && MEM_P (XEXP (note, 0)))
4918 : break;
4919 :
4920 11881784 : if (note)
4921 : {
4922 2544835 : if (dump_file)
4923 68 : fprintf (dump_file, "Ignoring reg %d, has equiv memory\n",
4924 : regno);
4925 2544835 : bitmap_set_bit (unusable_as_input, regno);
4926 2544835 : continue;
4927 : }
4928 :
4929 9336949 : use = DF_REG_USE_CHAIN (regno);
4930 9336949 : all_dominated = true;
4931 9336949 : all_local = true;
4932 9336949 : closest_use = NULL_RTX;
4933 27584266 : for (; use; use = DF_REF_NEXT_REG (use))
4934 : {
4935 18247317 : rtx_insn *insn;
4936 18247317 : if (!DF_REF_INSN_INFO (use))
4937 : {
4938 : all_dominated = false;
4939 : all_local = false;
4940 : break;
4941 : }
4942 18247317 : insn = DF_REF_INSN (use);
4943 18247317 : if (DEBUG_INSN_P (insn))
4944 2346928 : continue;
4945 15900389 : if (BLOCK_FOR_INSN (insn) != BLOCK_FOR_INSN (def_insn))
4946 6216316 : all_local = false;
4947 15900389 : if (!insn_dominated_by_p (insn, def_insn, uid_luid))
4948 9974 : all_dominated = false;
4949 15900389 : if (closest_use != insn && closest_use != const0_rtx)
4950 : {
4951 14014840 : if (closest_use == NULL_RTX)
4952 : closest_use = insn;
4953 4739817 : else if (insn_dominated_by_p (closest_use, insn, uid_luid))
4954 : closest_use = insn;
4955 1347795 : else if (!insn_dominated_by_p (insn, closest_use, uid_luid))
4956 642915 : closest_use = const0_rtx;
4957 : }
4958 : }
4959 9336949 : if (!all_dominated)
4960 : {
4961 4958 : if (dump_file)
4962 0 : fprintf (dump_file, "Reg %d not all uses dominated by set\n",
4963 : regno);
4964 4958 : continue;
4965 : }
4966 9331991 : if (all_local)
4967 6683274 : bitmap_set_bit (local, regno);
4968 8690866 : if (closest_use == const0_rtx || closest_use == NULL
4969 17960931 : || next_nonnote_nondebug_insn (def_insn) == closest_use)
4970 : {
4971 5896769 : if (dump_file)
4972 95 : fprintf (dump_file, "Reg %d uninteresting%s\n", regno,
4973 95 : closest_use == const0_rtx || closest_use == NULL
4974 : ? " (no unique first use)" : "");
4975 5896769 : continue;
4976 : }
4977 :
4978 3435222 : bitmap_set_bit (interesting, regno);
4979 : /* If we get here, we know closest_use is a non-NULL insn
4980 : (as opposed to const_0_rtx). */
4981 3435222 : closest_uses[regno] = as_a <rtx_insn *> (closest_use);
4982 :
4983 3435222 : if (dump_file && (all_local || all_dominated))
4984 : {
4985 78 : fprintf (dump_file, "Reg %u:", regno);
4986 78 : if (all_local)
4987 14 : fprintf (dump_file, " local to bb %d", bb->index);
4988 78 : if (all_dominated)
4989 78 : fprintf (dump_file, " def dominates all uses");
4990 78 : if (closest_use != const0_rtx)
4991 78 : fprintf (dump_file, " has unique first use");
4992 78 : fputs ("\n", dump_file);
4993 : }
4994 : }
4995 : }
4996 :
4997 4500185 : EXECUTE_IF_SET_IN_BITMAP (interesting, 0, i, bi)
4998 : {
4999 3435222 : df_ref def = DF_REG_DEF_CHAIN (i);
5000 3435222 : rtx_insn *def_insn = DF_REF_INSN (def);
5001 3435222 : basic_block def_block = BLOCK_FOR_INSN (def_insn);
5002 3435222 : bitmap def_bb_local = bb_local + def_block->index;
5003 3435222 : bitmap def_bb_moveable = bb_moveable_reg_sets + def_block->index;
5004 3435222 : bitmap def_bb_transp = bb_transp_live + def_block->index;
5005 3435222 : bool local_to_bb_p = bitmap_bit_p (def_bb_local, i);
5006 3435222 : rtx_insn *use_insn = closest_uses[i];
5007 3435222 : df_ref use;
5008 3435222 : bool all_ok = true;
5009 3435222 : bool all_transp = true;
5010 :
5011 3435222 : if (!REG_P (DF_REF_REG (def)))
5012 47831 : continue;
5013 :
5014 3387391 : if (!local_to_bb_p)
5015 : {
5016 1281321 : if (dump_file)
5017 64 : fprintf (dump_file, "Reg %u not local to one basic block\n",
5018 : i);
5019 1281321 : continue;
5020 : }
5021 2106070 : if (reg_equiv_init (i) != NULL_RTX)
5022 : {
5023 56903 : if (dump_file)
5024 0 : fprintf (dump_file, "Ignoring reg %u with equiv init insn\n",
5025 : i);
5026 56903 : continue;
5027 : }
5028 2049167 : if (!rtx_moveable_p (&PATTERN (def_insn), OP_IN))
5029 : {
5030 1435808 : if (dump_file)
5031 14 : fprintf (dump_file, "Found def insn %d for %d to be not moveable\n",
5032 14 : INSN_UID (def_insn), i);
5033 1435808 : continue;
5034 : }
5035 613359 : if (dump_file)
5036 0 : fprintf (dump_file, "Examining insn %d, def for %d\n",
5037 0 : INSN_UID (def_insn), i);
5038 1379654 : FOR_EACH_INSN_USE (use, def_insn)
5039 : {
5040 825231 : unsigned regno = DF_REF_REGNO (use);
5041 825231 : if (bitmap_bit_p (unusable_as_input, regno))
5042 : {
5043 58936 : all_ok = false;
5044 58936 : if (dump_file)
5045 0 : fprintf (dump_file, " found unusable input reg %u.\n", regno);
5046 : break;
5047 : }
5048 766295 : if (!bitmap_bit_p (def_bb_transp, regno))
5049 : {
5050 715921 : if (bitmap_bit_p (def_bb_moveable, regno)
5051 715921 : && !control_flow_insn_p (use_insn))
5052 : {
5053 31 : if (modified_between_p (DF_REF_REG (use), def_insn, use_insn))
5054 : {
5055 1 : rtx_insn *x = NEXT_INSN (def_insn);
5056 3 : while (!modified_in_p (DF_REF_REG (use), x))
5057 : {
5058 1 : gcc_assert (x != use_insn);
5059 1 : x = NEXT_INSN (x);
5060 : }
5061 1 : if (dump_file)
5062 0 : fprintf (dump_file, " input reg %u modified but insn %d moveable\n",
5063 0 : regno, INSN_UID (x));
5064 1 : emit_insn_after (PATTERN (x), use_insn);
5065 1 : set_insn_deleted (x);
5066 : }
5067 : else
5068 : {
5069 30 : if (dump_file)
5070 0 : fprintf (dump_file, " input reg %u modified between def and use\n",
5071 : regno);
5072 : all_transp = false;
5073 : }
5074 : }
5075 : else
5076 : all_transp = false;
5077 : }
5078 : }
5079 0 : if (!all_ok)
5080 58936 : continue;
5081 554423 : if (!dbg_cnt (ira_move))
5082 : break;
5083 554423 : if (dump_file)
5084 0 : fprintf (dump_file, " all ok%s\n", all_transp ? " and transp" : "");
5085 :
5086 554423 : if (all_transp)
5087 : {
5088 14498 : rtx def_reg = DF_REF_REG (def);
5089 14498 : rtx newreg = ira_create_new_reg (def_reg);
5090 14498 : if (validate_change (def_insn, DF_REF_REAL_LOC (def), newreg, 0))
5091 : {
5092 14498 : unsigned nregno = REGNO (newreg);
5093 14498 : emit_insn_before (gen_move_insn (def_reg, newreg), use_insn);
5094 14498 : nregno -= max_regs;
5095 14498 : pseudo_replaced_reg[nregno] = def_reg;
5096 : }
5097 : }
5098 : }
5099 :
5100 12272738 : FOR_EACH_BB_FN (bb, cfun)
5101 : {
5102 11207775 : bitmap_clear (bb_local + bb->index);
5103 11207775 : bitmap_clear (bb_transp_live + bb->index);
5104 11207775 : bitmap_clear (bb_moveable_reg_sets + bb->index);
5105 : }
5106 1064963 : free (uid_luid);
5107 1064963 : free (closest_uses);
5108 1064963 : free (bb_local);
5109 1064963 : free (bb_transp_live);
5110 1064963 : free (bb_moveable_reg_sets);
5111 :
5112 1064963 : last_moveable_pseudo = max_reg_num ();
5113 :
5114 1064963 : fix_reg_equiv_init ();
5115 1064963 : expand_reg_info ();
5116 1064963 : regstat_free_n_sets_and_refs ();
5117 1064963 : regstat_free_ri ();
5118 1064963 : regstat_init_n_sets_and_refs ();
5119 1064963 : regstat_compute_ri ();
5120 1064963 : free_dominance_info (CDI_DOMINATORS);
5121 1064963 : }
5122 :
5123 : /* If SET pattern SET is an assignment from a hard register to a pseudo which
5124 : is live at CALL_DOM (if non-NULL, otherwise this check is omitted), return
5125 : the destination. Otherwise return NULL. */
5126 :
5127 : static rtx
5128 2166192 : interesting_dest_for_shprep_1 (rtx set, basic_block call_dom)
5129 : {
5130 2166192 : rtx src = SET_SRC (set);
5131 2166192 : rtx dest = SET_DEST (set);
5132 727537 : if (!REG_P (src) || !HARD_REGISTER_P (src)
5133 564534 : || !REG_P (dest) || HARD_REGISTER_P (dest)
5134 2709796 : || (call_dom && !bitmap_bit_p (df_get_live_in (call_dom), REGNO (dest))))
5135 1713232 : return NULL;
5136 : return dest;
5137 : }
5138 :
5139 : /* If insn is interesting for parameter range-splitting shrink-wrapping
5140 : preparation, i.e. it is a single set from a hard register to a pseudo, which
5141 : is live at CALL_DOM (if non-NULL, otherwise this check is omitted), or a
5142 : parallel statement with only one such statement, return the destination.
5143 : Otherwise return NULL. */
5144 :
5145 : static rtx
5146 4527968 : interesting_dest_for_shprep (rtx_insn *insn, basic_block call_dom)
5147 : {
5148 4527968 : if (!INSN_P (insn))
5149 : return NULL;
5150 3644655 : rtx pat = PATTERN (insn);
5151 3644655 : if (GET_CODE (pat) == SET)
5152 1923309 : return interesting_dest_for_shprep_1 (pat, call_dom);
5153 :
5154 1721346 : if (GET_CODE (pat) != PARALLEL)
5155 : return NULL;
5156 : rtx ret = NULL;
5157 634767 : for (int i = 0; i < XVECLEN (pat, 0); i++)
5158 : {
5159 429082 : rtx sub = XVECEXP (pat, 0, i);
5160 429082 : if (GET_CODE (sub) == USE || GET_CODE (sub) == CLOBBER)
5161 178262 : continue;
5162 250820 : if (GET_CODE (sub) != SET
5163 250820 : || side_effects_p (sub))
5164 : return NULL;
5165 242883 : rtx dest = interesting_dest_for_shprep_1 (sub, call_dom);
5166 242883 : if (dest && ret)
5167 : return NULL;
5168 242883 : if (dest)
5169 421145 : ret = dest;
5170 : }
5171 : return ret;
5172 : }
5173 :
5174 : /* Split live ranges of pseudos that are loaded from hard registers in the
5175 : first BB in a BB that dominates all non-sibling call if such a BB can be
5176 : found and is not in a loop. Return true if the function has made any
5177 : changes. */
5178 :
5179 : static bool
5180 1064963 : split_live_ranges_for_shrink_wrap (void)
5181 : {
5182 1064963 : basic_block bb, call_dom = NULL;
5183 1064963 : basic_block first = single_succ (ENTRY_BLOCK_PTR_FOR_FN (cfun));
5184 1064963 : rtx_insn *insn, *last_interesting_insn = NULL;
5185 1064963 : auto_bitmap need_new, reachable;
5186 1064963 : vec<basic_block> queue;
5187 :
5188 1064963 : if (!SHRINK_WRAPPING_ENABLED)
5189 : return false;
5190 :
5191 1064717 : queue.create (n_basic_blocks_for_fn (cfun));
5192 :
5193 7603875 : FOR_EACH_BB_FN (bb, cfun)
5194 77324724 : FOR_BB_INSNS (bb, insn)
5195 72194087 : if (CALL_P (insn) && !SIBLING_CALL_P (insn))
5196 : {
5197 1824143 : if (bb == first)
5198 : {
5199 415622 : queue.release ();
5200 415622 : return false;
5201 : }
5202 :
5203 1408521 : bitmap_set_bit (need_new, bb->index);
5204 1408521 : bitmap_set_bit (reachable, bb->index);
5205 1408521 : queue.quick_push (bb);
5206 1408521 : break;
5207 : }
5208 :
5209 649095 : if (queue.is_empty ())
5210 : {
5211 390413 : queue.release ();
5212 390413 : return false;
5213 : }
5214 :
5215 4251759 : while (!queue.is_empty ())
5216 : {
5217 3993077 : edge e;
5218 3993077 : edge_iterator ei;
5219 :
5220 3993077 : bb = queue.pop ();
5221 9584968 : FOR_EACH_EDGE (e, ei, bb->succs)
5222 5591891 : if (e->dest != EXIT_BLOCK_PTR_FOR_FN (cfun)
5223 5591891 : && bitmap_set_bit (reachable, e->dest->index))
5224 2584556 : queue.quick_push (e->dest);
5225 : }
5226 258682 : queue.release ();
5227 :
5228 4070600 : FOR_BB_INSNS (first, insn)
5229 : {
5230 3812335 : rtx dest = interesting_dest_for_shprep (insn, NULL);
5231 3812335 : if (!dest)
5232 3415070 : continue;
5233 :
5234 397265 : if (DF_REG_DEF_COUNT (REGNO (dest)) > 1)
5235 : return false;
5236 :
5237 396848 : for (df_ref use = DF_REG_USE_CHAIN (REGNO(dest));
5238 2853209 : use;
5239 2456361 : use = DF_REF_NEXT_REG (use))
5240 : {
5241 2456361 : int ubbi = DF_REF_BB (use)->index;
5242 :
5243 : /* Only non debug insns should be taken into account. */
5244 2456361 : if (NONDEBUG_INSN_P (DF_REF_INSN (use))
5245 2456361 : && bitmap_bit_p (reachable, ubbi))
5246 1117835 : bitmap_set_bit (need_new, ubbi);
5247 : }
5248 : last_interesting_insn = insn;
5249 : }
5250 :
5251 258265 : if (!last_interesting_insn)
5252 : return false;
5253 :
5254 185262 : call_dom = nearest_common_dominator_for_set (CDI_DOMINATORS, need_new);
5255 185262 : if (call_dom == first)
5256 : return false;
5257 :
5258 97659 : loop_optimizer_init (AVOID_CFG_MODIFICATIONS);
5259 219706 : while (bb_loop_depth (call_dom) > 0)
5260 24388 : call_dom = get_immediate_dominator (CDI_DOMINATORS, call_dom);
5261 97659 : loop_optimizer_finalize ();
5262 :
5263 97659 : if (call_dom == first)
5264 : return false;
5265 :
5266 84885 : calculate_dominance_info (CDI_POST_DOMINATORS);
5267 84885 : if (dominated_by_p (CDI_POST_DOMINATORS, first, call_dom))
5268 : {
5269 7733 : free_dominance_info (CDI_POST_DOMINATORS);
5270 7733 : return false;
5271 : }
5272 77152 : free_dominance_info (CDI_POST_DOMINATORS);
5273 :
5274 77152 : if (dump_file)
5275 2 : fprintf (dump_file, "Will split live ranges of parameters at BB %i\n",
5276 : call_dom->index);
5277 :
5278 77152 : bool ret = false;
5279 768231 : FOR_BB_INSNS (first, insn)
5280 : {
5281 715633 : rtx dest = interesting_dest_for_shprep (insn, call_dom);
5282 715633 : if (!dest || dest == pic_offset_table_rtx)
5283 659938 : continue;
5284 :
5285 55695 : bool need_newreg = false;
5286 55695 : df_ref use, next;
5287 70024 : for (use = DF_REG_USE_CHAIN (REGNO (dest)); use; use = next)
5288 : {
5289 69960 : rtx_insn *uin = DF_REF_INSN (use);
5290 69960 : next = DF_REF_NEXT_REG (use);
5291 :
5292 69960 : if (DEBUG_INSN_P (uin))
5293 494 : continue;
5294 :
5295 69466 : basic_block ubb = BLOCK_FOR_INSN (uin);
5296 69466 : if (ubb == call_dom
5297 69466 : || dominated_by_p (CDI_DOMINATORS, ubb, call_dom))
5298 : {
5299 : need_newreg = true;
5300 : break;
5301 : }
5302 : }
5303 :
5304 55695 : if (need_newreg)
5305 : {
5306 55631 : rtx newreg = ira_create_new_reg (dest);
5307 :
5308 433210 : for (use = DF_REG_USE_CHAIN (REGNO (dest)); use; use = next)
5309 : {
5310 377579 : rtx_insn *uin = DF_REF_INSN (use);
5311 377579 : next = DF_REF_NEXT_REG (use);
5312 :
5313 377579 : basic_block ubb = BLOCK_FOR_INSN (uin);
5314 377579 : if (ubb == call_dom
5315 377579 : || dominated_by_p (CDI_DOMINATORS, ubb, call_dom))
5316 275515 : validate_change (uin, DF_REF_REAL_LOC (use), newreg, true);
5317 : }
5318 :
5319 55631 : rtx_insn *new_move = gen_move_insn (newreg, dest);
5320 55631 : emit_insn_after (new_move, bb_note (call_dom));
5321 55631 : if (dump_file)
5322 : {
5323 2 : fprintf (dump_file, "Split live-range of register ");
5324 2 : print_rtl_single (dump_file, dest);
5325 : }
5326 : ret = true;
5327 : }
5328 :
5329 55695 : if (insn == last_interesting_insn)
5330 : break;
5331 : }
5332 77152 : apply_change_group ();
5333 77152 : return ret;
5334 1064963 : }
5335 :
5336 : /* Perform the second half of the transformation started in
5337 : find_moveable_pseudos. We look for instances where the newly introduced
5338 : pseudo remains unallocated, and remove it by moving the definition to
5339 : just before its use, replacing the move instruction generated by
5340 : find_moveable_pseudos. */
5341 : static void
5342 1064963 : move_unallocated_pseudos (void)
5343 : {
5344 1064963 : int i;
5345 1079461 : for (i = first_moveable_pseudo; i < last_moveable_pseudo; i++)
5346 14498 : if (reg_renumber[i] < 0)
5347 : {
5348 4158 : int idx = i - first_moveable_pseudo;
5349 4158 : rtx other_reg = pseudo_replaced_reg[idx];
5350 : /* The iterating range [first_moveable_pseudo, last_moveable_pseudo)
5351 : covers every new pseudo created in find_moveable_pseudos,
5352 : regardless of the validation with it is successful or not.
5353 : So we need to skip the pseudos which were used in those failed
5354 : validations to avoid unexpected DF info and consequent ICE.
5355 : We only set pseudo_replaced_reg[] when the validation is successful
5356 : in find_moveable_pseudos, it's enough to check it here. */
5357 4158 : if (!other_reg)
5358 0 : continue;
5359 4158 : rtx_insn *def_insn = DF_REF_INSN (DF_REG_DEF_CHAIN (i));
5360 : /* The use must follow all definitions of OTHER_REG, so we can
5361 : insert the new definition immediately after any of them. */
5362 4158 : df_ref other_def = DF_REG_DEF_CHAIN (REGNO (other_reg));
5363 4158 : rtx_insn *move_insn = DF_REF_INSN (other_def);
5364 4158 : rtx_insn *newinsn = emit_insn_after (PATTERN (def_insn), move_insn);
5365 4158 : rtx set;
5366 4158 : int success;
5367 :
5368 4158 : if (dump_file)
5369 0 : fprintf (dump_file, "moving def of %d (insn %d now) ",
5370 0 : REGNO (other_reg), INSN_UID (def_insn));
5371 :
5372 4158 : delete_insn (move_insn);
5373 8316 : while ((other_def = DF_REG_DEF_CHAIN (REGNO (other_reg))))
5374 0 : delete_insn (DF_REF_INSN (other_def));
5375 4158 : delete_insn (def_insn);
5376 :
5377 4158 : set = single_set (newinsn);
5378 4158 : success = validate_change (newinsn, &SET_DEST (set), other_reg, 0);
5379 4158 : gcc_assert (success);
5380 4158 : if (dump_file)
5381 0 : fprintf (dump_file, " %d) rather than keep unallocated replacement %d\n",
5382 0 : INSN_UID (newinsn), i);
5383 4158 : SET_REG_N_REFS (i, 0);
5384 : }
5385 :
5386 1064963 : first_moveable_pseudo = last_moveable_pseudo = 0;
5387 1064963 : }
5388 :
5389 :
5390 :
5391 : /* Code dealing with scratches (changing them onto
5392 : pseudos and restoring them from the pseudos).
5393 :
5394 : We change scratches into pseudos at the beginning of IRA to
5395 : simplify dealing with them (conflicts, hard register assignments).
5396 :
5397 : If the pseudo denoting scratch was spilled it means that we do not
5398 : need a hard register for it. Such pseudos are transformed back to
5399 : scratches at the end of LRA. */
5400 :
5401 : /* Description of location of a former scratch operand. */
5402 : struct sloc
5403 : {
5404 : rtx_insn *insn; /* Insn where the scratch was. */
5405 : int nop; /* Number of the operand which was a scratch. */
5406 : unsigned regno; /* regno generated instead of scratch */
5407 : int icode; /* Original icode from which scratch was removed. */
5408 : };
5409 :
5410 : typedef struct sloc *sloc_t;
5411 :
5412 : /* Locations of the former scratches. */
5413 : static vec<sloc_t> scratches;
5414 :
5415 : /* Bitmap of scratch regnos. */
5416 : static bitmap_head scratch_bitmap;
5417 :
5418 : /* Bitmap of scratch operands. */
5419 : static bitmap_head scratch_operand_bitmap;
5420 :
5421 : /* Return true if pseudo REGNO is made of SCRATCH. */
5422 : bool
5423 380759235 : ira_former_scratch_p (int regno)
5424 : {
5425 380759235 : return bitmap_bit_p (&scratch_bitmap, regno);
5426 : }
5427 :
5428 : /* Return true if the operand NOP of INSN is a former scratch. */
5429 : bool
5430 0 : ira_former_scratch_operand_p (rtx_insn *insn, int nop)
5431 : {
5432 0 : return bitmap_bit_p (&scratch_operand_bitmap,
5433 0 : INSN_UID (insn) * MAX_RECOG_OPERANDS + nop) != 0;
5434 : }
5435 :
5436 : /* Register operand NOP in INSN as a former scratch. It will be
5437 : changed to scratch back, if it is necessary, at the LRA end. */
5438 : void
5439 98337 : ira_register_new_scratch_op (rtx_insn *insn, int nop, int icode)
5440 : {
5441 98337 : rtx op = *recog_data.operand_loc[nop];
5442 98337 : sloc_t loc = XNEW (struct sloc);
5443 98337 : ira_assert (REG_P (op));
5444 98337 : loc->insn = insn;
5445 98337 : loc->nop = nop;
5446 98337 : loc->regno = REGNO (op);
5447 98337 : loc->icode = icode;
5448 98337 : scratches.safe_push (loc);
5449 98337 : bitmap_set_bit (&scratch_bitmap, REGNO (op));
5450 196674 : bitmap_set_bit (&scratch_operand_bitmap,
5451 98337 : INSN_UID (insn) * MAX_RECOG_OPERANDS + nop);
5452 98337 : add_reg_note (insn, REG_UNUSED, op);
5453 98337 : }
5454 :
5455 : /* Return true if string STR contains constraint 'X'. */
5456 : static bool
5457 98337 : contains_X_constraint_p (const char *str)
5458 : {
5459 98337 : int c;
5460 :
5461 391393 : while ((c = *str))
5462 : {
5463 313435 : str += CONSTRAINT_LEN (c, str);
5464 313435 : if (c == 'X') return true;
5465 : }
5466 : return false;
5467 : }
5468 :
5469 : /* Change INSN's scratches into pseudos and save their location.
5470 : Return true if we changed any scratch. */
5471 : bool
5472 286458312 : ira_remove_insn_scratches (rtx_insn *insn, bool all_p, FILE *dump_file,
5473 : rtx (*get_reg) (rtx original))
5474 : {
5475 286458312 : int i;
5476 286458312 : bool insn_changed_p;
5477 286458312 : rtx reg, *loc;
5478 :
5479 286458312 : extract_insn (insn);
5480 286458312 : insn_changed_p = false;
5481 960154284 : for (i = 0; i < recog_data.n_operands; i++)
5482 : {
5483 387237660 : loc = recog_data.operand_loc[i];
5484 387237660 : if (GET_CODE (*loc) == SCRATCH && GET_MODE (*loc) != VOIDmode)
5485 : {
5486 118716 : if (! all_p && contains_X_constraint_p (recog_data.constraints[i]))
5487 20379 : continue;
5488 98337 : insn_changed_p = true;
5489 98337 : *loc = reg = get_reg (*loc);
5490 98337 : ira_register_new_scratch_op (insn, i, INSN_CODE (insn));
5491 98337 : if (dump_file != NULL)
5492 0 : fprintf (dump_file,
5493 : "Removing SCRATCH to p%u in insn #%u (nop %d)\n",
5494 0 : REGNO (reg), INSN_UID (insn), i);
5495 : }
5496 : }
5497 286458312 : return insn_changed_p;
5498 : }
5499 :
5500 : /* Return new register of the same mode as ORIGINAL. Used in
5501 : remove_scratches. */
5502 : static rtx
5503 77958 : get_scratch_reg (rtx original)
5504 : {
5505 77958 : return gen_reg_rtx (GET_MODE (original));
5506 : }
5507 :
5508 : /* Change scratches into pseudos and save their location. Return true
5509 : if we changed any scratch. */
5510 : static bool
5511 1517127 : remove_scratches (void)
5512 : {
5513 1517127 : bool change_p = false;
5514 1517127 : basic_block bb;
5515 1517127 : rtx_insn *insn;
5516 :
5517 1517127 : scratches.create (get_max_uid ());
5518 1517127 : bitmap_initialize (&scratch_bitmap, ®_obstack);
5519 1517127 : bitmap_initialize (&scratch_operand_bitmap, ®_obstack);
5520 16239609 : FOR_EACH_BB_FN (bb, cfun)
5521 180075400 : FOR_BB_INSNS (bb, insn)
5522 165352918 : if (INSN_P (insn)
5523 165352918 : && ira_remove_insn_scratches (insn, false, ira_dump_file, get_scratch_reg))
5524 : {
5525 : /* Because we might use DF, we need to keep DF info up to date. */
5526 76798 : df_insn_rescan (insn);
5527 76798 : change_p = true;
5528 : }
5529 1517127 : return change_p;
5530 : }
5531 :
5532 : /* Changes pseudos created by function remove_scratches onto scratches. */
5533 : void
5534 1517127 : ira_restore_scratches (FILE *dump_file)
5535 : {
5536 1517127 : int regno, n;
5537 1517127 : unsigned i;
5538 1517127 : rtx *op_loc;
5539 1517127 : sloc_t loc;
5540 :
5541 1615464 : for (i = 0; scratches.iterate (i, &loc); i++)
5542 : {
5543 : /* Ignore already deleted insns. */
5544 98337 : if (NOTE_P (loc->insn)
5545 0 : && NOTE_KIND (loc->insn) == NOTE_INSN_DELETED)
5546 0 : continue;
5547 98337 : extract_insn (loc->insn);
5548 98337 : if (loc->icode != INSN_CODE (loc->insn))
5549 : {
5550 : /* The icode doesn't match, which means the insn has been modified
5551 : (e.g. register elimination). The scratch cannot be restored. */
5552 0 : continue;
5553 : }
5554 98337 : op_loc = recog_data.operand_loc[loc->nop];
5555 98337 : if (REG_P (*op_loc)
5556 98337 : && ((regno = REGNO (*op_loc)) >= FIRST_PSEUDO_REGISTER)
5557 196674 : && reg_renumber[regno] < 0)
5558 : {
5559 : /* It should be only case when scratch register with chosen
5560 : constraint 'X' did not get memory or hard register. */
5561 4613 : ira_assert (ira_former_scratch_p (regno));
5562 4613 : *op_loc = gen_rtx_SCRATCH (GET_MODE (*op_loc));
5563 4613 : for (n = 0; n < recog_data.n_dups; n++)
5564 0 : *recog_data.dup_loc[n]
5565 0 : = *recog_data.operand_loc[(int) recog_data.dup_num[n]];
5566 4613 : if (dump_file != NULL)
5567 0 : fprintf (dump_file, "Restoring SCRATCH in insn #%u(nop %d)\n",
5568 0 : INSN_UID (loc->insn), loc->nop);
5569 : }
5570 : }
5571 1615464 : for (i = 0; scratches.iterate (i, &loc); i++)
5572 98337 : free (loc);
5573 1517127 : scratches.release ();
5574 1517127 : bitmap_clear (&scratch_bitmap);
5575 1517127 : bitmap_clear (&scratch_operand_bitmap);
5576 1517127 : }
5577 :
5578 :
5579 :
5580 : /* If the backend knows where to allocate pseudos for hard
5581 : register initial values, register these allocations now. */
5582 : static void
5583 1517127 : allocate_initial_values (void)
5584 : {
5585 1517127 : if (targetm.allocate_initial_value)
5586 : {
5587 : rtx hreg, preg, x;
5588 : int i, regno;
5589 :
5590 0 : for (i = 0; HARD_REGISTER_NUM_P (i); i++)
5591 : {
5592 0 : if (! initial_value_entry (i, &hreg, &preg))
5593 : break;
5594 :
5595 0 : x = targetm.allocate_initial_value (hreg);
5596 0 : regno = REGNO (preg);
5597 0 : if (x && REG_N_SETS (regno) <= 1)
5598 : {
5599 0 : if (MEM_P (x))
5600 0 : reg_equiv_memory_loc (regno) = x;
5601 : else
5602 : {
5603 0 : basic_block bb;
5604 0 : int new_regno;
5605 :
5606 0 : gcc_assert (REG_P (x));
5607 0 : new_regno = REGNO (x);
5608 0 : reg_renumber[regno] = new_regno;
5609 : /* Poke the regno right into regno_reg_rtx so that even
5610 : fixed regs are accepted. */
5611 0 : SET_REGNO (preg, new_regno);
5612 : /* Update global register liveness information. */
5613 0 : FOR_EACH_BB_FN (bb, cfun)
5614 : {
5615 0 : if (REGNO_REG_SET_P (df_get_live_in (bb), regno))
5616 0 : SET_REGNO_REG_SET (df_get_live_in (bb), new_regno);
5617 0 : if (REGNO_REG_SET_P (df_get_live_out (bb), regno))
5618 0 : SET_REGNO_REG_SET (df_get_live_out (bb), new_regno);
5619 : }
5620 : }
5621 : }
5622 : }
5623 :
5624 0 : gcc_checking_assert (! initial_value_entry (FIRST_PSEUDO_REGISTER,
5625 : &hreg, &preg));
5626 : }
5627 1517127 : }
5628 :
5629 :
5630 :
5631 :
5632 : /* True when we use LRA instead of reload pass for the current
5633 : function. */
5634 : bool ira_use_lra_p;
5635 :
5636 : /* True if we have allocno conflicts. It is false for non-optimized
5637 : mode or when the conflict table is too big. */
5638 : bool ira_conflicts_p;
5639 :
5640 : /* Saved between IRA and reload. */
5641 : static int saved_flag_ira_share_spill_slots;
5642 :
5643 : /* Set to true while in IRA. */
5644 : bool ira_in_progress = false;
5645 :
5646 : /* Set up array ira_hard_regno_nrefs. */
5647 : static void
5648 1517127 : setup_hard_regno_nrefs (void)
5649 : {
5650 1517127 : int i;
5651 :
5652 144127065 : for (i = 0; i < FIRST_PSEUDO_REGISTER; i++)
5653 : {
5654 142609938 : ira_hard_regno_nrefs[i] = 0;
5655 142609938 : for (df_ref use = DF_REG_USE_CHAIN (i);
5656 261726591 : use != NULL;
5657 119116653 : use = DF_REF_NEXT_REG (use))
5658 119116653 : if (DF_REF_CLASS (use) != DF_REF_ARTIFICIAL
5659 54834315 : && !(DF_REF_INSN_INFO (use) && DEBUG_INSN_P (DF_REF_INSN (use))))
5660 51374966 : ira_hard_regno_nrefs[i]++;
5661 142609938 : for (df_ref def = DF_REG_DEF_CHAIN (i);
5662 723086108 : def != NULL;
5663 580476170 : def = DF_REF_NEXT_REG (def))
5664 580476170 : if (DF_REF_CLASS (def) != DF_REF_ARTIFICIAL
5665 552254135 : && !(DF_REF_INSN_INFO (def) && DEBUG_INSN_P (DF_REF_INSN (def))))
5666 552254135 : ira_hard_regno_nrefs[i]++;
5667 : }
5668 1517127 : }
5669 :
5670 : /* This is the main entry of IRA. */
5671 : static void
5672 1517127 : ira (FILE *f)
5673 : {
5674 1517127 : bool loops_p;
5675 1517127 : int ira_max_point_before_emit;
5676 1517127 : bool saved_flag_caller_saves = flag_caller_saves;
5677 1517127 : enum ira_region saved_flag_ira_region = flag_ira_region;
5678 1517127 : basic_block bb;
5679 1517127 : edge_iterator ei;
5680 1517127 : edge e;
5681 1517127 : bool output_jump_reload_p = false;
5682 :
5683 1517127 : setup_hard_regno_nrefs ();
5684 1517127 : lra_reset_dependent_filters ();
5685 1517127 : if (ira_use_lra_p)
5686 : {
5687 : /* First put potential jump output reloads on the output edges
5688 : as USE which will be removed at the end of LRA. The major
5689 : goal is actually to create BBs for critical edges for LRA and
5690 : populate them later by live info. In LRA it will be
5691 : difficult to do this. */
5692 16239600 : FOR_EACH_BB_FN (bb, cfun)
5693 : {
5694 14722473 : rtx_insn *end = BB_END (bb);
5695 14722473 : if (!JUMP_P (end))
5696 5807368 : continue;
5697 8915105 : extract_insn (end);
5698 33070966 : for (int i = 0; i < recog_data.n_operands; i++)
5699 15240910 : if (recog_data.operand_type[i] != OP_IN)
5700 : {
5701 154 : bool skip_p = false;
5702 516 : FOR_EACH_EDGE (e, ei, bb->succs)
5703 735 : if (EDGE_CRITICAL_P (e)
5704 11 : && e->dest != EXIT_BLOCK_PTR_FOR_FN (cfun)
5705 373 : && (e->flags & EDGE_ABNORMAL))
5706 : {
5707 : skip_p = true;
5708 : break;
5709 : }
5710 154 : if (skip_p)
5711 : break;
5712 154 : output_jump_reload_p = true;
5713 516 : FOR_EACH_EDGE (e, ei, bb->succs)
5714 735 : if (EDGE_CRITICAL_P (e)
5715 373 : && e->dest != EXIT_BLOCK_PTR_FOR_FN (cfun))
5716 : {
5717 11 : start_sequence ();
5718 : /* We need to put some no-op insn here. We can
5719 : not put a note as commit_edges insertion will
5720 : fail. */
5721 11 : emit_insn (gen_rtx_USE (VOIDmode, const1_rtx));
5722 11 : rtx_insn *insns = end_sequence ();
5723 11 : insert_insn_on_edge (insns, e);
5724 : }
5725 : break;
5726 : }
5727 : }
5728 1517127 : if (output_jump_reload_p)
5729 148 : commit_edge_insertions ();
5730 : }
5731 :
5732 1517127 : if (flag_ira_verbose < 10)
5733 : {
5734 1517127 : internal_flag_ira_verbose = flag_ira_verbose;
5735 1517127 : ira_dump_file = f;
5736 : }
5737 : else
5738 : {
5739 0 : internal_flag_ira_verbose = flag_ira_verbose - 10;
5740 0 : ira_dump_file = stderr;
5741 : }
5742 :
5743 1517127 : clear_bb_flags ();
5744 :
5745 : /* Determine if the current function is a leaf before running IRA
5746 : since this can impact optimizations done by the prologue and
5747 : epilogue thus changing register elimination offsets.
5748 : Other target callbacks may use crtl->is_leaf too, including
5749 : SHRINK_WRAPPING_ENABLED, so initialize as early as possible. */
5750 1517127 : crtl->is_leaf = leaf_function_p ();
5751 :
5752 : /* Perform target specific PIC register initialization. */
5753 1517127 : targetm.init_pic_reg ();
5754 :
5755 1517127 : ira_conflicts_p = optimize > 0;
5756 :
5757 : /* Determine the number of pseudos actually requiring coloring. */
5758 1517127 : unsigned int num_used_regs = 0;
5759 69300916 : for (unsigned int i = FIRST_PSEUDO_REGISTER; i < DF_REG_SIZE (df); i++)
5760 67783789 : if (DF_REG_DEF_COUNT (i) || DF_REG_USE_COUNT (i))
5761 30875062 : num_used_regs++;
5762 :
5763 : /* If there are too many pseudos and/or basic blocks (e.g. 10K pseudos and
5764 : 10K blocks or 100K pseudos and 1K blocks) or we have too many function
5765 : insns, we will use simplified and faster algorithms in LRA. */
5766 1517127 : lra_simple_p
5767 1517127 : = (ira_use_lra_p
5768 1517127 : && (num_used_regs >= (1U << 26) / last_basic_block_for_fn (cfun)
5769 : /* get_max_insn_count is a good evaluation of the number of insns
5770 : as most optimizations are done on tree-SSA level. Unlike
5771 : get_max_uid, it excludes debug insns, so the choice below does
5772 : not depend on debug information. */
5773 1517122 : || ((uint64_t) get_max_insn_count ()
5774 1517122 : > (uint64_t) param_ira_simple_lra_insn_threshold * 1000)));
5775 :
5776 1517127 : if (lra_simple_p)
5777 : {
5778 : /* It permits to skip live range splitting in LRA. */
5779 5 : flag_caller_saves = false;
5780 : /* There is no sense to do regional allocation when we use
5781 : simplified LRA. */
5782 5 : flag_ira_region = IRA_REGION_ONE;
5783 5 : ira_conflicts_p = false;
5784 : }
5785 :
5786 : #ifndef IRA_NO_OBSTACK
5787 : gcc_obstack_init (&ira_obstack);
5788 : #endif
5789 1517127 : bitmap_obstack_initialize (&ira_bitmap_obstack);
5790 :
5791 : /* LRA uses its own infrastructure to handle caller save registers. */
5792 1517127 : if (flag_caller_saves && !ira_use_lra_p)
5793 0 : init_caller_save ();
5794 :
5795 1517127 : setup_prohibited_mode_move_regs ();
5796 1517127 : decrease_live_ranges_number ();
5797 1517127 : df_note_add_problem ();
5798 :
5799 : /* DF_LIVE can't be used in the register allocator, too many other
5800 : parts of the compiler depend on using the "classic" liveness
5801 : interpretation of the DF_LR problem. See PR38711.
5802 : Remove the problem, so that we don't spend time updating it in
5803 : any of the df_analyze() calls during IRA/LRA. */
5804 1517127 : if (optimize > 1)
5805 982741 : df_remove_problem (df_live);
5806 1517127 : gcc_checking_assert (df_live == NULL);
5807 :
5808 1517127 : if (flag_checking)
5809 1517107 : df->changeable_flags |= DF_VERIFY_SCHEDULED;
5810 :
5811 1517127 : df_analyze ();
5812 :
5813 1517127 : init_reg_equiv ();
5814 1517127 : if (ira_conflicts_p)
5815 : {
5816 1064963 : calculate_dominance_info (CDI_DOMINATORS);
5817 :
5818 1064963 : if (split_live_ranges_for_shrink_wrap ())
5819 27971 : df_analyze ();
5820 :
5821 1064963 : free_dominance_info (CDI_DOMINATORS);
5822 : }
5823 :
5824 1517127 : df_clear_flags (DF_NO_INSN_RESCAN);
5825 :
5826 1517127 : indirect_jump_optimize ();
5827 1517127 : if (delete_trivially_dead_insns (get_insns (), max_reg_num ()))
5828 5589 : df_analyze ();
5829 :
5830 1517127 : regstat_init_n_sets_and_refs ();
5831 1517127 : regstat_compute_ri ();
5832 :
5833 : /* If we are not optimizing, then this is the only place before
5834 : register allocation where dataflow is done. And that is needed
5835 : to generate these warnings. */
5836 1517127 : if (warn_clobbered)
5837 135555 : generate_setjmp_warnings ();
5838 :
5839 : /* update_equiv_regs can use reg classes of pseudos and they are set up in
5840 : register pressure sensitive scheduling and loop invariant motion and in
5841 : live range shrinking. This info can become obsolete if we add new pseudos
5842 : since the last set up. Recalculate it again if the new pseudos were
5843 : added. */
5844 1517127 : if (resize_reg_info () && (flag_sched_pressure || flag_live_range_shrinkage
5845 1517032 : || flag_ira_loop_pressure))
5846 43 : ira_set_pseudo_classes (true, ira_dump_file);
5847 :
5848 1517127 : init_alias_analysis ();
5849 1517127 : loop_optimizer_init (AVOID_CFG_MODIFICATIONS);
5850 1517127 : reg_equiv = XCNEWVEC (struct equivalence, max_reg_num ());
5851 1517127 : update_equiv_regs_prescan ();
5852 1517127 : update_equiv_regs ();
5853 :
5854 : /* Don't move insns if live range shrinkage or register
5855 : pressure-sensitive scheduling were done because it will not
5856 : improve allocation but likely worsen insn scheduling. */
5857 1517127 : if (optimize
5858 1064963 : && !flag_live_range_shrinkage
5859 1064932 : && !(flag_sched_pressure && flag_schedule_insns))
5860 1064910 : combine_and_move_insns ();
5861 :
5862 : /* Gather additional equivalences with memory. */
5863 1517127 : if (optimize && flag_expensive_optimizations)
5864 982697 : add_store_equivs ();
5865 :
5866 1517127 : loop_optimizer_finalize ();
5867 1517127 : free_dominance_info (CDI_DOMINATORS);
5868 1517127 : end_alias_analysis ();
5869 1517127 : free (reg_equiv);
5870 :
5871 : /* Once max_regno changes, we need to free and re-init/re-compute
5872 : some data structures like regstat_n_sets_and_refs and reg_info_p. */
5873 1589748 : auto regstat_recompute_for_max_regno = []() {
5874 72621 : regstat_free_n_sets_and_refs ();
5875 72621 : regstat_free_ri ();
5876 72621 : regstat_init_n_sets_and_refs ();
5877 72621 : regstat_compute_ri ();
5878 72621 : resize_reg_info ();
5879 72621 : };
5880 :
5881 1517127 : int max_regno_before_rm = max_reg_num ();
5882 1517127 : if (ira_use_lra_p && remove_scratches ())
5883 : {
5884 36915 : ira_expand_reg_equiv ();
5885 : /* For now remove_scatches is supposed to create pseudos when it
5886 : succeeds, assert this happens all the time. Once it doesn't
5887 : hold, we should guard the regstat recompute for the case
5888 : max_regno changes. */
5889 36915 : gcc_assert (max_regno_before_rm != max_reg_num ());
5890 36915 : regstat_recompute_for_max_regno ();
5891 : }
5892 :
5893 1517127 : setup_reg_equiv ();
5894 1517127 : grow_reg_equivs ();
5895 1517127 : setup_reg_equiv_init ();
5896 :
5897 1517127 : allocated_reg_info_size = max_reg_num ();
5898 :
5899 : /* It is not worth to do such improvement when we use a simple
5900 : allocation because of -O0 usage or because the function is too
5901 : big. */
5902 1517127 : if (ira_conflicts_p)
5903 1064963 : find_moveable_pseudos ();
5904 :
5905 1517127 : max_regno_before_ira = max_reg_num ();
5906 1517127 : ira_setup_eliminable_regset ();
5907 :
5908 1517127 : ira_overall_cost = ira_reg_cost = ira_mem_cost = 0;
5909 1517127 : ira_load_cost = ira_store_cost = ira_shuffle_cost = 0;
5910 1517127 : ira_move_loops_num = ira_additional_jumps_num = 0;
5911 :
5912 1517127 : ira_assert (current_loops == NULL);
5913 1517127 : if (flag_ira_region == IRA_REGION_ALL || flag_ira_region == IRA_REGION_MIXED)
5914 1017475 : loop_optimizer_init (AVOID_CFG_MODIFICATIONS | LOOPS_HAVE_RECORDED_EXITS);
5915 :
5916 1517127 : if (internal_flag_ira_verbose > 0 && ira_dump_file != NULL)
5917 95 : fprintf (ira_dump_file, "Building IRA IR\n");
5918 1517127 : loops_p = ira_build ();
5919 :
5920 1517127 : ira_assert (ira_conflicts_p || !loops_p);
5921 :
5922 1517127 : saved_flag_ira_share_spill_slots = flag_ira_share_spill_slots;
5923 1517127 : if (too_high_register_pressure_p () || cfun->calls_setjmp)
5924 : /* It is just wasting compiler's time to pack spilled pseudos into
5925 : stack slots in this case -- prohibit it. We also do this if
5926 : there is setjmp call because a variable not modified between
5927 : setjmp and longjmp the compiler is required to preserve its
5928 : value and sharing slots does not guarantee it. */
5929 1412 : flag_ira_share_spill_slots = false;
5930 :
5931 1517127 : ira_color ();
5932 :
5933 1517127 : ira_max_point_before_emit = ira_max_point;
5934 :
5935 1517127 : ira_initiate_emit_data ();
5936 :
5937 1517127 : ira_emit (loops_p);
5938 :
5939 1517127 : max_regno = max_reg_num ();
5940 1517127 : if (ira_conflicts_p)
5941 : {
5942 1064963 : if (! loops_p)
5943 : {
5944 1028795 : if (! ira_use_lra_p)
5945 0 : ira_initiate_assign ();
5946 : }
5947 : else
5948 : {
5949 36168 : expand_reg_info ();
5950 :
5951 36168 : if (ira_use_lra_p)
5952 : {
5953 36168 : ira_allocno_t a;
5954 36168 : ira_allocno_iterator ai;
5955 :
5956 11813310 : FOR_EACH_ALLOCNO (a, ai)
5957 : {
5958 11740974 : int old_regno = ALLOCNO_REGNO (a);
5959 11740974 : int new_regno = REGNO (ALLOCNO_EMIT_DATA (a)->reg);
5960 :
5961 11740974 : ALLOCNO_REGNO (a) = new_regno;
5962 :
5963 11740974 : if (old_regno != new_regno)
5964 1297993 : setup_reg_classes (new_regno, reg_preferred_class (old_regno),
5965 : reg_alternate_class (old_regno),
5966 : reg_allocno_class (old_regno));
5967 : }
5968 : }
5969 : else
5970 : {
5971 0 : if (internal_flag_ira_verbose > 0 && ira_dump_file != NULL)
5972 0 : fprintf (ira_dump_file, "Flattening IR\n");
5973 0 : ira_flattening (max_regno_before_ira, ira_max_point_before_emit);
5974 : }
5975 : /* New insns were generated: add notes and recalculate live
5976 : info. */
5977 36168 : df_analyze ();
5978 :
5979 : /* ??? Rebuild the loop tree, but why? Does the loop tree
5980 : change if new insns were generated? Can that be handled
5981 : by updating the loop tree incrementally? */
5982 36168 : loop_optimizer_finalize ();
5983 36168 : free_dominance_info (CDI_DOMINATORS);
5984 36168 : loop_optimizer_init (AVOID_CFG_MODIFICATIONS
5985 : | LOOPS_HAVE_RECORDED_EXITS);
5986 :
5987 36168 : if (! ira_use_lra_p)
5988 : {
5989 0 : setup_allocno_assignment_flags ();
5990 0 : ira_initiate_assign ();
5991 0 : ira_reassign_conflict_allocnos (max_regno);
5992 : }
5993 : }
5994 : }
5995 :
5996 1517127 : ira_finish_emit_data ();
5997 :
5998 1517127 : setup_reg_renumber ();
5999 :
6000 1517127 : calculate_allocation_cost ();
6001 :
6002 : #ifdef ENABLE_IRA_CHECKING
6003 1517127 : if (ira_conflicts_p && ! ira_use_lra_p)
6004 : /* Opposite to reload pass, LRA does not use any conflict info
6005 : from IRA. We don't rebuild conflict info for LRA (through
6006 : ira_flattening call) and cannot use the check here. We could
6007 : rebuild this info for LRA in the check mode but there is a risk
6008 : that code generated with the check and without it will be a bit
6009 : different. Calling ira_flattening in any mode would be a
6010 : wasting CPU time. So do not check the allocation for LRA. */
6011 0 : check_allocation ();
6012 : #endif
6013 :
6014 1517127 : if (max_regno != max_regno_before_ira)
6015 35706 : regstat_recompute_for_max_regno ();
6016 :
6017 1517127 : overall_cost_before = ira_overall_cost;
6018 1517127 : if (! ira_conflicts_p)
6019 452164 : grow_reg_equivs ();
6020 : else
6021 : {
6022 1064963 : fix_reg_equiv_init ();
6023 :
6024 : #ifdef ENABLE_IRA_CHECKING
6025 1064963 : print_redundant_copies ();
6026 : #endif
6027 1064963 : if (! ira_use_lra_p)
6028 : {
6029 0 : ira_spilled_reg_stack_slots_num = 0;
6030 0 : ira_spilled_reg_stack_slots
6031 0 : = ((class ira_spilled_reg_stack_slot *)
6032 0 : ira_allocate (max_regno
6033 : * sizeof (class ira_spilled_reg_stack_slot)));
6034 0 : memset ((void *)ira_spilled_reg_stack_slots, 0,
6035 0 : max_regno * sizeof (class ira_spilled_reg_stack_slot));
6036 : }
6037 : }
6038 1517127 : allocate_initial_values ();
6039 :
6040 : /* See comment for find_moveable_pseudos call. */
6041 1517127 : if (ira_conflicts_p)
6042 1064963 : move_unallocated_pseudos ();
6043 :
6044 : /* Restore original values. */
6045 1517127 : if (lra_simple_p)
6046 : {
6047 5 : flag_caller_saves = saved_flag_caller_saves;
6048 5 : flag_ira_region = saved_flag_ira_region;
6049 : }
6050 1517127 : }
6051 :
6052 : /* Modify asm goto to avoid further trouble with this insn. We can
6053 : not replace the insn by USE as in other asm insns as we still
6054 : need to keep CFG consistency. */
6055 : void
6056 6 : ira_nullify_asm_goto (rtx_insn *insn)
6057 : {
6058 6 : ira_assert (JUMP_P (insn) && INSN_CODE (insn) < 0);
6059 6 : rtx tmp = extract_asm_operands (PATTERN (insn));
6060 6 : PATTERN (insn) = gen_rtx_ASM_OPERANDS (VOIDmode, ggc_strdup (""), "", 0,
6061 : rtvec_alloc (0),
6062 : rtvec_alloc (0),
6063 : ASM_OPERANDS_LABEL_VEC (tmp),
6064 : ASM_OPERANDS_SOURCE_LOCATION(tmp));
6065 6 : }
6066 :
6067 : static void
6068 1517127 : do_reload (void)
6069 : {
6070 1517127 : basic_block bb;
6071 1517127 : bool need_dce;
6072 1517127 : unsigned pic_offset_table_regno = INVALID_REGNUM;
6073 :
6074 1517127 : if (flag_ira_verbose < 10)
6075 1517127 : ira_dump_file = dump_file;
6076 :
6077 : /* If pic_offset_table_rtx is a pseudo register, then keep it so
6078 : after reload to avoid possible wrong usages of hard reg assigned
6079 : to it. */
6080 1517127 : if (pic_offset_table_rtx
6081 1517127 : && REGNO (pic_offset_table_rtx) >= FIRST_PSEUDO_REGISTER)
6082 : pic_offset_table_regno = REGNO (pic_offset_table_rtx);
6083 :
6084 1517127 : timevar_push (TV_RELOAD);
6085 1517127 : if (ira_use_lra_p)
6086 : {
6087 1517127 : if (current_loops != NULL)
6088 : {
6089 1017475 : loop_optimizer_finalize ();
6090 1017475 : free_dominance_info (CDI_DOMINATORS);
6091 : }
6092 19377855 : FOR_ALL_BB_FN (bb, cfun)
6093 17860728 : bb->loop_father = NULL;
6094 1517127 : current_loops = NULL;
6095 :
6096 1517127 : ira_destroy ();
6097 :
6098 1517127 : lra (ira_dump_file, internal_flag_ira_verbose);
6099 : /* ???!!! Move it before lra () when we use ira_reg_equiv in
6100 : LRA. */
6101 1517127 : vec_free (reg_equivs);
6102 1517127 : reg_equivs = NULL;
6103 1517127 : need_dce = false;
6104 : }
6105 : else
6106 : {
6107 0 : df_set_flags (DF_NO_INSN_RESCAN);
6108 0 : build_insn_chain ();
6109 :
6110 0 : need_dce = reload (get_insns (), ira_conflicts_p);
6111 : }
6112 :
6113 1517127 : timevar_pop (TV_RELOAD);
6114 :
6115 1517127 : timevar_push (TV_IRA);
6116 :
6117 1517127 : if (ira_conflicts_p && ! ira_use_lra_p)
6118 : {
6119 0 : ira_free (ira_spilled_reg_stack_slots);
6120 0 : ira_finish_assign ();
6121 : }
6122 :
6123 1517127 : if (internal_flag_ira_verbose > 0 && ira_dump_file != NULL
6124 96 : && overall_cost_before != ira_overall_cost)
6125 0 : fprintf (ira_dump_file, "+++Overall after reload %" PRId64 "\n",
6126 : ira_overall_cost);
6127 :
6128 1517127 : flag_ira_share_spill_slots = saved_flag_ira_share_spill_slots;
6129 :
6130 1517127 : if (! ira_use_lra_p)
6131 : {
6132 0 : ira_destroy ();
6133 0 : if (current_loops != NULL)
6134 : {
6135 0 : loop_optimizer_finalize ();
6136 0 : free_dominance_info (CDI_DOMINATORS);
6137 : }
6138 0 : FOR_ALL_BB_FN (bb, cfun)
6139 0 : bb->loop_father = NULL;
6140 0 : current_loops = NULL;
6141 :
6142 0 : regstat_free_ri ();
6143 0 : regstat_free_n_sets_and_refs ();
6144 : }
6145 :
6146 1517127 : if (optimize)
6147 1064963 : cleanup_cfg (CLEANUP_EXPENSIVE);
6148 :
6149 1517127 : finish_reg_equiv ();
6150 :
6151 1517127 : bitmap_obstack_release (&ira_bitmap_obstack);
6152 : #ifndef IRA_NO_OBSTACK
6153 : obstack_free (&ira_obstack, NULL);
6154 : #endif
6155 :
6156 : /* The code after the reload has changed so much that at this point
6157 : we might as well just rescan everything. Note that
6158 : df_rescan_all_insns is not going to help here because it does not
6159 : touch the artificial uses and defs. */
6160 1517127 : df_finish_pass (true);
6161 1517127 : df_scan_alloc (NULL);
6162 1517127 : df_scan_blocks ();
6163 :
6164 1517127 : if (optimize > 1)
6165 : {
6166 982741 : df_live_add_problem ();
6167 982741 : df_live_set_all_dirty ();
6168 : }
6169 :
6170 1517127 : if (optimize)
6171 1064963 : df_analyze ();
6172 :
6173 1517127 : if (need_dce && optimize)
6174 0 : run_fast_dce ();
6175 :
6176 : /* Diagnose uses of the hard frame pointer when it is used as a global
6177 : register. Often we can get away with letting the user appropriate
6178 : the frame pointer, but we should let them know when code generation
6179 : makes that impossible. */
6180 1517127 : if (global_regs[HARD_FRAME_POINTER_REGNUM] && frame_pointer_needed)
6181 : {
6182 2 : tree decl = global_regs_decl[HARD_FRAME_POINTER_REGNUM];
6183 2 : error_at (DECL_SOURCE_LOCATION (current_function_decl),
6184 : "frame pointer required, but reserved");
6185 2 : inform (DECL_SOURCE_LOCATION (decl), "for %qD", decl);
6186 : }
6187 :
6188 : /* If we are doing generic stack checking, give a warning if this
6189 : function's frame size is larger than we expect. */
6190 1517127 : if (flag_stack_check == GENERIC_STACK_CHECK)
6191 : {
6192 49 : poly_int64 size = get_frame_size () + STACK_CHECK_FIXED_FRAME_SIZE;
6193 :
6194 4655 : for (int i = 0; i < FIRST_PSEUDO_REGISTER; i++)
6195 4606 : if (df_regs_ever_live_p (i)
6196 230 : && !fixed_regs[i]
6197 4772 : && !crtl->abi->clobbers_full_reg_p (i))
6198 80 : size += UNITS_PER_WORD;
6199 :
6200 49 : if (constant_lower_bound (size) > STACK_CHECK_MAX_FRAME_SIZE)
6201 1 : warning (0, "frame size too large for reliable stack checking");
6202 : }
6203 :
6204 1517127 : if (pic_offset_table_regno != INVALID_REGNUM)
6205 81059 : pic_offset_table_rtx = gen_rtx_REG (Pmode, pic_offset_table_regno);
6206 :
6207 1517127 : timevar_pop (TV_IRA);
6208 1517127 : }
6209 :
6210 : /* Run the integrated register allocator. */
6211 :
6212 : namespace {
6213 :
6214 : const pass_data pass_data_ira =
6215 : {
6216 : RTL_PASS, /* type */
6217 : "ira", /* name */
6218 : OPTGROUP_NONE, /* optinfo_flags */
6219 : TV_IRA, /* tv_id */
6220 : 0, /* properties_required */
6221 : 0, /* properties_provided */
6222 : 0, /* properties_destroyed */
6223 : 0, /* todo_flags_start */
6224 : TODO_do_not_ggc_collect, /* todo_flags_finish */
6225 : };
6226 :
6227 : class pass_ira : public rtl_opt_pass
6228 : {
6229 : public:
6230 297180 : pass_ira (gcc::context *ctxt)
6231 594360 : : rtl_opt_pass (pass_data_ira, ctxt)
6232 : {}
6233 :
6234 : /* opt_pass methods: */
6235 1517135 : bool gate (function *) final override
6236 : {
6237 1517135 : return !targetm.no_register_allocation;
6238 : }
6239 1517127 : unsigned int execute (function *) final override
6240 : {
6241 1517127 : ira_in_progress = true;
6242 1517127 : ira (dump_file);
6243 1517127 : ira_in_progress = false;
6244 1517127 : return 0;
6245 : }
6246 :
6247 : }; // class pass_ira
6248 :
6249 : } // anon namespace
6250 :
6251 : rtl_opt_pass *
6252 297180 : make_pass_ira (gcc::context *ctxt)
6253 : {
6254 297180 : return new pass_ira (ctxt);
6255 : }
6256 :
6257 : namespace {
6258 :
6259 : const pass_data pass_data_reload =
6260 : {
6261 : RTL_PASS, /* type */
6262 : "reload", /* name */
6263 : OPTGROUP_NONE, /* optinfo_flags */
6264 : TV_RELOAD, /* tv_id */
6265 : 0, /* properties_required */
6266 : 0, /* properties_provided */
6267 : 0, /* properties_destroyed */
6268 : 0, /* todo_flags_start */
6269 : 0, /* todo_flags_finish */
6270 : };
6271 :
6272 : class pass_reload : public rtl_opt_pass
6273 : {
6274 : public:
6275 297180 : pass_reload (gcc::context *ctxt)
6276 594360 : : rtl_opt_pass (pass_data_reload, ctxt)
6277 : {}
6278 :
6279 : /* opt_pass methods: */
6280 1517135 : bool gate (function *) final override
6281 : {
6282 1517135 : return !targetm.no_register_allocation;
6283 : }
6284 1517127 : unsigned int execute (function *) final override
6285 : {
6286 1517127 : do_reload ();
6287 1517127 : return 0;
6288 : }
6289 :
6290 : }; // class pass_reload
6291 :
6292 : } // anon namespace
6293 :
6294 : rtl_opt_pass *
6295 297180 : make_pass_reload (gcc::context *ctxt)
6296 : {
6297 297180 : return new pass_reload (ctxt);
6298 : }
|