Line data Source code
1 : /* Integrated Register Allocator (IRA) entry point.
2 : Copyright (C) 2006-2026 Free Software Foundation, Inc.
3 : Contributed by Vladimir Makarov <vmakarov@redhat.com>.
4 :
5 : This file is part of GCC.
6 :
7 : GCC is free software; you can redistribute it and/or modify it under
8 : the terms of the GNU General Public License as published by the Free
9 : Software Foundation; either version 3, or (at your option) any later
10 : version.
11 :
12 : GCC is distributed in the hope that it will be useful, but WITHOUT ANY
13 : WARRANTY; without even the implied warranty of MERCHANTABILITY or
14 : FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
15 : for more details.
16 :
17 : You should have received a copy of the GNU General Public License
18 : along with GCC; see the file COPYING3. If not see
19 : <http://www.gnu.org/licenses/>. */
20 :
21 : /* The integrated register allocator (IRA) is a
22 : regional register allocator performing graph coloring on a top-down
23 : traversal of nested regions. Graph coloring in a region is based
24 : on Chaitin-Briggs algorithm. It is called integrated because
25 : register coalescing, register live range splitting, and choosing a
26 : better hard register are done on-the-fly during coloring. Register
27 : coalescing and choosing a cheaper hard register is done by hard
28 : register preferencing during hard register assigning. The live
29 : range splitting is a byproduct of the regional register allocation.
30 :
31 : Major IRA notions are:
32 :
33 : o *Region* is a part of CFG where graph coloring based on
34 : Chaitin-Briggs algorithm is done. IRA can work on any set of
35 : nested CFG regions forming a tree. Currently the regions are
36 : the entire function for the root region and natural loops for
37 : the other regions. Therefore data structure representing a
38 : region is called loop_tree_node.
39 :
40 : o *Allocno class* is a register class used for allocation of
41 : given allocno. It means that only hard register of given
42 : register class can be assigned to given allocno. In reality,
43 : even smaller subset of (*profitable*) hard registers can be
44 : assigned. In rare cases, the subset can be even smaller
45 : because our modification of Chaitin-Briggs algorithm requires
46 : that sets of hard registers can be assigned to allocnos forms a
47 : forest, i.e. the sets can be ordered in a way where any
48 : previous set is not intersected with given set or is a superset
49 : of given set.
50 :
51 : o *Pressure class* is a register class belonging to a set of
52 : register classes containing all of the hard-registers available
53 : for register allocation. The set of all pressure classes for a
54 : target is defined in the corresponding machine-description file
55 : according some criteria. Register pressure is calculated only
56 : for pressure classes and it affects some IRA decisions as
57 : forming allocation regions.
58 :
59 : o *Allocno* represents the live range of a pseudo-register in a
60 : region. Besides the obvious attributes like the corresponding
61 : pseudo-register number, allocno class, conflicting allocnos and
62 : conflicting hard-registers, there are a few allocno attributes
63 : which are important for understanding the allocation algorithm:
64 :
65 : - *Live ranges*. This is a list of ranges of *program points*
66 : where the allocno lives. Program points represent places
67 : where a pseudo can be born or become dead (there are
68 : approximately two times more program points than the insns)
69 : and they are represented by integers starting with 0. The
70 : live ranges are used to find conflicts between allocnos.
71 : They also play very important role for the transformation of
72 : the IRA internal representation of several regions into a one
73 : region representation. The later is used during the reload
74 : pass work because each allocno represents all of the
75 : corresponding pseudo-registers.
76 :
77 : - *Hard-register costs*. This is a vector of size equal to the
78 : number of available hard-registers of the allocno class. The
79 : cost of a callee-clobbered hard-register for an allocno is
80 : increased by the cost of save/restore code around the calls
81 : through the given allocno's life. If the allocno is a move
82 : instruction operand and another operand is a hard-register of
83 : the allocno class, the cost of the hard-register is decreased
84 : by the move cost.
85 :
86 : When an allocno is assigned, the hard-register with minimal
87 : full cost is used. Initially, a hard-register's full cost is
88 : the corresponding value from the hard-register's cost vector.
89 : If the allocno is connected by a *copy* (see below) to
90 : another allocno which has just received a hard-register, the
91 : cost of the hard-register is decreased. Before choosing a
92 : hard-register for an allocno, the allocno's current costs of
93 : the hard-registers are modified by the conflict hard-register
94 : costs of all of the conflicting allocnos which are not
95 : assigned yet.
96 :
97 : - *Conflict hard-register costs*. This is a vector of the same
98 : size as the hard-register costs vector. To permit an
99 : unassigned allocno to get a better hard-register, IRA uses
100 : this vector to calculate the final full cost of the
101 : available hard-registers. Conflict hard-register costs of an
102 : unassigned allocno are also changed with a change of the
103 : hard-register cost of the allocno when a copy involving the
104 : allocno is processed as described above. This is done to
105 : show other unassigned allocnos that a given allocno prefers
106 : some hard-registers in order to remove the move instruction
107 : corresponding to the copy.
108 :
109 : o *Cap*. If a pseudo-register does not live in a region but
110 : lives in a nested region, IRA creates a special allocno called
111 : a cap in the outer region. A region cap is also created for a
112 : subregion cap.
113 :
114 : o *Copy*. Allocnos can be connected by copies. Copies are used
115 : to modify hard-register costs for allocnos during coloring.
116 : Such modifications reflects a preference to use the same
117 : hard-register for the allocnos connected by copies. Usually
118 : copies are created for move insns (in this case it results in
119 : register coalescing). But IRA also creates copies for operands
120 : of an insn which should be assigned to the same hard-register
121 : due to constraints in the machine description (it usually
122 : results in removing a move generated in reload to satisfy
123 : the constraints) and copies referring to the allocno which is
124 : the output operand of an instruction and the allocno which is
125 : an input operand dying in the instruction (creation of such
126 : copies results in less register shuffling). IRA *does not*
127 : create copies between the same register allocnos from different
128 : regions because we use another technique for propagating
129 : hard-register preference on the borders of regions.
130 :
131 : Allocnos (including caps) for the upper region in the region tree
132 : *accumulate* information important for coloring from allocnos with
133 : the same pseudo-register from nested regions. This includes
134 : hard-register and memory costs, conflicts with hard-registers,
135 : allocno conflicts, allocno copies and more. *Thus, attributes for
136 : allocnos in a region have the same values as if the region had no
137 : subregions*. It means that attributes for allocnos in the
138 : outermost region corresponding to the function have the same values
139 : as though the allocation used only one region which is the entire
140 : function. It also means that we can look at IRA work as if the
141 : first IRA did allocation for all function then it improved the
142 : allocation for loops then their subloops and so on.
143 :
144 : IRA major passes are:
145 :
146 : o Building IRA internal representation which consists of the
147 : following subpasses:
148 :
149 : * First, IRA builds regions and creates allocnos (file
150 : ira-build.cc) and initializes most of their attributes.
151 :
152 : * Then IRA finds an allocno class for each allocno and
153 : calculates its initial (non-accumulated) cost of memory and
154 : each hard-register of its allocno class (file ira-cost.c).
155 :
156 : * IRA creates live ranges of each allocno, calculates register
157 : pressure for each pressure class in each region, sets up
158 : conflict hard registers for each allocno and info about calls
159 : the allocno lives through (file ira-lives.cc).
160 :
161 : * IRA removes low register pressure loops from the regions
162 : mostly to speed IRA up (file ira-build.cc).
163 :
164 : * IRA propagates accumulated allocno info from lower region
165 : allocnos to corresponding upper region allocnos (file
166 : ira-build.cc).
167 :
168 : * IRA creates all caps (file ira-build.cc).
169 :
170 : * Having live-ranges of allocnos and their classes, IRA creates
171 : conflicting allocnos for each allocno. Conflicting allocnos
172 : are stored as a bit vector or array of pointers to the
173 : conflicting allocnos whatever is more profitable (file
174 : ira-conflicts.cc). At this point IRA creates allocno copies.
175 :
176 : o Coloring. Now IRA has all necessary info to start graph coloring
177 : process. It is done in each region on top-down traverse of the
178 : region tree (file ira-color.cc). There are following subpasses:
179 :
180 : * Finding profitable hard registers of corresponding allocno
181 : class for each allocno. For example, only callee-saved hard
182 : registers are frequently profitable for allocnos living
183 : through colors. If the profitable hard register set of
184 : allocno does not form a tree based on subset relation, we use
185 : some approximation to form the tree. This approximation is
186 : used to figure out trivial colorability of allocnos. The
187 : approximation is a pretty rare case.
188 :
189 : * Putting allocnos onto the coloring stack. IRA uses Briggs
190 : optimistic coloring which is a major improvement over
191 : Chaitin's coloring. Therefore IRA does not spill allocnos at
192 : this point. There is some freedom in the order of putting
193 : allocnos on the stack which can affect the final result of
194 : the allocation. IRA uses some heuristics to improve the
195 : order. The major one is to form *threads* from colorable
196 : allocnos and push them on the stack by threads. Thread is a
197 : set of non-conflicting colorable allocnos connected by
198 : copies. The thread contains allocnos from the colorable
199 : bucket or colorable allocnos already pushed onto the coloring
200 : stack. Pushing thread allocnos one after another onto the
201 : stack increases chances of removing copies when the allocnos
202 : get the same hard reg.
203 :
204 : We also use a modification of Chaitin-Briggs algorithm which
205 : works for intersected register classes of allocnos. To
206 : figure out trivial colorability of allocnos, the mentioned
207 : above tree of hard register sets is used. To get an idea how
208 : the algorithm works in i386 example, let us consider an
209 : allocno to which any general hard register can be assigned.
210 : If the allocno conflicts with eight allocnos to which only
211 : EAX register can be assigned, given allocno is still
212 : trivially colorable because all conflicting allocnos might be
213 : assigned only to EAX and all other general hard registers are
214 : still free.
215 :
216 : To get an idea of the used trivial colorability criterion, it
217 : is also useful to read article "Graph-Coloring Register
218 : Allocation for Irregular Architectures" by Michael D. Smith
219 : and Glen Holloway. Major difference between the article
220 : approach and approach used in IRA is that Smith's approach
221 : takes register classes only from machine description and IRA
222 : calculate register classes from intermediate code too
223 : (e.g. an explicit usage of hard registers in RTL code for
224 : parameter passing can result in creation of additional
225 : register classes which contain or exclude the hard
226 : registers). That makes IRA approach useful for improving
227 : coloring even for architectures with regular register files
228 : and in fact some benchmarking shows the improvement for
229 : regular class architectures is even bigger than for irregular
230 : ones. Another difference is that Smith's approach chooses
231 : intersection of classes of all insn operands in which a given
232 : pseudo occurs. IRA can use bigger classes if it is still
233 : more profitable than memory usage.
234 :
235 : * Popping the allocnos from the stack and assigning them hard
236 : registers. If IRA cannot assign a hard register to an
237 : allocno and the allocno is coalesced, IRA undoes the
238 : coalescing and puts the uncoalesced allocnos onto the stack in
239 : the hope that some such allocnos will get a hard register
240 : separately. If IRA fails to assign hard register or memory
241 : is more profitable for it, IRA spills the allocno. IRA
242 : assigns the allocno the hard-register with minimal full
243 : allocation cost which reflects the cost of usage of the
244 : hard-register for the allocno and cost of usage of the
245 : hard-register for allocnos conflicting with given allocno.
246 :
247 : * Chaitin-Briggs coloring assigns as many pseudos as possible
248 : to hard registers. After coloring we try to improve
249 : allocation with cost point of view. We improve the
250 : allocation by spilling some allocnos and assigning the freed
251 : hard registers to other allocnos if it decreases the overall
252 : allocation cost.
253 :
254 : * After allocno assigning in the region, IRA modifies the hard
255 : register and memory costs for the corresponding allocnos in
256 : the subregions to reflect the cost of possible loads, stores,
257 : or moves on the border of the region and its subregions.
258 : When default regional allocation algorithm is used
259 : (-fira-algorithm=mixed), IRA just propagates the assignment
260 : for allocnos if the register pressure in the region for the
261 : corresponding pressure class is less than number of available
262 : hard registers for given pressure class.
263 :
264 : o Spill/restore code moving. When IRA performs an allocation
265 : by traversing regions in top-down order, it does not know what
266 : happens below in the region tree. Therefore, sometimes IRA
267 : misses opportunities to perform a better allocation. A simple
268 : optimization tries to improve allocation in a region having
269 : subregions and containing in another region. If the
270 : corresponding allocnos in the subregion are spilled, it spills
271 : the region allocno if it is profitable. The optimization
272 : implements a simple iterative algorithm performing profitable
273 : transformations while they are still possible. It is fast in
274 : practice, so there is no real need for a better time complexity
275 : algorithm.
276 :
277 : o Code change. After coloring, two allocnos representing the
278 : same pseudo-register outside and inside a region respectively
279 : may be assigned to different locations (hard-registers or
280 : memory). In this case IRA creates and uses a new
281 : pseudo-register inside the region and adds code to move allocno
282 : values on the region's borders. This is done during top-down
283 : traversal of the regions (file ira-emit.cc). In some
284 : complicated cases IRA can create a new allocno to move allocno
285 : values (e.g. when a swap of values stored in two hard-registers
286 : is needed). At this stage, the new allocno is marked as
287 : spilled. IRA still creates the pseudo-register and the moves
288 : on the region borders even when both allocnos were assigned to
289 : the same hard-register. If the reload pass spills a
290 : pseudo-register for some reason, the effect will be smaller
291 : because another allocno will still be in the hard-register. In
292 : most cases, this is better then spilling both allocnos. If
293 : reload does not change the allocation for the two
294 : pseudo-registers, the trivial move will be removed by
295 : post-reload optimizations. IRA does not generate moves for
296 : allocnos assigned to the same hard register when the default
297 : regional allocation algorithm is used and the register pressure
298 : in the region for the corresponding pressure class is less than
299 : number of available hard registers for given pressure class.
300 : IRA also does some optimizations to remove redundant stores and
301 : to reduce code duplication on the region borders.
302 :
303 : o Flattening internal representation. After changing code, IRA
304 : transforms its internal representation for several regions into
305 : one region representation (file ira-build.cc). This process is
306 : called IR flattening. Such process is more complicated than IR
307 : rebuilding would be, but is much faster.
308 :
309 : o After IR flattening, IRA tries to assign hard registers to all
310 : spilled allocnos. This is implemented by a simple and fast
311 : priority coloring algorithm (see function
312 : ira_reassign_conflict_allocnos::ira-color.cc). Here new allocnos
313 : created during the code change pass can be assigned to hard
314 : registers.
315 :
316 : o At the end IRA calls the reload pass. The reload pass
317 : communicates with IRA through several functions in file
318 : ira-color.cc to improve its decisions in
319 :
320 : * sharing stack slots for the spilled pseudos based on IRA info
321 : about pseudo-register conflicts.
322 :
323 : * reassigning hard-registers to all spilled pseudos at the end
324 : of each reload iteration.
325 :
326 : * choosing a better hard-register to spill based on IRA info
327 : about pseudo-register live ranges and the register pressure
328 : in places where the pseudo-register lives.
329 :
330 : IRA uses a lot of data representing the target processors. These
331 : data are initialized in file ira.cc.
332 :
333 : If function has no loops (or the loops are ignored when
334 : -fira-algorithm=CB is used), we have classic Chaitin-Briggs
335 : coloring (only instead of separate pass of coalescing, we use hard
336 : register preferencing). In such case, IRA works much faster
337 : because many things are not made (like IR flattening, the
338 : spill/restore optimization, and the code change).
339 :
340 : Literature is worth to read for better understanding the code:
341 :
342 : o Preston Briggs, Keith D. Cooper, Linda Torczon. Improvements to
343 : Graph Coloring Register Allocation.
344 :
345 : o David Callahan, Brian Koblenz. Register allocation via
346 : hierarchical graph coloring.
347 :
348 : o Keith Cooper, Anshuman Dasgupta, Jason Eckhardt. Revisiting Graph
349 : Coloring Register Allocation: A Study of the Chaitin-Briggs and
350 : Callahan-Koblenz Algorithms.
351 :
352 : o Guei-Yuan Lueh, Thomas Gross, and Ali-Reza Adl-Tabatabai. Global
353 : Register Allocation Based on Graph Fusion.
354 :
355 : o Michael D. Smith and Glenn Holloway. Graph-Coloring Register
356 : Allocation for Irregular Architectures
357 :
358 : o Vladimir Makarov. The Integrated Register Allocator for GCC.
359 :
360 : o Vladimir Makarov. The top-down register allocator for irregular
361 : register file architectures.
362 :
363 : */
364 :
365 :
366 : #include "config.h"
367 : #include "system.h"
368 : #include "coretypes.h"
369 : #include "backend.h"
370 : #include "target.h"
371 : #include "rtl.h"
372 : #include "tree.h"
373 : #include "df.h"
374 : #include "memmodel.h"
375 : #include "tm_p.h"
376 : #include "insn-config.h"
377 : #include "regs.h"
378 : #include "ira.h"
379 : #include "ira-int.h"
380 : #include "diagnostic-core.h"
381 : #include "cfgrtl.h"
382 : #include "cfgbuild.h"
383 : #include "cfgcleanup.h"
384 : #include "expr.h"
385 : #include "tree-pass.h"
386 : #include "output.h"
387 : #include "reload.h"
388 : #include "cfgloop.h"
389 : #include "lra.h"
390 : #include "dce.h"
391 : #include "dbgcnt.h"
392 : #include "rtl-iter.h"
393 : #include "shrink-wrap.h"
394 : #include "print-rtl.h"
395 :
396 : struct target_ira default_target_ira;
397 : class target_ira_int default_target_ira_int;
398 : #if SWITCHABLE_TARGET
399 : struct target_ira *this_target_ira = &default_target_ira;
400 : class target_ira_int *this_target_ira_int = &default_target_ira_int;
401 : #endif
402 :
403 : /* A modified value of flag `-fira-verbose' used internally. */
404 : int internal_flag_ira_verbose;
405 :
406 : /* Dump file of the allocator if it is not NULL. */
407 : FILE *ira_dump_file;
408 :
409 : /* The number of elements in the following array. */
410 : int ira_spilled_reg_stack_slots_num;
411 :
412 : /* The following array contains info about spilled pseudo-registers
413 : stack slots used in current function so far. */
414 : class ira_spilled_reg_stack_slot *ira_spilled_reg_stack_slots;
415 :
416 : /* Correspondingly overall cost of the allocation, overall cost before
417 : reload, cost of the allocnos assigned to hard-registers, cost of
418 : the allocnos assigned to memory, cost of loads, stores and register
419 : move insns generated for pseudo-register live range splitting (see
420 : ira-emit.cc). */
421 : int64_t ira_overall_cost, overall_cost_before;
422 : int64_t ira_reg_cost, ira_mem_cost;
423 : int64_t ira_load_cost, ira_store_cost, ira_shuffle_cost;
424 : int ira_move_loops_num, ira_additional_jumps_num;
425 :
426 : /* All registers that can be eliminated. */
427 :
428 : HARD_REG_SET eliminable_regset;
429 :
430 : /* Value of max_reg_num () before IRA work start. This value helps
431 : us to recognize a situation when new pseudos were created during
432 : IRA work. */
433 : static int max_regno_before_ira;
434 :
435 : /* Temporary hard reg set used for a different calculation. */
436 : static HARD_REG_SET temp_hard_regset;
437 :
438 : #define last_mode_for_init_move_cost \
439 : (this_target_ira_int->x_last_mode_for_init_move_cost)
440 :
441 :
442 : /* The function sets up the map IRA_REG_MODE_HARD_REGSET. */
443 : static void
444 218792 : setup_reg_mode_hard_regset (void)
445 : {
446 218792 : int i, m, hard_regno;
447 :
448 27349000 : for (m = 0; m < NUM_MACHINE_MODES; m++)
449 2523109344 : for (hard_regno = 0; hard_regno < FIRST_PSEUDO_REGISTER; hard_regno++)
450 : {
451 2495979136 : CLEAR_HARD_REG_SET (ira_reg_mode_hard_regset[hard_regno][m]);
452 7554349664 : for (i = hard_regno_nregs (hard_regno, (machine_mode) m) - 1;
453 7554349664 : i >= 0; i--)
454 5058370528 : if (hard_regno + i < FIRST_PSEUDO_REGISTER)
455 4640104495 : SET_HARD_REG_BIT (ira_reg_mode_hard_regset[hard_regno][m],
456 : hard_regno + i);
457 : }
458 218792 : }
459 :
460 :
461 : #define no_unit_alloc_regs \
462 : (this_target_ira_int->x_no_unit_alloc_regs)
463 :
464 : /* The function sets up the three arrays declared above. */
465 : static void
466 218792 : setup_class_hard_regs (void)
467 : {
468 218792 : int cl, i, hard_regno, n;
469 218792 : unsigned int j;
470 218792 : HARD_REG_SET processed_hard_reg_set;
471 :
472 218792 : ira_assert (SHRT_MAX >= FIRST_PSEUDO_REGISTER);
473 7657720 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
474 : {
475 7438928 : temp_hard_regset = reg_class_contents[cl] & ~no_unit_alloc_regs;
476 7438928 : CLEAR_HARD_REG_SET (processed_hard_reg_set);
477 691820304 : for (i = 0; i < FIRST_PSEUDO_REGISTER; i++)
478 : {
479 684381376 : ira_non_ordered_class_hard_regs[cl][i] = -1;
480 684381376 : ira_class_hard_reg_index[cl][i] = -1;
481 : }
482 691820304 : for (n = 0, i = 0; i < FIRST_PSEUDO_REGISTER; i++)
483 : {
484 : #ifdef REG_ALLOC_ORDER
485 684381376 : hard_regno = reg_alloc_order[i];
486 : #else
487 : hard_regno = i;
488 : #endif
489 684381376 : if (TEST_HARD_REG_BIT (processed_hard_reg_set, hard_regno))
490 29755712 : continue;
491 654625664 : SET_HARD_REG_BIT (processed_hard_reg_set, hard_regno);
492 654625664 : if (! TEST_HARD_REG_BIT (temp_hard_regset, hard_regno))
493 579237865 : ira_class_hard_reg_index[cl][hard_regno] = -1;
494 : else
495 : {
496 75387799 : ira_class_hard_reg_index[cl][hard_regno] = n;
497 75387799 : ira_class_hard_regs[cl][n++] = hard_regno;
498 : }
499 : }
500 7438928 : ira_class_hard_regs_num[cl] = n;
501 7438928 : n = 0;
502 7438928 : j = 0;
503 7438928 : hard_reg_set_iterator hrsi;
504 82826727 : EXECUTE_IF_SET_IN_HARD_REG_SET (temp_hard_regset, 0, j, hrsi)
505 75387799 : ira_non_ordered_class_hard_regs[cl][n++] = j;
506 :
507 7438928 : ira_assert (ira_class_hard_regs_num[cl] == n);
508 : }
509 218792 : }
510 :
511 : /* Set up global variables defining info about hard registers for the
512 : allocation. These depend on USE_HARD_FRAME_P whose TRUE value means
513 : that we can use the hard frame pointer for the allocation. */
514 : static void
515 218792 : setup_alloc_regs (bool use_hard_frame_p)
516 : {
517 : #ifdef ADJUST_REG_ALLOC_ORDER
518 218792 : ADJUST_REG_ALLOC_ORDER;
519 : #endif
520 218792 : no_unit_alloc_regs = fixed_nonglobal_reg_set;
521 218792 : if (! use_hard_frame_p)
522 69126 : add_to_hard_reg_set (&no_unit_alloc_regs, Pmode,
523 : HARD_FRAME_POINTER_REGNUM);
524 218792 : setup_class_hard_regs ();
525 218792 : }
526 :
527 :
528 :
529 : #define alloc_reg_class_subclasses \
530 : (this_target_ira_int->x_alloc_reg_class_subclasses)
531 :
532 : /* Initialize the table of subclasses of each reg class. */
533 : static void
534 218792 : setup_reg_subclasses (void)
535 : {
536 218792 : int i, j;
537 218792 : HARD_REG_SET temp_hard_regset2;
538 :
539 7657720 : for (i = 0; i < N_REG_CLASSES; i++)
540 260362480 : for (j = 0; j < N_REG_CLASSES; j++)
541 252923552 : alloc_reg_class_subclasses[i][j] = LIM_REG_CLASSES;
542 :
543 7657720 : for (i = 0; i < N_REG_CLASSES; i++)
544 : {
545 7438928 : if (i == (int) NO_REGS)
546 218792 : continue;
547 :
548 7220136 : temp_hard_regset = reg_class_contents[i] & ~no_unit_alloc_regs;
549 14440272 : if (hard_reg_set_empty_p (temp_hard_regset))
550 433843 : continue;
551 237520255 : for (j = 0; j < N_REG_CLASSES; j++)
552 230733962 : if (i != j)
553 : {
554 223947669 : enum reg_class *p;
555 :
556 223947669 : temp_hard_regset2 = reg_class_contents[j] & ~no_unit_alloc_regs;
557 447895338 : if (! hard_reg_set_subset_p (temp_hard_regset,
558 : temp_hard_regset2))
559 170652240 : continue;
560 53295429 : p = &alloc_reg_class_subclasses[j][0];
561 512616177 : while (*p != LIM_REG_CLASSES) p++;
562 53295429 : *p = (enum reg_class) i;
563 : }
564 : }
565 218792 : }
566 :
567 :
568 :
569 : /* Set up IRA_MEMORY_MOVE_COST and IRA_MAX_MEMORY_MOVE_COST. */
570 : static void
571 218792 : setup_class_subset_and_memory_move_costs (void)
572 : {
573 218792 : int cl, cl2, mode, cost;
574 218792 : HARD_REG_SET temp_hard_regset2;
575 :
576 27349000 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
577 27130208 : ira_memory_move_cost[mode][NO_REGS][0]
578 27130208 : = ira_memory_move_cost[mode][NO_REGS][1] = SHRT_MAX;
579 7657720 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
580 : {
581 7438928 : if (cl != (int) NO_REGS)
582 902517000 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
583 : {
584 1790593728 : ira_max_memory_move_cost[mode][cl][0]
585 895296864 : = ira_memory_move_cost[mode][cl][0]
586 895296864 : = memory_move_cost ((machine_mode) mode,
587 : (reg_class_t) cl, false);
588 1790593728 : ira_max_memory_move_cost[mode][cl][1]
589 895296864 : = ira_memory_move_cost[mode][cl][1]
590 895296864 : = memory_move_cost ((machine_mode) mode,
591 : (reg_class_t) cl, true);
592 : /* Costs for NO_REGS are used in cost calculation on the
593 : 1st pass when the preferred register classes are not
594 : known yet. In this case we take the best scenario. */
595 895296864 : if (!targetm.hard_regno_mode_ok (ira_class_hard_regs[cl][0],
596 : (machine_mode) mode))
597 660205442 : continue;
598 :
599 235091422 : if (ira_memory_move_cost[mode][NO_REGS][0]
600 235091422 : > ira_memory_move_cost[mode][cl][0])
601 13210664 : ira_max_memory_move_cost[mode][NO_REGS][0]
602 13210664 : = ira_memory_move_cost[mode][NO_REGS][0]
603 13210664 : = ira_memory_move_cost[mode][cl][0];
604 235091422 : if (ira_memory_move_cost[mode][NO_REGS][1]
605 235091422 : > ira_memory_move_cost[mode][cl][1])
606 13201506 : ira_max_memory_move_cost[mode][NO_REGS][1]
607 13201506 : = ira_memory_move_cost[mode][NO_REGS][1]
608 13201506 : = ira_memory_move_cost[mode][cl][1];
609 : }
610 : }
611 7657720 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
612 260362480 : for (cl2 = (int) N_REG_CLASSES - 1; cl2 >= 0; cl2--)
613 : {
614 252923552 : temp_hard_regset = reg_class_contents[cl] & ~no_unit_alloc_regs;
615 252923552 : temp_hard_regset2 = reg_class_contents[cl2] & ~no_unit_alloc_regs;
616 252923552 : ira_class_subset_p[cl][cl2]
617 252923552 : = hard_reg_set_subset_p (temp_hard_regset, temp_hard_regset2);
618 252923552 : if (! hard_reg_set_empty_p (temp_hard_regset2)
619 483657514 : && hard_reg_set_subset_p (reg_class_contents[cl2],
620 : reg_class_contents[cl]))
621 6809177250 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
622 : {
623 6754703832 : cost = ira_memory_move_cost[mode][cl2][0];
624 6754703832 : if (cost > ira_max_memory_move_cost[mode][cl][0])
625 115293442 : ira_max_memory_move_cost[mode][cl][0] = cost;
626 6754703832 : cost = ira_memory_move_cost[mode][cl2][1];
627 6754703832 : if (cost > ira_max_memory_move_cost[mode][cl][1])
628 115347108 : ira_max_memory_move_cost[mode][cl][1] = cost;
629 : }
630 : }
631 7657720 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
632 929866000 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
633 : {
634 922427072 : ira_memory_move_cost[mode][cl][0]
635 922427072 : = ira_max_memory_move_cost[mode][cl][0];
636 922427072 : ira_memory_move_cost[mode][cl][1]
637 922427072 : = ira_max_memory_move_cost[mode][cl][1];
638 : }
639 218792 : setup_reg_subclasses ();
640 218792 : }
641 :
642 :
643 :
644 : /* Define the following macro if allocation through malloc if
645 : preferable. */
646 : #define IRA_NO_OBSTACK
647 :
648 : #ifndef IRA_NO_OBSTACK
649 : /* Obstack used for storing all dynamic data (except bitmaps) of the
650 : IRA. */
651 : static struct obstack ira_obstack;
652 : #endif
653 :
654 : /* Obstack used for storing all bitmaps of the IRA. */
655 : static struct bitmap_obstack ira_bitmap_obstack;
656 :
657 : /* Allocate memory of size LEN for IRA data. */
658 : void *
659 208171941 : ira_allocate (size_t len)
660 : {
661 208171941 : void *res;
662 :
663 : #ifndef IRA_NO_OBSTACK
664 : res = obstack_alloc (&ira_obstack, len);
665 : #else
666 208171941 : res = xmalloc (len);
667 : #endif
668 208171941 : return res;
669 : }
670 :
671 : /* Free memory ADDR allocated for IRA data. */
672 : void
673 208171941 : ira_free (void *addr ATTRIBUTE_UNUSED)
674 : {
675 : #ifndef IRA_NO_OBSTACK
676 : /* do nothing */
677 : #else
678 208171941 : free (addr);
679 : #endif
680 208171941 : }
681 :
682 :
683 : /* Allocate and returns bitmap for IRA. */
684 : bitmap
685 10714414 : ira_allocate_bitmap (void)
686 : {
687 10714414 : return BITMAP_ALLOC (&ira_bitmap_obstack);
688 : }
689 :
690 : /* Free bitmap B allocated for IRA. */
691 : void
692 10714414 : ira_free_bitmap (bitmap b ATTRIBUTE_UNUSED)
693 : {
694 : /* do nothing */
695 10714414 : }
696 :
697 :
698 :
699 : /* Output information about allocation of all allocnos (except for
700 : caps) into file F. */
701 : void
702 95 : ira_print_disposition (FILE *f)
703 : {
704 95 : int i, n, max_regno;
705 95 : ira_allocno_t a;
706 95 : basic_block bb;
707 :
708 95 : fprintf (f, "Disposition:");
709 95 : max_regno = max_reg_num ();
710 1981 : for (n = 0, i = FIRST_PSEUDO_REGISTER; i < max_regno; i++)
711 1791 : for (a = ira_regno_allocno_map[i];
712 2386 : a != NULL;
713 595 : a = ALLOCNO_NEXT_REGNO_ALLOCNO (a))
714 : {
715 595 : if (n % 4 == 0)
716 178 : fprintf (f, "\n");
717 595 : n++;
718 595 : fprintf (f, " %4d:r%-4d", ALLOCNO_NUM (a), ALLOCNO_REGNO (a));
719 595 : if ((bb = ALLOCNO_LOOP_TREE_NODE (a)->bb) != NULL)
720 0 : fprintf (f, "b%-3d", bb->index);
721 : else
722 595 : fprintf (f, "l%-3d", ALLOCNO_LOOP_TREE_NODE (a)->loop_num);
723 595 : if (ALLOCNO_HARD_REGNO (a) >= 0)
724 594 : fprintf (f, " %3d", ALLOCNO_HARD_REGNO (a));
725 : else
726 1 : fprintf (f, " mem");
727 : }
728 95 : fprintf (f, "\n");
729 95 : }
730 :
731 : /* Outputs information about allocation of all allocnos into
732 : stderr. */
733 : void
734 0 : ira_debug_disposition (void)
735 : {
736 0 : ira_print_disposition (stderr);
737 0 : }
738 :
739 :
740 :
741 : /* Set up ira_stack_reg_pressure_class which is the biggest pressure
742 : register class containing stack registers or NO_REGS if there are
743 : no stack registers. To find this class, we iterate through all
744 : register pressure classes and choose the first register pressure
745 : class containing all the stack registers and having the biggest
746 : size. */
747 : static void
748 218792 : setup_stack_reg_pressure_class (void)
749 : {
750 218792 : ira_stack_reg_pressure_class = NO_REGS;
751 : #ifdef STACK_REGS
752 218792 : {
753 218792 : int i, best, size;
754 218792 : enum reg_class cl;
755 218792 : HARD_REG_SET temp_hard_regset2;
756 :
757 218792 : CLEAR_HARD_REG_SET (temp_hard_regset);
758 1969128 : for (i = FIRST_STACK_REG; i <= LAST_STACK_REG; i++)
759 1750336 : SET_HARD_REG_BIT (temp_hard_regset, i);
760 : best = 0;
761 1095934 : for (i = 0; i < ira_pressure_classes_num; i++)
762 : {
763 877142 : cl = ira_pressure_classes[i];
764 877142 : temp_hard_regset2 = temp_hard_regset & reg_class_contents[cl];
765 877142 : size = hard_reg_set_popcount (temp_hard_regset2);
766 877142 : if (best < size)
767 : {
768 218261 : best = size;
769 218261 : ira_stack_reg_pressure_class = cl;
770 : }
771 : }
772 : }
773 : #endif
774 218792 : }
775 :
776 : /* Find pressure classes which are register classes for which we
777 : calculate register pressure in IRA, register pressure sensitive
778 : insn scheduling, and register pressure sensitive loop invariant
779 : motion.
780 :
781 : To make register pressure calculation easy, we always use
782 : non-intersected register pressure classes. A move of hard
783 : registers from one register pressure class is not more expensive
784 : than load and store of the hard registers. Most likely an allocno
785 : class will be a subset of a register pressure class and in many
786 : cases a register pressure class. That makes usage of register
787 : pressure classes a good approximation to find a high register
788 : pressure. */
789 : static void
790 218792 : setup_pressure_classes (void)
791 : {
792 218792 : int cost, i, n, curr;
793 218792 : int cl, cl2;
794 218792 : enum reg_class pressure_classes[N_REG_CLASSES];
795 218792 : int m;
796 218792 : HARD_REG_SET temp_hard_regset2;
797 218792 : bool insert_p;
798 :
799 218792 : if (targetm.compute_pressure_classes)
800 0 : n = targetm.compute_pressure_classes (pressure_classes);
801 : else
802 : {
803 : n = 0;
804 7657720 : for (cl = 0; cl < N_REG_CLASSES; cl++)
805 : {
806 7438928 : if (ira_class_hard_regs_num[cl] == 0)
807 652635 : continue;
808 6786293 : if (ira_class_hard_regs_num[cl] != 1
809 : /* A register class without subclasses may contain a few
810 : hard registers and movement between them is costly
811 : (e.g. SPARC FPCC registers). We still should consider it
812 : as a candidate for a pressure class. */
813 4818822 : && alloc_reg_class_subclasses[cl][0] < cl)
814 : {
815 : /* Check that the moves between any hard registers of the
816 : current class are not more expensive for a legal mode
817 : than load/store of the hard registers of the current
818 : class. Such class is a potential candidate to be a
819 : register pressure class. */
820 223787247 : for (m = 0; m < NUM_MACHINE_MODES; m++)
821 : {
822 222475009 : temp_hard_regset
823 222475009 : = (reg_class_contents[cl]
824 222475009 : & ~(no_unit_alloc_regs
825 222475009 : | ira_prohibited_class_mode_regs[cl][m]));
826 444950018 : if (hard_reg_set_empty_p (temp_hard_regset))
827 164460028 : continue;
828 58014981 : ira_init_register_move_cost_if_necessary ((machine_mode) m);
829 58014981 : cost = ira_register_move_cost[m][cl][cl];
830 58014981 : if (cost <= ira_max_memory_move_cost[m][cl][1]
831 54730289 : || cost <= ira_max_memory_move_cost[m][cl][0])
832 : break;
833 : }
834 4596930 : if (m >= NUM_MACHINE_MODES)
835 1312238 : continue;
836 : }
837 5474055 : curr = 0;
838 5474055 : insert_p = true;
839 5474055 : temp_hard_regset = reg_class_contents[cl] & ~no_unit_alloc_regs;
840 : /* Remove so far added pressure classes which are subset of the
841 : current candidate class. Prefer GENERAL_REGS as a pressure
842 : register class to another class containing the same
843 : allocatable hard registers. We do this because machine
844 : dependent cost hooks might give wrong costs for the latter
845 : class but always give the right cost for the former class
846 : (GENERAL_REGS). */
847 19640220 : for (i = 0; i < n; i++)
848 : {
849 14166165 : cl2 = pressure_classes[i];
850 14166165 : temp_hard_regset2 = (reg_class_contents[cl2]
851 14166165 : & ~no_unit_alloc_regs);
852 14166165 : if (hard_reg_set_subset_p (temp_hard_regset, temp_hard_regset2)
853 15334462 : && (temp_hard_regset != temp_hard_regset2
854 1099356 : || cl2 == (int) GENERAL_REGS))
855 : {
856 725146 : pressure_classes[curr++] = (enum reg_class) cl2;
857 725146 : insert_p = false;
858 725146 : continue;
859 : }
860 16875279 : if (hard_reg_set_subset_p (temp_hard_regset2, temp_hard_regset)
861 17312786 : && (temp_hard_regset2 != temp_hard_regset
862 443151 : || cl == (int) GENERAL_REGS))
863 3434260 : continue;
864 10006759 : if (temp_hard_regset2 == temp_hard_regset)
865 437507 : insert_p = false;
866 10006759 : pressure_classes[curr++] = (enum reg_class) cl2;
867 : }
868 : /* If the current candidate is a subset of a so far added
869 : pressure class, don't add it to the list of the pressure
870 : classes. */
871 5474055 : if (insert_p)
872 4311402 : pressure_classes[curr++] = (enum reg_class) cl;
873 : n = curr;
874 : }
875 : }
876 : #ifdef ENABLE_IRA_CHECKING
877 218792 : {
878 218792 : HARD_REG_SET ignore_hard_regs;
879 :
880 : /* Check pressure classes correctness: here we check that hard
881 : registers from all register pressure classes contains all hard
882 : registers available for the allocation. */
883 875168 : CLEAR_HARD_REG_SET (temp_hard_regset);
884 218792 : CLEAR_HARD_REG_SET (temp_hard_regset2);
885 218792 : ignore_hard_regs = no_unit_alloc_regs;
886 7657720 : for (cl = 0; cl < LIM_REG_CLASSES; cl++)
887 : {
888 : /* For some targets (like MIPS with MD_REGS), there are some
889 : classes with hard registers available for allocation but
890 : not able to hold value of any mode. */
891 211330372 : for (m = 0; m < NUM_MACHINE_MODES; m++)
892 210677737 : if (contains_reg_of_mode[cl][m])
893 : break;
894 7438928 : if (m >= NUM_MACHINE_MODES)
895 : {
896 652635 : ignore_hard_regs |= reg_class_contents[cl];
897 652635 : continue;
898 : }
899 31816294 : for (i = 0; i < n; i++)
900 25907143 : if ((int) pressure_classes[i] == cl)
901 : break;
902 6786293 : temp_hard_regset2 |= reg_class_contents[cl];
903 6786293 : if (i < n)
904 7438928 : temp_hard_regset |= reg_class_contents[cl];
905 : }
906 20347656 : for (i = 0; i < FIRST_PSEUDO_REGISTER; i++)
907 : /* Some targets (like SPARC with ICC reg) have allocatable regs
908 : for which no reg class is defined. */
909 20128864 : if (REGNO_REG_CLASS (i) == NO_REGS)
910 437584 : SET_HARD_REG_BIT (ignore_hard_regs, i);
911 218792 : temp_hard_regset &= ~ignore_hard_regs;
912 218792 : temp_hard_regset2 &= ~ignore_hard_regs;
913 437584 : ira_assert (hard_reg_set_subset_p (temp_hard_regset2, temp_hard_regset));
914 : }
915 : #endif
916 218792 : ira_pressure_classes_num = 0;
917 1095934 : for (i = 0; i < n; i++)
918 : {
919 877142 : cl = (int) pressure_classes[i];
920 877142 : ira_reg_pressure_class_p[cl] = true;
921 877142 : ira_pressure_classes[ira_pressure_classes_num++] = (enum reg_class) cl;
922 : }
923 218792 : setup_stack_reg_pressure_class ();
924 218792 : }
925 :
926 : /* Set up IRA_UNIFORM_CLASS_P. Uniform class is a register class
927 : whose register move cost between any registers of the class is the
928 : same as for all its subclasses. We use the data to speed up the
929 : 2nd pass of calculations of allocno costs. */
930 : static void
931 218792 : setup_uniform_class_p (void)
932 : {
933 218792 : int i, cl, cl2, m;
934 :
935 7657720 : for (cl = 0; cl < N_REG_CLASSES; cl++)
936 : {
937 7438928 : ira_uniform_class_p[cl] = false;
938 7438928 : if (ira_class_hard_regs_num[cl] == 0)
939 652635 : continue;
940 : /* We cannot use alloc_reg_class_subclasses here because move
941 : cost hooks does not take into account that some registers are
942 : unavailable for the subtarget. E.g. for i686, INT_SSE_REGS
943 : is element of alloc_reg_class_subclasses for GENERAL_REGS
944 : because SSE regs are unavailable. */
945 26466884 : for (i = 0; (cl2 = reg_class_subclasses[cl][i]) != LIM_REG_CLASSES; i++)
946 : {
947 20992830 : if (ira_class_hard_regs_num[cl2] == 0)
948 373 : continue;
949 2487570922 : for (m = 0; m < NUM_MACHINE_MODES; m++)
950 2467890704 : if (contains_reg_of_mode[cl][m] && contains_reg_of_mode[cl2][m])
951 : {
952 592923142 : ira_init_register_move_cost_if_necessary ((machine_mode) m);
953 592923142 : if (ira_register_move_cost[m][cl][cl]
954 592923142 : != ira_register_move_cost[m][cl2][cl2])
955 : break;
956 : }
957 20992457 : if (m < NUM_MACHINE_MODES)
958 : break;
959 : }
960 6786293 : if (cl2 == LIM_REG_CLASSES)
961 5474054 : ira_uniform_class_p[cl] = true;
962 : }
963 218792 : }
964 :
965 : /* Set up IRA_ALLOCNO_CLASSES, IRA_ALLOCNO_CLASSES_NUM,
966 : IRA_IMPORTANT_CLASSES, and IRA_IMPORTANT_CLASSES_NUM.
967 :
968 : Target may have many subtargets and not all target hard registers can
969 : be used for allocation, e.g. x86 port in 32-bit mode cannot use
970 : hard registers introduced in x86-64 like r8-r15). Some classes
971 : might have the same allocatable hard registers, e.g. INDEX_REGS
972 : and GENERAL_REGS in x86 port in 32-bit mode. To decrease different
973 : calculations efforts we introduce allocno classes which contain
974 : unique non-empty sets of allocatable hard-registers.
975 :
976 : Pseudo class cost calculation in ira-costs.cc is very expensive.
977 : Therefore we are trying to decrease number of classes involved in
978 : such calculation. Register classes used in the cost calculation
979 : are called important classes. They are allocno classes and other
980 : non-empty classes whose allocatable hard register sets are inside
981 : of an allocno class hard register set. From the first sight, it
982 : looks like that they are just allocno classes. It is not true. In
983 : example of x86-port in 32-bit mode, allocno classes will contain
984 : GENERAL_REGS but not LEGACY_REGS (because allocatable hard
985 : registers are the same for the both classes). The important
986 : classes will contain GENERAL_REGS and LEGACY_REGS. It is done
987 : because a machine description insn constraint may refers for
988 : LEGACY_REGS and code in ira-costs.cc is mostly base on investigation
989 : of the insn constraints. */
990 : static void
991 218792 : setup_allocno_and_important_classes (void)
992 : {
993 218792 : int i, j, n, cl;
994 218792 : bool set_p;
995 218792 : HARD_REG_SET temp_hard_regset2;
996 218792 : static enum reg_class classes[LIM_REG_CLASSES + 1];
997 :
998 218792 : n = 0;
999 : /* Collect classes which contain unique sets of allocatable hard
1000 : registers. Prefer GENERAL_REGS to other classes containing the
1001 : same set of hard registers. */
1002 7657720 : for (i = 0; i < LIM_REG_CLASSES; i++)
1003 : {
1004 7438928 : temp_hard_regset = reg_class_contents[i] & ~no_unit_alloc_regs;
1005 96807811 : for (j = 0; j < n; j++)
1006 : {
1007 91119693 : cl = classes[j];
1008 91119693 : temp_hard_regset2 = reg_class_contents[cl] & ~no_unit_alloc_regs;
1009 182239386 : if (temp_hard_regset == temp_hard_regset2)
1010 : break;
1011 : }
1012 7438928 : if (j >= n || targetm.additional_allocno_class_p (i))
1013 5688118 : classes[n++] = (enum reg_class) i;
1014 1750810 : else if (i == GENERAL_REGS)
1015 : /* Prefer general regs. For i386 example, it means that
1016 : we prefer GENERAL_REGS over INDEX_REGS or LEGACY_REGS
1017 : (all of them consists of the same available hard
1018 : registers). */
1019 5644 : classes[j] = (enum reg_class) i;
1020 : }
1021 218792 : classes[n] = LIM_REG_CLASSES;
1022 :
1023 : /* Set up classes which can be used for allocnos as classes
1024 : containing non-empty unique sets of allocatable hard
1025 : registers. */
1026 218792 : ira_allocno_classes_num = 0;
1027 5906910 : for (i = 0; (cl = classes[i]) != LIM_REG_CLASSES; i++)
1028 5688118 : if (ira_class_hard_regs_num[cl] > 0)
1029 5469326 : ira_allocno_classes[ira_allocno_classes_num++] = (enum reg_class) cl;
1030 218792 : ira_important_classes_num = 0;
1031 : /* Add non-allocno classes containing to non-empty set of
1032 : allocatable hard regs. */
1033 7657720 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1034 7438928 : if (ira_class_hard_regs_num[cl] > 0)
1035 : {
1036 6786293 : temp_hard_regset = reg_class_contents[cl] & ~no_unit_alloc_regs;
1037 6786293 : set_p = false;
1038 105341430 : for (j = 0; j < ira_allocno_classes_num; j++)
1039 : {
1040 104024463 : temp_hard_regset2 = (reg_class_contents[ira_allocno_classes[j]]
1041 104024463 : & ~no_unit_alloc_regs);
1042 104024463 : if ((enum reg_class) cl == ira_allocno_classes[j])
1043 : break;
1044 98555137 : else if (hard_reg_set_subset_p (temp_hard_regset,
1045 : temp_hard_regset2))
1046 6852070 : set_p = true;
1047 : }
1048 6786293 : if (set_p && j >= ira_allocno_classes_num)
1049 1316967 : ira_important_classes[ira_important_classes_num++]
1050 1316967 : = (enum reg_class) cl;
1051 : }
1052 : /* Now add allocno classes to the important classes. */
1053 5688118 : for (j = 0; j < ira_allocno_classes_num; j++)
1054 5469326 : ira_important_classes[ira_important_classes_num++]
1055 5469326 : = ira_allocno_classes[j];
1056 7657720 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1057 : {
1058 7438928 : ira_reg_allocno_class_p[cl] = false;
1059 7438928 : ira_reg_pressure_class_p[cl] = false;
1060 : }
1061 5688118 : for (j = 0; j < ira_allocno_classes_num; j++)
1062 5469326 : ira_reg_allocno_class_p[ira_allocno_classes[j]] = true;
1063 218792 : setup_pressure_classes ();
1064 218792 : setup_uniform_class_p ();
1065 218792 : }
1066 :
1067 : /* Setup translation in CLASS_TRANSLATE of all classes into a class
1068 : given by array CLASSES of length CLASSES_NUM. The function is used
1069 : make translation any reg class to an allocno class or to an
1070 : pressure class. This translation is necessary for some
1071 : calculations when we can use only allocno or pressure classes and
1072 : such translation represents an approximate representation of all
1073 : classes.
1074 :
1075 : The translation in case when allocatable hard register set of a
1076 : given class is subset of allocatable hard register set of a class
1077 : in CLASSES is pretty simple. We use smallest classes from CLASSES
1078 : containing a given class. If allocatable hard register set of a
1079 : given class is not a subset of any corresponding set of a class
1080 : from CLASSES, we use the cheapest (with load/store point of view)
1081 : class from CLASSES whose set intersects with given class set. */
1082 : static void
1083 437584 : setup_class_translate_array (enum reg_class *class_translate,
1084 : int classes_num, enum reg_class *classes)
1085 : {
1086 437584 : int cl, mode;
1087 437584 : enum reg_class aclass, best_class, *cl_ptr;
1088 437584 : int i, cost, min_cost, best_cost;
1089 :
1090 15315440 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1091 14877856 : class_translate[cl] = NO_REGS;
1092 :
1093 6784052 : for (i = 0; i < classes_num; i++)
1094 : {
1095 6346468 : aclass = classes[i];
1096 46601757 : for (cl_ptr = &alloc_reg_class_subclasses[aclass][0];
1097 46601757 : (cl = *cl_ptr) != LIM_REG_CLASSES;
1098 : cl_ptr++)
1099 40255289 : if (class_translate[cl] == NO_REGS)
1100 6198616 : class_translate[cl] = aclass;
1101 6346468 : class_translate[aclass] = aclass;
1102 : }
1103 : /* For classes which are not fully covered by one of given classes
1104 : (in other words covered by more one given class), use the
1105 : cheapest class. */
1106 15315440 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1107 : {
1108 14877856 : if (cl == NO_REGS || class_translate[cl] != NO_REGS)
1109 12913819 : continue;
1110 : best_class = NO_REGS;
1111 : best_cost = INT_MAX;
1112 18828380 : for (i = 0; i < classes_num; i++)
1113 : {
1114 16864343 : aclass = classes[i];
1115 16864343 : temp_hard_regset = (reg_class_contents[aclass]
1116 16864343 : & reg_class_contents[cl]
1117 16864343 : & ~no_unit_alloc_regs);
1118 33728686 : if (! hard_reg_set_empty_p (temp_hard_regset))
1119 : {
1120 : min_cost = INT_MAX;
1121 356369500 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
1122 : {
1123 353518544 : cost = (ira_memory_move_cost[mode][aclass][0]
1124 353518544 : + ira_memory_move_cost[mode][aclass][1]);
1125 353518544 : if (min_cost > cost)
1126 : min_cost = cost;
1127 : }
1128 2850956 : if (best_class == NO_REGS || best_cost > min_cost)
1129 : {
1130 16864343 : best_class = aclass;
1131 16864343 : best_cost = min_cost;
1132 : }
1133 : }
1134 : }
1135 1964037 : class_translate[cl] = best_class;
1136 : }
1137 437584 : }
1138 :
1139 : /* Set up array IRA_ALLOCNO_CLASS_TRANSLATE and
1140 : IRA_PRESSURE_CLASS_TRANSLATE. */
1141 : static void
1142 218792 : setup_class_translate (void)
1143 : {
1144 218792 : setup_class_translate_array (ira_allocno_class_translate,
1145 218792 : ira_allocno_classes_num, ira_allocno_classes);
1146 218792 : setup_class_translate_array (ira_pressure_class_translate,
1147 218792 : ira_pressure_classes_num, ira_pressure_classes);
1148 218792 : }
1149 :
1150 : /* Order numbers of allocno classes in original target allocno class
1151 : array, -1 for non-allocno classes. */
1152 : static int allocno_class_order[N_REG_CLASSES];
1153 :
1154 : /* The function used to sort the important classes. */
1155 : static int
1156 182415764 : comp_reg_classes_func (const void *v1p, const void *v2p)
1157 : {
1158 182415764 : enum reg_class cl1 = *(const enum reg_class *) v1p;
1159 182415764 : enum reg_class cl2 = *(const enum reg_class *) v2p;
1160 182415764 : enum reg_class tcl1, tcl2;
1161 182415764 : int diff;
1162 :
1163 182415764 : tcl1 = ira_allocno_class_translate[cl1];
1164 182415764 : tcl2 = ira_allocno_class_translate[cl2];
1165 182415764 : if (tcl1 != NO_REGS && tcl2 != NO_REGS
1166 182415764 : && (diff = allocno_class_order[tcl1] - allocno_class_order[tcl2]) != 0)
1167 : return diff;
1168 8258708 : return (int) cl1 - (int) cl2;
1169 : }
1170 :
1171 : /* For correct work of function setup_reg_class_relation we need to
1172 : reorder important classes according to the order of their allocno
1173 : classes. It places important classes containing the same
1174 : allocatable hard register set adjacent to each other and allocno
1175 : class with the allocatable hard register set right after the other
1176 : important classes with the same set.
1177 :
1178 : In example from comments of function
1179 : setup_allocno_and_important_classes, it places LEGACY_REGS and
1180 : GENERAL_REGS close to each other and GENERAL_REGS is after
1181 : LEGACY_REGS. */
1182 : static void
1183 218792 : reorder_important_classes (void)
1184 : {
1185 218792 : int i;
1186 :
1187 7657720 : for (i = 0; i < N_REG_CLASSES; i++)
1188 7438928 : allocno_class_order[i] = -1;
1189 5688118 : for (i = 0; i < ira_allocno_classes_num; i++)
1190 5469326 : allocno_class_order[ira_allocno_classes[i]] = i;
1191 218792 : qsort (ira_important_classes, ira_important_classes_num,
1192 : sizeof (enum reg_class), comp_reg_classes_func);
1193 7223877 : for (i = 0; i < ira_important_classes_num; i++)
1194 6786293 : ira_important_class_nums[ira_important_classes[i]] = i;
1195 218792 : }
1196 :
1197 : /* Set up IRA_REG_CLASS_SUBUNION, IRA_REG_CLASS_SUPERUNION,
1198 : IRA_REG_CLASS_SUPER_CLASSES, IRA_REG_CLASSES_INTERSECT, and
1199 : IRA_REG_CLASSES_INTERSECT_P. For the meaning of the relations,
1200 : please see corresponding comments in ira-int.h. */
1201 : static void
1202 218792 : setup_reg_class_relations (void)
1203 : {
1204 218792 : int i, cl1, cl2, cl3;
1205 218792 : HARD_REG_SET intersection_set, union_set, temp_set2;
1206 218792 : bool important_class_p[N_REG_CLASSES];
1207 :
1208 218792 : memset (important_class_p, 0, sizeof (important_class_p));
1209 7005085 : for (i = 0; i < ira_important_classes_num; i++)
1210 6786293 : important_class_p[ira_important_classes[i]] = true;
1211 7657720 : for (cl1 = 0; cl1 < N_REG_CLASSES; cl1++)
1212 : {
1213 7438928 : ira_reg_class_super_classes[cl1][0] = LIM_REG_CLASSES;
1214 260362480 : for (cl2 = 0; cl2 < N_REG_CLASSES; cl2++)
1215 : {
1216 252923552 : ira_reg_classes_intersect_p[cl1][cl2] = false;
1217 252923552 : ira_reg_class_intersect[cl1][cl2] = NO_REGS;
1218 252923552 : ira_reg_class_subset[cl1][cl2] = NO_REGS;
1219 252923552 : temp_hard_regset = reg_class_contents[cl1] & ~no_unit_alloc_regs;
1220 252923552 : temp_set2 = reg_class_contents[cl2] & ~no_unit_alloc_regs;
1221 252923552 : if (hard_reg_set_empty_p (temp_hard_regset)
1222 275113142 : && hard_reg_set_empty_p (temp_set2))
1223 : {
1224 : /* The both classes have no allocatable hard registers
1225 : -- take all class hard registers into account and use
1226 : reg_class_subunion and reg_class_superunion. */
1227 743778 : for (i = 0;; i++)
1228 : {
1229 2755943 : cl3 = reg_class_subclasses[cl1][i];
1230 2755943 : if (cl3 == LIM_REG_CLASSES)
1231 : break;
1232 743778 : if (reg_class_subset_p (ira_reg_class_intersect[cl1][cl2],
1233 : (enum reg_class) cl3))
1234 706528 : ira_reg_class_intersect[cl1][cl2] = (enum reg_class) cl3;
1235 : }
1236 2012165 : ira_reg_class_subunion[cl1][cl2] = reg_class_subunion[cl1][cl2];
1237 2012165 : ira_reg_class_superunion[cl1][cl2] = reg_class_superunion[cl1][cl2];
1238 2012165 : continue;
1239 : }
1240 : ira_reg_classes_intersect_p[cl1][cl2]
1241 250911387 : = hard_reg_set_intersect_p (temp_hard_regset, temp_set2);
1242 230733962 : if (important_class_p[cl1] && important_class_p[cl2]
1243 461467924 : && hard_reg_set_subset_p (temp_hard_regset, temp_set2))
1244 : {
1245 : /* CL1 and CL2 are important classes and CL1 allocatable
1246 : hard register set is inside of CL2 allocatable hard
1247 : registers -- make CL1 a superset of CL2. */
1248 60081722 : enum reg_class *p;
1249 :
1250 60081722 : p = &ira_reg_class_super_classes[cl1][0];
1251 362619458 : while (*p != LIM_REG_CLASSES)
1252 302537736 : p++;
1253 60081722 : *p++ = (enum reg_class) cl2;
1254 60081722 : *p = LIM_REG_CLASSES;
1255 : }
1256 250911387 : ira_reg_class_subunion[cl1][cl2] = NO_REGS;
1257 250911387 : ira_reg_class_superunion[cl1][cl2] = NO_REGS;
1258 250911387 : intersection_set = (reg_class_contents[cl1]
1259 250911387 : & reg_class_contents[cl2]
1260 250911387 : & ~no_unit_alloc_regs);
1261 250911387 : union_set = ((reg_class_contents[cl1] | reg_class_contents[cl2])
1262 250911387 : & ~no_unit_alloc_regs);
1263 8781898545 : for (cl3 = 0; cl3 < N_REG_CLASSES; cl3++)
1264 : {
1265 8530987158 : temp_hard_regset = reg_class_contents[cl3] & ~no_unit_alloc_regs;
1266 17061974316 : if (hard_reg_set_empty_p (temp_hard_regset))
1267 747707355 : continue;
1268 :
1269 7783279803 : if (hard_reg_set_subset_p (temp_hard_regset, intersection_set))
1270 : {
1271 : /* CL3 allocatable hard register set is inside of
1272 : intersection of allocatable hard register sets
1273 : of CL1 and CL2. */
1274 665157194 : if (important_class_p[cl3])
1275 : {
1276 665157194 : temp_set2
1277 665157194 : = (reg_class_contents
1278 665157194 : [ira_reg_class_intersect[cl1][cl2]]);
1279 665157194 : temp_set2 &= ~no_unit_alloc_regs;
1280 665157194 : if (! hard_reg_set_subset_p (temp_hard_regset, temp_set2)
1281 : /* If the allocatable hard register sets are
1282 : the same, prefer GENERAL_REGS or the
1283 : smallest class for debugging
1284 : purposes. */
1285 773190852 : || (temp_hard_regset == temp_set2
1286 103734117 : && (cl3 == GENERAL_REGS
1287 103051187 : || ((ira_reg_class_intersect[cl1][cl2]
1288 : != GENERAL_REGS)
1289 34507129 : && hard_reg_set_subset_p
1290 34507129 : (reg_class_contents[cl3],
1291 : reg_class_contents
1292 : [(int)
1293 : ira_reg_class_intersect[cl1][cl2]])))))
1294 584282691 : ira_reg_class_intersect[cl1][cl2] = (enum reg_class) cl3;
1295 : }
1296 665157194 : temp_set2
1297 665157194 : = (reg_class_contents[ira_reg_class_subset[cl1][cl2]]
1298 665157194 : & ~no_unit_alloc_regs);
1299 665157194 : if (! hard_reg_set_subset_p (temp_hard_regset, temp_set2)
1300 : /* Ignore unavailable hard registers and prefer
1301 : smallest class for debugging purposes. */
1302 773190852 : || (temp_hard_regset == temp_set2
1303 103734117 : && hard_reg_set_subset_p
1304 103734117 : (reg_class_contents[cl3],
1305 : reg_class_contents
1306 : [(int) ira_reg_class_subset[cl1][cl2]])))
1307 618120019 : ira_reg_class_subset[cl1][cl2] = (enum reg_class) cl3;
1308 : }
1309 7783279803 : if (important_class_p[cl3]
1310 15566559606 : && hard_reg_set_subset_p (temp_hard_regset, union_set))
1311 : {
1312 : /* CL3 allocatable hard register set is inside of
1313 : union of allocatable hard register sets of CL1
1314 : and CL2. */
1315 3464507166 : temp_set2
1316 3464507166 : = (reg_class_contents[ira_reg_class_subunion[cl1][cl2]]
1317 3464507166 : & ~no_unit_alloc_regs);
1318 3464507166 : if (ira_reg_class_subunion[cl1][cl2] == NO_REGS
1319 6678102945 : || (hard_reg_set_subset_p (temp_set2, temp_hard_regset)
1320 :
1321 1069441547 : && (temp_set2 != temp_hard_regset
1322 445237001 : || cl3 == GENERAL_REGS
1323 : /* If the allocatable hard register sets are the
1324 : same, prefer GENERAL_REGS or the smallest
1325 : class for debugging purposes. */
1326 441641809 : || (ira_reg_class_subunion[cl1][cl2] != GENERAL_REGS
1327 29311315 : && hard_reg_set_subset_p
1328 29311315 : (reg_class_contents[cl3],
1329 : reg_class_contents
1330 : [(int) ira_reg_class_subunion[cl1][cl2]])))))
1331 901506684 : ira_reg_class_subunion[cl1][cl2] = (enum reg_class) cl3;
1332 : }
1333 15566559606 : if (hard_reg_set_subset_p (union_set, temp_hard_regset))
1334 : {
1335 : /* CL3 allocatable hard register set contains union
1336 : of allocatable hard register sets of CL1 and
1337 : CL2. */
1338 1443551976 : temp_set2
1339 1443551976 : = (reg_class_contents[ira_reg_class_superunion[cl1][cl2]]
1340 1443551976 : & ~no_unit_alloc_regs);
1341 1443551976 : if (ira_reg_class_superunion[cl1][cl2] == NO_REGS
1342 2636192565 : || (hard_reg_set_subset_p (temp_hard_regset, temp_set2)
1343 :
1344 204810944 : && (temp_set2 != temp_hard_regset
1345 203907740 : || cl3 == GENERAL_REGS
1346 : /* If the allocatable hard register sets are the
1347 : same, prefer GENERAL_REGS or the smallest
1348 : class for debugging purposes. */
1349 202345500 : || (ira_reg_class_superunion[cl1][cl2] != GENERAL_REGS
1350 20178460 : && hard_reg_set_subset_p
1351 20178460 : (reg_class_contents[cl3],
1352 : reg_class_contents
1353 : [(int) ira_reg_class_superunion[cl1][cl2]])))))
1354 267870300 : ira_reg_class_superunion[cl1][cl2] = (enum reg_class) cl3;
1355 : }
1356 : }
1357 : }
1358 : }
1359 218792 : }
1360 :
1361 : /* Output all uniform and important classes into file F. */
1362 : static void
1363 0 : print_uniform_and_important_classes (FILE *f)
1364 : {
1365 0 : int i, cl;
1366 :
1367 0 : fprintf (f, "Uniform classes:\n");
1368 0 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1369 0 : if (ira_uniform_class_p[cl])
1370 0 : fprintf (f, " %s", reg_class_names[cl]);
1371 0 : fprintf (f, "\nImportant classes:\n");
1372 0 : for (i = 0; i < ira_important_classes_num; i++)
1373 0 : fprintf (f, " %s", reg_class_names[ira_important_classes[i]]);
1374 0 : fprintf (f, "\n");
1375 0 : }
1376 :
1377 : /* Output all possible allocno or pressure classes and their
1378 : translation map into file F. */
1379 : static void
1380 0 : print_translated_classes (FILE *f, bool pressure_p)
1381 : {
1382 0 : int classes_num = (pressure_p
1383 0 : ? ira_pressure_classes_num : ira_allocno_classes_num);
1384 0 : enum reg_class *classes = (pressure_p
1385 0 : ? ira_pressure_classes : ira_allocno_classes);
1386 0 : enum reg_class *class_translate = (pressure_p
1387 : ? ira_pressure_class_translate
1388 : : ira_allocno_class_translate);
1389 0 : int i;
1390 :
1391 0 : fprintf (f, "%s classes:\n", pressure_p ? "Pressure" : "Allocno");
1392 0 : for (i = 0; i < classes_num; i++)
1393 0 : fprintf (f, " %s", reg_class_names[classes[i]]);
1394 0 : fprintf (f, "\nClass translation:\n");
1395 0 : for (i = 0; i < N_REG_CLASSES; i++)
1396 0 : fprintf (f, " %s -> %s\n", reg_class_names[i],
1397 0 : reg_class_names[class_translate[i]]);
1398 0 : }
1399 :
1400 : /* Output all possible allocno and translation classes and the
1401 : translation maps into stderr. */
1402 : void
1403 0 : ira_debug_allocno_classes (void)
1404 : {
1405 0 : print_uniform_and_important_classes (stderr);
1406 0 : print_translated_classes (stderr, false);
1407 0 : print_translated_classes (stderr, true);
1408 0 : }
1409 :
1410 : /* Set up different arrays concerning class subsets, allocno and
1411 : important classes. */
1412 : static void
1413 218792 : find_reg_classes (void)
1414 : {
1415 218792 : setup_allocno_and_important_classes ();
1416 218792 : setup_class_translate ();
1417 218792 : reorder_important_classes ();
1418 218792 : setup_reg_class_relations ();
1419 218792 : }
1420 :
1421 :
1422 :
1423 : /* Set up array ira_hard_regno_allocno_class. */
1424 : static void
1425 218792 : setup_hard_regno_aclass (void)
1426 : {
1427 218792 : int i;
1428 :
1429 20347656 : for (i = 0; i < FIRST_PSEUDO_REGISTER; i++)
1430 : {
1431 40257728 : ira_hard_regno_allocno_class[i]
1432 30347262 : = (TEST_HARD_REG_BIT (no_unit_alloc_regs, i)
1433 20128864 : ? NO_REGS
1434 10218398 : : ira_allocno_class_translate[REGNO_REG_CLASS (i)]);
1435 : }
1436 218792 : }
1437 :
1438 :
1439 :
1440 : /* Form IRA_REG_CLASS_MAX_NREGS and IRA_REG_CLASS_MIN_NREGS maps. */
1441 : static void
1442 218792 : setup_reg_class_nregs (void)
1443 : {
1444 218792 : int i, cl, cl2, m;
1445 :
1446 27349000 : for (m = 0; m < MAX_MACHINE_MODE; m++)
1447 : {
1448 949557280 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1449 1844854144 : ira_reg_class_max_nregs[cl][m]
1450 1844854144 : = ira_reg_class_min_nregs[cl][m]
1451 922427072 : = targetm.class_max_nregs ((reg_class_t) cl, (machine_mode) m);
1452 949557280 : for (cl = 0; cl < N_REG_CLASSES; cl++)
1453 6608633196 : for (i = 0;
1454 7531060268 : (cl2 = alloc_reg_class_subclasses[cl][i]) != LIM_REG_CLASSES;
1455 : i++)
1456 6608633196 : if (ira_reg_class_min_nregs[cl2][m]
1457 6608633196 : < ira_reg_class_min_nregs[cl][m])
1458 53051424 : ira_reg_class_min_nregs[cl][m] = ira_reg_class_min_nregs[cl2][m];
1459 : }
1460 218792 : }
1461 :
1462 :
1463 :
1464 : /* Set up IRA_PROHIBITED_CLASS_MODE_REGS, IRA_EXCLUDE_CLASS_MODE_REGS, and
1465 : IRA_CLASS_SINGLETON. This function is called once IRA_CLASS_HARD_REGS has
1466 : been initialized. */
1467 : static void
1468 218792 : setup_prohibited_and_exclude_class_mode_regs (void)
1469 : {
1470 218792 : int j, k, hard_regno, cl, last_hard_regno, count;
1471 :
1472 7657720 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
1473 : {
1474 7438928 : temp_hard_regset = reg_class_contents[cl] & ~no_unit_alloc_regs;
1475 929866000 : for (j = 0; j < NUM_MACHINE_MODES; j++)
1476 : {
1477 922427072 : count = 0;
1478 922427072 : last_hard_regno = -1;
1479 3689708288 : CLEAR_HARD_REG_SET (ira_prohibited_class_mode_regs[cl][j]);
1480 922427072 : CLEAR_HARD_REG_SET (ira_exclude_class_mode_regs[cl][j]);
1481 10270514148 : for (k = ira_class_hard_regs_num[cl] - 1; k >= 0; k--)
1482 : {
1483 9348087076 : hard_regno = ira_class_hard_regs[cl][k];
1484 9348087076 : if (!targetm.hard_regno_mode_ok (hard_regno, (machine_mode) j))
1485 7047412498 : SET_HARD_REG_BIT (ira_prohibited_class_mode_regs[cl][j],
1486 : hard_regno);
1487 2300674578 : else if (in_hard_reg_set_p (temp_hard_regset,
1488 : (machine_mode) j, hard_regno))
1489 : {
1490 2203357605 : last_hard_regno = hard_regno;
1491 2203357605 : count++;
1492 : }
1493 : else
1494 : {
1495 97316973 : SET_HARD_REG_BIT (ira_exclude_class_mode_regs[cl][j], hard_regno);
1496 : }
1497 : }
1498 970283246 : ira_class_singleton[cl][j] = (count == 1 ? last_hard_regno : -1);
1499 : }
1500 : }
1501 218792 : }
1502 :
1503 : /* Clarify IRA_PROHIBITED_CLASS_MODE_REGS by excluding hard registers
1504 : spanning from one register pressure class to another one. It is
1505 : called after defining the pressure classes. */
1506 : static void
1507 218792 : clarify_prohibited_class_mode_regs (void)
1508 : {
1509 218792 : int j, k, hard_regno, cl, pclass, nregs;
1510 :
1511 7657720 : for (cl = (int) N_REG_CLASSES - 1; cl >= 0; cl--)
1512 929866000 : for (j = 0; j < NUM_MACHINE_MODES; j++)
1513 : {
1514 922427072 : CLEAR_HARD_REG_SET (ira_useful_class_mode_regs[cl][j]);
1515 10270514148 : for (k = ira_class_hard_regs_num[cl] - 1; k >= 0; k--)
1516 : {
1517 9348087076 : hard_regno = ira_class_hard_regs[cl][k];
1518 9348087076 : if (TEST_HARD_REG_BIT (ira_prohibited_class_mode_regs[cl][j], hard_regno))
1519 7047412498 : continue;
1520 2300674578 : nregs = hard_regno_nregs (hard_regno, (machine_mode) j);
1521 2300674578 : if (hard_regno + nregs > FIRST_PSEUDO_REGISTER)
1522 : {
1523 10320 : SET_HARD_REG_BIT (ira_prohibited_class_mode_regs[cl][j],
1524 : hard_regno);
1525 10320 : continue;
1526 : }
1527 2300664258 : pclass = ira_pressure_class_translate[REGNO_REG_CLASS (hard_regno)];
1528 5054773301 : for (nregs-- ;nregs >= 0; nregs--)
1529 2805257749 : if (((enum reg_class) pclass
1530 2805257749 : != ira_pressure_class_translate[REGNO_REG_CLASS
1531 2805257749 : (hard_regno + nregs)]))
1532 : {
1533 51148706 : SET_HARD_REG_BIT (ira_prohibited_class_mode_regs[cl][j],
1534 : hard_regno);
1535 51148706 : break;
1536 : }
1537 2300664258 : if (!TEST_HARD_REG_BIT (ira_prohibited_class_mode_regs[cl][j],
1538 : hard_regno))
1539 2249515552 : add_to_hard_reg_set (&ira_useful_class_mode_regs[cl][j],
1540 : (machine_mode) j, hard_regno);
1541 : }
1542 : }
1543 218792 : }
1544 :
1545 : /* Allocate and initialize IRA_REGISTER_MOVE_COST, IRA_MAY_MOVE_IN_COST
1546 : and IRA_MAY_MOVE_OUT_COST for MODE. */
1547 : void
1548 10203609 : ira_init_register_move_cost (machine_mode mode)
1549 : {
1550 10203609 : static unsigned short last_move_cost[N_REG_CLASSES][N_REG_CLASSES];
1551 10203609 : bool all_match = true;
1552 10203609 : unsigned int i, cl1, cl2;
1553 10203609 : HARD_REG_SET ok_regs;
1554 :
1555 10203609 : ira_assert (ira_register_move_cost[mode] == NULL
1556 : && ira_may_move_in_cost[mode] == NULL
1557 : && ira_may_move_out_cost[mode] == NULL);
1558 948935637 : CLEAR_HARD_REG_SET (ok_regs);
1559 948935637 : for (i = 0; i < FIRST_PSEUDO_REGISTER; i++)
1560 938732028 : if (targetm.hard_regno_mode_ok (i, mode))
1561 426877936 : SET_HARD_REG_BIT (ok_regs, i);
1562 :
1563 : /* Note that we might be asked about the move costs of modes that
1564 : cannot be stored in any hard register, for example if an inline
1565 : asm tries to create a register operand with an impossible mode.
1566 : We therefore can't assert have_regs_of_mode[mode] here. */
1567 357126315 : for (cl1 = 0; cl1 < N_REG_CLASSES; cl1++)
1568 12142294710 : for (cl2 = 0; cl2 < N_REG_CLASSES; cl2++)
1569 : {
1570 11795372004 : int cost;
1571 11795372004 : if (!hard_reg_set_intersect_p (ok_regs, reg_class_contents[cl1])
1572 19500606160 : || !hard_reg_set_intersect_p (ok_regs, reg_class_contents[cl2]))
1573 : {
1574 6159932160 : if ((ira_reg_class_max_nregs[cl1][mode]
1575 6159932160 : > ira_class_hard_regs_num[cl1])
1576 4371775055 : || (ira_reg_class_max_nregs[cl2][mode]
1577 4371775055 : > ira_class_hard_regs_num[cl2]))
1578 : cost = 65535;
1579 : else
1580 3007898287 : cost = (ira_memory_move_cost[mode][cl1][0]
1581 3007898287 : + ira_memory_move_cost[mode][cl2][1]) * 2;
1582 : }
1583 : else
1584 : {
1585 5635439844 : cost = register_move_cost (mode, (enum reg_class) cl1,
1586 : (enum reg_class) cl2);
1587 5635439844 : ira_assert (cost < 65535);
1588 : }
1589 11795372004 : all_match &= (last_move_cost[cl1][cl2] == cost);
1590 11795372004 : last_move_cost[cl1][cl2] = cost;
1591 : }
1592 10203609 : if (all_match && last_mode_for_init_move_cost != -1)
1593 : {
1594 4235720 : ira_register_move_cost[mode]
1595 4235720 : = ira_register_move_cost[last_mode_for_init_move_cost];
1596 4235720 : ira_may_move_in_cost[mode]
1597 4235720 : = ira_may_move_in_cost[last_mode_for_init_move_cost];
1598 4235720 : ira_may_move_out_cost[mode]
1599 4235720 : = ira_may_move_out_cost[last_mode_for_init_move_cost];
1600 4235720 : return;
1601 : }
1602 5967889 : last_mode_for_init_move_cost = mode;
1603 5967889 : ira_register_move_cost[mode] = XNEWVEC (move_table, N_REG_CLASSES);
1604 5967889 : ira_may_move_in_cost[mode] = XNEWVEC (move_table, N_REG_CLASSES);
1605 5967889 : ira_may_move_out_cost[mode] = XNEWVEC (move_table, N_REG_CLASSES);
1606 208876115 : for (cl1 = 0; cl1 < N_REG_CLASSES; cl1++)
1607 7101787910 : for (cl2 = 0; cl2 < N_REG_CLASSES; cl2++)
1608 : {
1609 6898879684 : int cost;
1610 6898879684 : enum reg_class *p1, *p2;
1611 :
1612 6898879684 : if (last_move_cost[cl1][cl2] == 65535)
1613 : {
1614 1722333776 : ira_register_move_cost[mode][cl1][cl2] = 65535;
1615 1722333776 : ira_may_move_in_cost[mode][cl1][cl2] = 65535;
1616 1722333776 : ira_may_move_out_cost[mode][cl1][cl2] = 65535;
1617 : }
1618 : else
1619 : {
1620 5176545908 : cost = last_move_cost[cl1][cl2];
1621 :
1622 43376537418 : for (p2 = ®_class_subclasses[cl2][0];
1623 43376537418 : *p2 != LIM_REG_CLASSES; p2++)
1624 38199991510 : if (ira_class_hard_regs_num[*p2] > 0
1625 37497822343 : && (ira_reg_class_max_nregs[*p2][mode]
1626 : <= ira_class_hard_regs_num[*p2]))
1627 30755133173 : cost = MAX (cost, ira_register_move_cost[mode][cl1][*p2]);
1628 :
1629 43376537418 : for (p1 = ®_class_subclasses[cl1][0];
1630 43376537418 : *p1 != LIM_REG_CLASSES; p1++)
1631 38199991510 : if (ira_class_hard_regs_num[*p1] > 0
1632 37497822343 : && (ira_reg_class_max_nregs[*p1][mode]
1633 : <= ira_class_hard_regs_num[*p1]))
1634 30755133173 : cost = MAX (cost, ira_register_move_cost[mode][*p1][cl2]);
1635 :
1636 5176545908 : ira_assert (cost <= 65535);
1637 5176545908 : ira_register_move_cost[mode][cl1][cl2] = cost;
1638 :
1639 5176545908 : if (ira_class_subset_p[cl1][cl2])
1640 1562444828 : ira_may_move_in_cost[mode][cl1][cl2] = 0;
1641 : else
1642 3614101080 : ira_may_move_in_cost[mode][cl1][cl2] = cost;
1643 :
1644 5176545908 : if (ira_class_subset_p[cl2][cl1])
1645 1562444828 : ira_may_move_out_cost[mode][cl1][cl2] = 0;
1646 : else
1647 3614101080 : ira_may_move_out_cost[mode][cl1][cl2] = cost;
1648 : }
1649 : }
1650 : }
1651 :
1652 :
1653 :
1654 : /* This is called once during compiler work. It sets up
1655 : different arrays whose values don't depend on the compiled
1656 : function. */
1657 : void
1658 214556 : ira_init_once (void)
1659 : {
1660 214556 : ira_init_costs_once ();
1661 214556 : lra_init_once ();
1662 :
1663 214556 : ira_use_lra_p = targetm.lra_p ();
1664 214556 : }
1665 :
1666 : /* Free ira_max_register_move_cost, ira_may_move_in_cost and
1667 : ira_may_move_out_cost for each mode. */
1668 : void
1669 540020 : target_ira_int::free_register_move_costs (void)
1670 : {
1671 540020 : int mode, i;
1672 :
1673 : /* Reset move_cost and friends, making sure we only free shared
1674 : table entries once. */
1675 67502500 : for (mode = 0; mode < MAX_MACHINE_MODE; mode++)
1676 66962480 : if (x_ira_register_move_cost[mode])
1677 : {
1678 613536830 : for (i = 0;
1679 622963748 : i < mode && (x_ira_register_move_cost[i]
1680 : != x_ira_register_move_cost[mode]);
1681 : i++)
1682 : ;
1683 9426918 : if (i == mode)
1684 : {
1685 5516618 : free (x_ira_register_move_cost[mode]);
1686 5516618 : free (x_ira_may_move_in_cost[mode]);
1687 5516618 : free (x_ira_may_move_out_cost[mode]);
1688 : }
1689 : }
1690 540020 : memset (x_ira_register_move_cost, 0, sizeof x_ira_register_move_cost);
1691 540020 : memset (x_ira_may_move_in_cost, 0, sizeof x_ira_may_move_in_cost);
1692 540020 : memset (x_ira_may_move_out_cost, 0, sizeof x_ira_may_move_out_cost);
1693 540020 : last_mode_for_init_move_cost = -1;
1694 540020 : }
1695 :
1696 321228 : target_ira_int::~target_ira_int ()
1697 : {
1698 321228 : free_ira_costs ();
1699 321228 : free_register_move_costs ();
1700 321228 : }
1701 :
1702 : /* This is called every time when register related information is
1703 : changed. */
1704 : void
1705 218792 : ira_init (void)
1706 : {
1707 218792 : this_target_ira_int->free_register_move_costs ();
1708 218792 : setup_reg_mode_hard_regset ();
1709 218792 : setup_alloc_regs (flag_omit_frame_pointer != 0);
1710 218792 : setup_class_subset_and_memory_move_costs ();
1711 218792 : setup_reg_class_nregs ();
1712 218792 : setup_prohibited_and_exclude_class_mode_regs ();
1713 218792 : find_reg_classes ();
1714 218792 : clarify_prohibited_class_mode_regs ();
1715 218792 : setup_hard_regno_aclass ();
1716 218792 : ira_init_costs ();
1717 218792 : }
1718 :
1719 :
1720 : #define ira_prohibited_mode_move_regs_initialized_p \
1721 : (this_target_ira_int->x_ira_prohibited_mode_move_regs_initialized_p)
1722 :
1723 : /* Set up IRA_PROHIBITED_MODE_MOVE_REGS. */
1724 : static void
1725 1504950 : setup_prohibited_mode_move_regs (void)
1726 : {
1727 1504950 : int i, j;
1728 1504950 : rtx test_reg1, test_reg2, move_pat;
1729 1504950 : rtx_insn *move_insn;
1730 :
1731 1504950 : if (ira_prohibited_mode_move_regs_initialized_p)
1732 : return;
1733 217519 : ira_prohibited_mode_move_regs_initialized_p = true;
1734 217519 : test_reg1 = gen_rtx_REG (word_mode, LAST_VIRTUAL_REGISTER + 1);
1735 217519 : test_reg2 = gen_rtx_REG (word_mode, LAST_VIRTUAL_REGISTER + 2);
1736 217519 : move_pat = gen_rtx_SET (test_reg1, test_reg2);
1737 217519 : move_insn = gen_rtx_INSN (VOIDmode, 0, 0, 0, move_pat, 0, -1, 0);
1738 27407394 : for (i = 0; i < NUM_MACHINE_MODES; i++)
1739 : {
1740 26972356 : SET_HARD_REG_SET (ira_prohibited_mode_move_regs[i]);
1741 2508429108 : for (j = 0; j < FIRST_PSEUDO_REGISTER; j++)
1742 : {
1743 2481456752 : if (!targetm.hard_regno_mode_ok (j, (machine_mode) i))
1744 2052652055 : continue;
1745 428804697 : set_mode_and_regno (test_reg1, (machine_mode) i, j);
1746 428804697 : set_mode_and_regno (test_reg2, (machine_mode) i, j);
1747 428804697 : INSN_CODE (move_insn) = -1;
1748 428804697 : recog_memoized (move_insn);
1749 428804697 : if (INSN_CODE (move_insn) < 0)
1750 202890739 : continue;
1751 225913958 : extract_insn (move_insn);
1752 : /* We don't know whether the move will be in code that is optimized
1753 : for size or speed, so consider all enabled alternatives. */
1754 225913958 : if (! constrain_operands (1, get_enabled_alternatives (move_insn)))
1755 1335100 : continue;
1756 224578858 : CLEAR_HARD_REG_BIT (ira_prohibited_mode_move_regs[i], j);
1757 : }
1758 : }
1759 : }
1760 :
1761 :
1762 :
1763 : /* Extract INSN and return the set of alternatives that we should consider.
1764 : This excludes any alternatives whose constraints are obviously impossible
1765 : to meet (e.g. because the constraint requires a constant and the operand
1766 : is nonconstant). It also excludes alternatives that are bound to need
1767 : a spill or reload, as long as we have other alternatives that match
1768 : exactly. */
1769 : alternative_mask
1770 104910053 : ira_setup_alts (rtx_insn *insn)
1771 : {
1772 104910053 : int nop, nalt;
1773 104910053 : bool curr_swapped;
1774 104910053 : const char *p;
1775 104910053 : int commutative = -1;
1776 :
1777 104910053 : extract_insn (insn);
1778 104910053 : preprocess_constraints (insn);
1779 104910053 : alternative_mask preferred = get_preferred_alternatives (insn);
1780 104910053 : alternative_mask alts = 0;
1781 104910053 : alternative_mask exact_alts = 0;
1782 : /* Check that the hard reg set is enough for holding all
1783 : alternatives. It is hard to imagine the situation when the
1784 : assertion is wrong. */
1785 104910053 : ira_assert (recog_data.n_alternatives
1786 : <= (int) MAX (sizeof (HARD_REG_ELT_TYPE) * CHAR_BIT,
1787 : FIRST_PSEUDO_REGISTER));
1788 308221780 : for (nop = 0; nop < recog_data.n_operands; nop++)
1789 215567710 : if (recog_data.constraints[nop][0] == '%')
1790 : {
1791 : commutative = nop;
1792 : break;
1793 : }
1794 104910053 : for (curr_swapped = false;; curr_swapped = true)
1795 : {
1796 1410945092 : for (nalt = 0; nalt < recog_data.n_alternatives; nalt++)
1797 : {
1798 1293779056 : if (!TEST_BIT (preferred, nalt) || TEST_BIT (exact_alts, nalt))
1799 447809910 : continue;
1800 :
1801 845969146 : const operand_alternative *op_alt
1802 845969146 : = &recog_op_alt[nalt * recog_data.n_operands];
1803 845969146 : int this_reject = 0;
1804 2416083308 : for (nop = 0; nop < recog_data.n_operands; nop++)
1805 : {
1806 1763494045 : int c, len;
1807 :
1808 1763494045 : this_reject += op_alt[nop].reject;
1809 :
1810 1763494045 : rtx op = recog_data.operand[nop];
1811 1763494045 : p = op_alt[nop].constraint;
1812 1763494045 : if (*p == 0 || *p == ',')
1813 24786242 : continue;
1814 :
1815 : bool win_p = false;
1816 3527844939 : do
1817 3527844939 : switch (c = *p, len = CONSTRAINT_LEN (c, p), c)
1818 : {
1819 : case '#':
1820 : case ',':
1821 : c = '\0';
1822 : /* FALLTHRU */
1823 740756784 : case '\0':
1824 740756784 : len = 0;
1825 740756784 : break;
1826 :
1827 : case '%':
1828 : /* The commutative modifier is handled above. */
1829 : break;
1830 :
1831 75102084 : case '0': case '1': case '2': case '3': case '4':
1832 75102084 : case '5': case '6': case '7': case '8': case '9':
1833 75102084 : {
1834 75102084 : char *end;
1835 75102084 : unsigned long dup = strtoul (p, &end, 10);
1836 75102084 : rtx other = recog_data.operand[dup];
1837 75102084 : len = end - p;
1838 1862827 : if (MEM_P (other)
1839 75102084 : ? rtx_equal_p (other, op)
1840 73239257 : : REG_P (op) || SUBREG_P (op))
1841 52290100 : goto op_success;
1842 22811984 : win_p = true;
1843 : }
1844 22811984 : break;
1845 :
1846 10723593 : case 'g':
1847 10723593 : goto op_success;
1848 145 : break;
1849 :
1850 145 : case '{':
1851 145 : if (REG_P (op) || SUBREG_P (op))
1852 143 : goto op_success;
1853 : win_p = true;
1854 : break;
1855 :
1856 2681926375 : default:
1857 2681926375 : {
1858 2681926375 : enum constraint_num cn = lookup_constraint (p);
1859 2681926375 : rtx mem = NULL;
1860 2681926375 : switch (get_constraint_type (cn))
1861 : {
1862 2187343058 : case CT_REGISTER:
1863 3444237542 : if (reg_class_for_constraint (cn) != NO_REGS)
1864 : {
1865 1209517522 : if (REG_P (op) || SUBREG_P (op))
1866 790375287 : goto op_success;
1867 : win_p = true;
1868 : }
1869 : break;
1870 :
1871 4139638 : case CT_CONST_INT:
1872 4139638 : if (CONST_INT_P (op)
1873 6791691 : && (insn_const_int_ok_for_constraint
1874 2652053 : (INTVAL (op), cn)))
1875 1913287 : goto op_success;
1876 : break;
1877 :
1878 794908 : case CT_ADDRESS:
1879 794908 : goto op_success;
1880 :
1881 165188593 : case CT_MEMORY:
1882 165188593 : case CT_RELAXED_MEMORY:
1883 165188593 : mem = op;
1884 : /* Fall through. */
1885 165188593 : case CT_SPECIAL_MEMORY:
1886 165188593 : if (!mem)
1887 68057055 : mem = extract_mem_from_operand (op);
1888 233245648 : if (MEM_P (mem))
1889 70387762 : goto op_success;
1890 : win_p = true;
1891 : break;
1892 :
1893 256403123 : case CT_FIXED_FORM:
1894 256403123 : if (constraint_satisfied_p (op, cn))
1895 71465939 : goto op_success;
1896 : break;
1897 : }
1898 : break;
1899 : }
1900 : }
1901 2529893920 : while (p += len, c);
1902 740756784 : if (!win_p)
1903 : break;
1904 : /* We can make the alternative match by spilling a register
1905 : to memory or loading something into a register. Count a
1906 : cost of one reload (the equivalent of the '?' constraint). */
1907 547376901 : this_reject += 6;
1908 1570114162 : op_success:
1909 1570114162 : ;
1910 : }
1911 :
1912 845969146 : if (nop >= recog_data.n_operands)
1913 : {
1914 652589263 : alts |= ALTERNATIVE_BIT (nalt);
1915 652589263 : if (this_reject == 0)
1916 138226072 : exact_alts |= ALTERNATIVE_BIT (nalt);
1917 : }
1918 : }
1919 117166036 : if (commutative < 0)
1920 : break;
1921 : /* Swap forth and back to avoid changing recog_data. */
1922 24511966 : std::swap (recog_data.operand[commutative],
1923 24511966 : recog_data.operand[commutative + 1]);
1924 24511966 : if (curr_swapped)
1925 : break;
1926 : }
1927 104910053 : return exact_alts ? exact_alts : alts;
1928 : }
1929 :
1930 : /* Return the number of the output non-early clobber operand which
1931 : should be the same in any case as operand with number OP_NUM (or
1932 : negative value if there is no such operand). ALTS is the mask
1933 : of alternatives that we should consider. SINGLE_INPUT_OP_HAS_CSTR_P
1934 : should be set in this function, it indicates whether there is only
1935 : a single input operand which has the matching constraint on the
1936 : output operand at the position specified in return value. If the
1937 : pattern allows any one of several input operands holds the matching
1938 : constraint, it's set as false, one typical case is destructive FMA
1939 : instruction on target rs6000. Note that for a non-NO_REG preferred
1940 : register class with no free register move copy, if the parameter
1941 : PARAM_IRA_CONSIDER_DUP_IN_ALL_ALTS is set to one, this function
1942 : will check all available alternatives for matching constraints,
1943 : even if it has found or will find one alternative with non-NO_REG
1944 : regclass, it can respect more cases with matching constraints. If
1945 : PARAM_IRA_CONSIDER_DUP_IN_ALL_ALTS is set to zero,
1946 : SINGLE_INPUT_OP_HAS_CSTR_P is always true, it will stop to find
1947 : matching constraint relationship once it hits some alternative with
1948 : some non-NO_REG regclass. */
1949 : int
1950 20854159 : ira_get_dup_out_num (int op_num, alternative_mask alts,
1951 : bool &single_input_op_has_cstr_p)
1952 : {
1953 20854159 : int curr_alt, c, original;
1954 20854159 : bool ignore_p, use_commut_op_p;
1955 20854159 : const char *str;
1956 :
1957 20854159 : if (op_num < 0 || recog_data.n_alternatives == 0)
1958 : return -1;
1959 : /* We should find duplications only for input operands. */
1960 20854159 : if (recog_data.operand_type[op_num] != OP_IN)
1961 : return -1;
1962 15039228 : str = recog_data.constraints[op_num];
1963 15039228 : use_commut_op_p = false;
1964 15039228 : single_input_op_has_cstr_p = true;
1965 :
1966 15039228 : rtx op = recog_data.operand[op_num];
1967 15039228 : int op_regno = reg_or_subregno (op);
1968 15039228 : enum reg_class op_pref_cl = reg_preferred_class (op_regno);
1969 15039228 : machine_mode op_mode = GET_MODE (op);
1970 :
1971 15039228 : ira_init_register_move_cost_if_necessary (op_mode);
1972 : /* If the preferred regclass isn't NO_REG, continue to find the matching
1973 : constraint in all available alternatives with preferred regclass, even
1974 : if we have found or will find one alternative whose constraint stands
1975 : for a REG (non-NO_REG) regclass. Note that it would be fine not to
1976 : respect matching constraint if the register copy is free, so exclude
1977 : it. */
1978 15039228 : bool respect_dup_despite_reg_cstr
1979 15039228 : = param_ira_consider_dup_in_all_alts
1980 482 : && op_pref_cl != NO_REGS
1981 15039706 : && ira_register_move_cost[op_mode][op_pref_cl][op_pref_cl] > 0;
1982 :
1983 : /* Record the alternative whose constraint uses the same regclass as the
1984 : preferred regclass, later if we find one matching constraint for this
1985 : operand with preferred reclass, we will visit these recorded
1986 : alternatives to check whether if there is one alternative in which no
1987 : any INPUT operands have one matching constraint same as our candidate.
1988 : If yes, it means there is one alternative which is perfectly fine
1989 : without satisfying this matching constraint. If no, it means in any
1990 : alternatives there is one other INPUT operand holding this matching
1991 : constraint, it's fine to respect this matching constraint and further
1992 : create this constraint copy since it would become harmless once some
1993 : other takes preference and it's interfered. */
1994 17524916 : alternative_mask pref_cl_alts;
1995 :
1996 17524916 : for (;;)
1997 : {
1998 17524916 : pref_cl_alts = 0;
1999 :
2000 17524916 : for (curr_alt = 0, ignore_p = !TEST_BIT (alts, curr_alt),
2001 17524916 : original = -1;;)
2002 : {
2003 104191411 : c = *str;
2004 104191411 : if (c == '\0')
2005 : break;
2006 99215790 : if (c == '#')
2007 : ignore_p = true;
2008 99215790 : else if (c == ',')
2009 : {
2010 33607316 : curr_alt++;
2011 33607316 : ignore_p = !TEST_BIT (alts, curr_alt);
2012 : }
2013 65608474 : else if (! ignore_p)
2014 21020945 : switch (c)
2015 : {
2016 922 : case 'g':
2017 922 : goto fail;
2018 15799089 : default:
2019 15799089 : {
2020 15799089 : enum constraint_num cn = lookup_constraint (str);
2021 15799089 : enum reg_class cl = reg_class_for_constraint (cn);
2022 12830390 : if (cl != NO_REGS && !targetm.class_likely_spilled_p (cl))
2023 : {
2024 12546926 : if (respect_dup_despite_reg_cstr)
2025 : {
2026 : /* If it's free to move from one preferred class to
2027 : the one without matching constraint, it doesn't
2028 : have to respect this constraint with costs. */
2029 693 : if (cl != op_pref_cl
2030 116 : && (ira_reg_class_intersect[cl][op_pref_cl]
2031 : != NO_REGS)
2032 104 : && (ira_may_move_in_cost[op_mode][op_pref_cl][cl]
2033 : == 0))
2034 88 : goto fail;
2035 605 : else if (cl == op_pref_cl)
2036 577 : pref_cl_alts |= ALTERNATIVE_BIT (curr_alt);
2037 : }
2038 : else
2039 12546233 : goto fail;
2040 : }
2041 3252768 : if (constraint_satisfied_p (op, cn))
2042 2052 : goto fail;
2043 : break;
2044 : }
2045 :
2046 5220934 : case '0': case '1': case '2': case '3': case '4':
2047 5220934 : case '5': case '6': case '7': case '8': case '9':
2048 5220934 : {
2049 5220934 : char *end;
2050 5220934 : int n = (int) strtoul (str, &end, 10);
2051 5220934 : str = end;
2052 5220934 : if (original != -1 && original != n)
2053 0 : goto fail;
2054 5220934 : gcc_assert (n < recog_data.n_operands);
2055 5220934 : if (respect_dup_despite_reg_cstr)
2056 : {
2057 231 : const operand_alternative *op_alt
2058 231 : = &recog_op_alt[curr_alt * recog_data.n_operands];
2059 : /* Only respect the one with preferred rclass, without
2060 : respect_dup_despite_reg_cstr it's possible to get
2061 : one whose regclass isn't preferred first before,
2062 : but it would fail since there should be other
2063 : alternatives with preferred regclass. */
2064 231 : if (op_alt[n].cl == op_pref_cl)
2065 5220874 : original = n;
2066 : }
2067 : else
2068 : original = n;
2069 5220934 : continue;
2070 5220934 : }
2071 : }
2072 81445561 : str += CONSTRAINT_LEN (c, str);
2073 : }
2074 4975621 : if (original == -1)
2075 1884602 : goto fail;
2076 3091019 : if (recog_data.operand_type[original] == OP_OUT)
2077 : {
2078 3090709 : if (pref_cl_alts == 0)
2079 : return original;
2080 : /* Visit these recorded alternatives to check whether
2081 : there is one alternative in which no any INPUT operands
2082 : have one matching constraint same as our candidate.
2083 : Give up this candidate if so. */
2084 : int nop, nalt;
2085 392 : for (nalt = 0; nalt < recog_data.n_alternatives; nalt++)
2086 : {
2087 369 : if (!TEST_BIT (pref_cl_alts, nalt))
2088 264 : continue;
2089 105 : const operand_alternative *op_alt
2090 105 : = &recog_op_alt[nalt * recog_data.n_operands];
2091 105 : bool dup_in_other = false;
2092 387 : for (nop = 0; nop < recog_data.n_operands; nop++)
2093 : {
2094 327 : if (recog_data.operand_type[nop] != OP_IN)
2095 105 : continue;
2096 222 : if (nop == op_num)
2097 94 : continue;
2098 128 : if (op_alt[nop].matches == original)
2099 : {
2100 : dup_in_other = true;
2101 : break;
2102 : }
2103 : }
2104 105 : if (!dup_in_other)
2105 : return -1;
2106 : }
2107 23 : single_input_op_has_cstr_p = false;
2108 23 : return original;
2109 : }
2110 310 : fail:
2111 14434207 : if (use_commut_op_p)
2112 : break;
2113 12760973 : use_commut_op_p = true;
2114 12760973 : if (recog_data.constraints[op_num][0] == '%')
2115 1083227 : str = recog_data.constraints[op_num + 1];
2116 11677746 : else if (op_num > 0 && recog_data.constraints[op_num - 1][0] == '%')
2117 : str = recog_data.constraints[op_num - 1];
2118 : else
2119 : break;
2120 : }
2121 : return -1;
2122 : }
2123 :
2124 :
2125 :
2126 : /* Return true if a replacement of SRC by DEST does not lead to unsatisfiable
2127 : asm. A replacement is valid if SRC or DEST are not constrained in asm
2128 : inputs of a single asm statement. See match_asm_constraints_2() for more
2129 : details. TODO: As in match_asm_constraints_2() consider alternatives more
2130 : precisely. */
2131 :
2132 : static bool
2133 7617 : valid_replacement_for_asm_input_p_1 (const_rtx asmops, const_rtx src, const_rtx dest)
2134 : {
2135 7617 : int ninputs = ASM_OPERANDS_INPUT_LENGTH (asmops);
2136 7617 : rtvec inputs = ASM_OPERANDS_INPUT_VEC (asmops);
2137 38345 : for (int i = 0; i < ninputs; ++i)
2138 : {
2139 30728 : rtx input_src = RTVEC_ELT (inputs, i);
2140 30728 : const char *constraint_src
2141 30728 : = ASM_OPERANDS_INPUT_CONSTRAINT (asmops, i);
2142 30728 : if (rtx_equal_p (input_src, src)
2143 30728 : && strchr (constraint_src, '{') != nullptr)
2144 0 : for (int j = 0; j < ninputs; ++j)
2145 : {
2146 0 : rtx input_dest = RTVEC_ELT (inputs, j);
2147 0 : const char *constraint_dest
2148 0 : = ASM_OPERANDS_INPUT_CONSTRAINT (asmops, j);
2149 0 : if (rtx_equal_p (input_dest, dest)
2150 0 : && strchr (constraint_dest, '{') != nullptr)
2151 : return false;
2152 : }
2153 : }
2154 : return true;
2155 : }
2156 :
2157 : /* Return true if a replacement of SRC by DEST does not lead to unsatisfiable
2158 : asm. A replacement is valid if SRC or DEST are not constrained in asm
2159 : inputs of a single asm statement. The final check is done in function
2160 : valid_replacement_for_asm_input_p_1. */
2161 :
2162 : static bool
2163 539217 : valid_replacement_for_asm_input_p (const_rtx src, const_rtx dest)
2164 : {
2165 : /* Bail out early if there is no asm statement. */
2166 539217 : if (!crtl->has_asm_statement)
2167 : return true;
2168 29196 : for (df_ref use = DF_REG_USE_CHAIN (REGNO (src));
2169 803888 : use;
2170 774692 : use = DF_REF_NEXT_REG (use))
2171 : {
2172 774692 : struct df_insn_info *use_info = DF_REF_INSN_INFO (use);
2173 : /* Only check real uses, not artificial ones. */
2174 774692 : if (use_info)
2175 : {
2176 774692 : rtx_insn *insn = DF_REF_INSN (use);
2177 774692 : rtx pat = PATTERN (insn);
2178 774692 : if (asm_noperands (pat) <= 0)
2179 771395 : continue;
2180 3297 : if (GET_CODE (pat) == SET)
2181 : {
2182 0 : if (!valid_replacement_for_asm_input_p_1 (SET_SRC (pat), src, dest))
2183 : return false;
2184 : }
2185 3297 : else if (GET_CODE (pat) == PARALLEL)
2186 14254 : for (int i = 0, len = XVECLEN (pat, 0); i < len; ++i)
2187 : {
2188 10957 : rtx asmops = XVECEXP (pat, 0, i);
2189 10957 : if (GET_CODE (asmops) == SET)
2190 7588 : asmops = SET_SRC (asmops);
2191 10957 : if (GET_CODE (asmops) == ASM_OPERANDS
2192 10957 : && !valid_replacement_for_asm_input_p_1 (asmops, src, dest))
2193 : return false;
2194 : }
2195 0 : else if (GET_CODE (pat) == ASM_OPERANDS)
2196 : {
2197 0 : if (!valid_replacement_for_asm_input_p_1 (pat, src, dest))
2198 : return false;
2199 : }
2200 : else
2201 0 : gcc_unreachable ();
2202 : }
2203 : }
2204 : return true;
2205 : }
2206 :
2207 : /* Search forward to see if the source register of a copy insn dies
2208 : before either it or the destination register is modified, but don't
2209 : scan past the end of the basic block. If so, we can replace the
2210 : source with the destination and let the source die in the copy
2211 : insn.
2212 :
2213 : This will reduce the number of registers live in that range and may
2214 : enable the destination and the source coalescing, thus often saving
2215 : one register in addition to a register-register copy. */
2216 :
2217 : static void
2218 1504950 : decrease_live_ranges_number (void)
2219 : {
2220 1504950 : basic_block bb;
2221 1504950 : rtx_insn *insn;
2222 1504950 : rtx set, src, dest, dest_death, note;
2223 1504950 : rtx_insn *p, *q;
2224 1504950 : int sregno, dregno;
2225 :
2226 1504950 : if (! flag_expensive_optimizations)
2227 : return;
2228 :
2229 976444 : if (ira_dump_file)
2230 32 : fprintf (ira_dump_file, "Starting decreasing number of live ranges...\n");
2231 :
2232 11514749 : FOR_EACH_BB_FN (bb, cfun)
2233 138349305 : FOR_BB_INSNS (bb, insn)
2234 : {
2235 127811000 : set = single_set (insn);
2236 127811000 : if (! set)
2237 75630482 : continue;
2238 52180518 : src = SET_SRC (set);
2239 52180518 : dest = SET_DEST (set);
2240 14319033 : if (! REG_P (src) || ! REG_P (dest)
2241 61192510 : || find_reg_note (insn, REG_DEAD, src))
2242 49277635 : continue;
2243 2902883 : sregno = REGNO (src);
2244 2902883 : dregno = REGNO (dest);
2245 :
2246 : /* We don't want to mess with hard regs if register classes
2247 : are small. */
2248 5266549 : if (sregno == dregno
2249 2902849 : || (targetm.small_register_classes_for_mode_p (GET_MODE (src))
2250 2902849 : && (sregno < FIRST_PSEUDO_REGISTER
2251 2902849 : || dregno < FIRST_PSEUDO_REGISTER))
2252 : /* We don't see all updates to SP if they are in an
2253 : auto-inc memory reference, so we must disallow this
2254 : optimization on them. */
2255 539217 : || sregno == STACK_POINTER_REGNUM
2256 539217 : || dregno == STACK_POINTER_REGNUM
2257 3442100 : || !valid_replacement_for_asm_input_p (src, dest))
2258 2363666 : continue;
2259 :
2260 539217 : dest_death = NULL_RTX;
2261 :
2262 7094084 : for (p = NEXT_INSN (insn); p; p = NEXT_INSN (p))
2263 : {
2264 7088317 : if (! INSN_P (p))
2265 1121348 : continue;
2266 5966969 : if (BLOCK_FOR_INSN (p) != bb)
2267 : break;
2268 :
2269 10964365 : if (reg_set_p (src, p) || reg_set_p (dest, p)
2270 : /* If SRC is an asm-declared register, it must not be
2271 : replaced in any asm. Unfortunately, the REG_EXPR
2272 : tree for the asm variable may be absent in the SRC
2273 : rtx, so we can't check the actual register
2274 : declaration easily (the asm operand will have it,
2275 : though). To avoid complicating the test for a rare
2276 : case, we just don't perform register replacement
2277 : for a hard reg mentioned in an asm. */
2278 5440403 : || (sregno < FIRST_PSEUDO_REGISTER
2279 0 : && asm_noperands (PATTERN (p)) >= 0
2280 0 : && reg_overlap_mentioned_p (src, PATTERN (p)))
2281 : /* Don't change hard registers used by a call. */
2282 5440403 : || (CALL_P (p) && sregno < FIRST_PSEUDO_REGISTER
2283 0 : && find_reg_fusage (p, USE, src))
2284 : /* Don't change a USE of a register. */
2285 10936010 : || (GET_CODE (PATTERN (p)) == USE
2286 1359 : && reg_overlap_mentioned_p (src, XEXP (PATTERN (p), 0))))
2287 : break;
2288 :
2289 : /* See if all of SRC dies in P. This test is slightly
2290 : more conservative than it needs to be. */
2291 5440403 : if ((note = find_regno_note (p, REG_DEAD, sregno))
2292 5440403 : && GET_MODE (XEXP (note, 0)) == GET_MODE (src))
2293 : {
2294 6884 : int failed = 0;
2295 :
2296 : /* We can do the optimization. Scan forward from INSN
2297 : again, replacing regs as we go. Set FAILED if a
2298 : replacement can't be done. In that case, we can't
2299 : move the death note for SRC. This should be
2300 : rare. */
2301 :
2302 : /* Set to stop at next insn. */
2303 6884 : for (q = next_real_insn (insn);
2304 38936 : q != next_real_insn (p);
2305 32052 : q = next_real_insn (q))
2306 : {
2307 32052 : if (reg_overlap_mentioned_p (src, PATTERN (q)))
2308 : {
2309 : /* If SRC is a hard register, we might miss
2310 : some overlapping registers with
2311 : validate_replace_rtx, so we would have to
2312 : undo it. We can't if DEST is present in
2313 : the insn, so fail in that combination of
2314 : cases. */
2315 7913 : if (sregno < FIRST_PSEUDO_REGISTER
2316 7913 : && reg_mentioned_p (dest, PATTERN (q)))
2317 : failed = 1;
2318 :
2319 : /* Attempt to replace all uses. */
2320 7913 : else if (!validate_replace_rtx (src, dest, q))
2321 : failed = 1;
2322 :
2323 : /* If this succeeded, but some part of the
2324 : register is still present, undo the
2325 : replacement. */
2326 7913 : else if (sregno < FIRST_PSEUDO_REGISTER
2327 7913 : && reg_overlap_mentioned_p (src, PATTERN (q)))
2328 : {
2329 0 : validate_replace_rtx (dest, src, q);
2330 0 : failed = 1;
2331 : }
2332 : }
2333 :
2334 : /* If DEST dies here, remove the death note and
2335 : save it for later. Make sure ALL of DEST dies
2336 : here; again, this is overly conservative. */
2337 32052 : if (! dest_death
2338 32052 : && (dest_death = find_regno_note (q, REG_DEAD, dregno)))
2339 : {
2340 2 : if (GET_MODE (XEXP (dest_death, 0)) == GET_MODE (dest))
2341 2 : remove_note (q, dest_death);
2342 : else
2343 : {
2344 : failed = 1;
2345 : dest_death = 0;
2346 : }
2347 : }
2348 : }
2349 :
2350 6884 : if (! failed)
2351 : {
2352 : /* Move death note of SRC from P to INSN. */
2353 6884 : remove_note (p, note);
2354 6884 : XEXP (note, 1) = REG_NOTES (insn);
2355 6884 : REG_NOTES (insn) = note;
2356 : }
2357 :
2358 : /* DEST is also dead if INSN has a REG_UNUSED note for
2359 : DEST. */
2360 6884 : if (! dest_death
2361 6884 : && (dest_death
2362 6882 : = find_regno_note (insn, REG_UNUSED, dregno)))
2363 : {
2364 0 : PUT_REG_NOTE_KIND (dest_death, REG_DEAD);
2365 0 : remove_note (insn, dest_death);
2366 : }
2367 :
2368 : /* Put death note of DEST on P if we saw it die. */
2369 6884 : if (dest_death)
2370 : {
2371 2 : XEXP (dest_death, 1) = REG_NOTES (p);
2372 2 : REG_NOTES (p) = dest_death;
2373 : }
2374 : break;
2375 : }
2376 :
2377 : /* If SRC is a hard register which is set or killed in
2378 : some other way, we can't do this optimization. */
2379 5433519 : else if (sregno < FIRST_PSEUDO_REGISTER && dead_or_set_p (p, src))
2380 : break;
2381 : }
2382 : }
2383 : }
2384 :
2385 :
2386 :
2387 : /* Return nonzero if REGNO is a particularly bad choice for reloading X. */
2388 : static bool
2389 0 : ira_bad_reload_regno_1 (int regno, rtx x)
2390 : {
2391 0 : int x_regno, n, i;
2392 0 : ira_allocno_t a;
2393 0 : enum reg_class pref;
2394 :
2395 : /* We only deal with pseudo regs. */
2396 0 : if (! x || GET_CODE (x) != REG)
2397 : return false;
2398 :
2399 0 : x_regno = REGNO (x);
2400 0 : if (x_regno < FIRST_PSEUDO_REGISTER)
2401 : return false;
2402 :
2403 : /* If the pseudo prefers REGNO explicitly, then do not consider
2404 : REGNO a bad spill choice. */
2405 0 : pref = reg_preferred_class (x_regno);
2406 0 : if (reg_class_size[pref] == 1)
2407 0 : return !TEST_HARD_REG_BIT (reg_class_contents[pref], regno);
2408 :
2409 : /* If the pseudo conflicts with REGNO, then we consider REGNO a
2410 : poor choice for a reload regno. */
2411 0 : a = ira_regno_allocno_map[x_regno];
2412 0 : n = ALLOCNO_NUM_OBJECTS (a);
2413 0 : for (i = 0; i < n; i++)
2414 : {
2415 0 : ira_object_t obj = ALLOCNO_OBJECT (a, i);
2416 0 : if (TEST_HARD_REG_BIT (OBJECT_TOTAL_CONFLICT_HARD_REGS (obj), regno))
2417 : return true;
2418 : }
2419 : return false;
2420 : }
2421 :
2422 : /* Return nonzero if REGNO is a particularly bad choice for reloading
2423 : IN or OUT. */
2424 : bool
2425 0 : ira_bad_reload_regno (int regno, rtx in, rtx out)
2426 : {
2427 0 : return (ira_bad_reload_regno_1 (regno, in)
2428 0 : || ira_bad_reload_regno_1 (regno, out));
2429 : }
2430 :
2431 : /* Add register clobbers from asm statements. */
2432 : static void
2433 1534714 : compute_regs_asm_clobbered (void)
2434 : {
2435 1534714 : basic_block bb;
2436 :
2437 16504499 : FOR_EACH_BB_FN (bb, cfun)
2438 : {
2439 14969785 : rtx_insn *insn;
2440 180948322 : FOR_BB_INSNS_REVERSE (bb, insn)
2441 : {
2442 165978537 : df_ref def;
2443 :
2444 165978537 : if (NONDEBUG_INSN_P (insn) && asm_noperands (PATTERN (insn)) >= 0)
2445 336518 : FOR_EACH_INSN_DEF (def, insn)
2446 : {
2447 225009 : unsigned int dregno = DF_REF_REGNO (def);
2448 225009 : if (HARD_REGISTER_NUM_P (dregno))
2449 308832 : add_to_hard_reg_set (&crtl->asm_clobbers,
2450 154416 : GET_MODE (DF_REF_REAL_REG (def)),
2451 : dregno);
2452 : }
2453 : }
2454 : }
2455 1534714 : }
2456 :
2457 :
2458 : /* Set up ELIMINABLE_REGSET, IRA_NO_ALLOC_REGS, and
2459 : REGS_EVER_LIVE. */
2460 : void
2461 1534714 : ira_setup_eliminable_regset (void)
2462 : {
2463 1534714 : int i;
2464 1534714 : static const struct {const int from, to; } eliminables[] = ELIMINABLE_REGS;
2465 1665550 : int fp_reg_count = hard_regno_nregs (HARD_FRAME_POINTER_REGNUM, Pmode);
2466 :
2467 : /* Setup is_leaf as frame_pointer_required may use it. This function
2468 : is called by sched_init before ira if scheduling is enabled. */
2469 1534714 : crtl->is_leaf = leaf_function_p ();
2470 :
2471 : /* FIXME: If EXIT_IGNORE_STACK is set, we will not save and restore
2472 : sp for alloca. So we can't eliminate the frame pointer in that
2473 : case. At some point, we should improve this by emitting the
2474 : sp-adjusting insns for this case. */
2475 1534714 : frame_pointer_needed
2476 3069428 : = (! flag_omit_frame_pointer
2477 1072622 : || (cfun->calls_alloca && EXIT_IGNORE_STACK)
2478 : /* We need the frame pointer to catch stack overflow exceptions if
2479 : the stack pointer is moving (as for the alloca case just above). */
2480 1063130 : || (STACK_CHECK_MOVING_SP
2481 1063130 : && flag_stack_check
2482 62 : && flag_exceptions
2483 26 : && cfun->can_throw_non_call_exceptions)
2484 1063126 : || crtl->accesses_prior_frames
2485 1060196 : || (SUPPORTS_STACK_ALIGNMENT && crtl->stack_realign_needed)
2486 2549013 : || targetm.frame_pointer_required ());
2487 :
2488 : /* The chance that FRAME_POINTER_NEEDED is changed from inspecting
2489 : RTL is very small. So if we use frame pointer for RA and RTL
2490 : actually prevents this, we will spill pseudos assigned to the
2491 : frame pointer in LRA. */
2492 :
2493 1534714 : if (frame_pointer_needed)
2494 1040920 : for (i = 0; i < fp_reg_count; i++)
2495 520460 : df_set_regs_ever_live (HARD_FRAME_POINTER_REGNUM + i, true);
2496 :
2497 1534714 : ira_no_alloc_regs = no_unit_alloc_regs;
2498 1534714 : CLEAR_HARD_REG_SET (eliminable_regset);
2499 :
2500 1534714 : compute_regs_asm_clobbered ();
2501 :
2502 : /* Build the regset of all eliminable registers and show we can't
2503 : use those that we already know won't be eliminated. */
2504 7673570 : for (i = 0; i < (int) ARRAY_SIZE (eliminables); i++)
2505 : {
2506 6138856 : bool cannot_elim
2507 6138856 : = (! targetm.can_eliminate (eliminables[i].from, eliminables[i].to)
2508 6138856 : || (eliminables[i].to == STACK_POINTER_REGNUM && frame_pointer_needed));
2509 :
2510 6138856 : if (!TEST_HARD_REG_BIT (crtl->asm_clobbers, eliminables[i].from))
2511 : {
2512 6138856 : SET_HARD_REG_BIT (eliminable_regset, eliminables[i].from);
2513 :
2514 6138856 : if (cannot_elim)
2515 1100914 : SET_HARD_REG_BIT (ira_no_alloc_regs, eliminables[i].from);
2516 : }
2517 0 : else if (cannot_elim)
2518 0 : error ("%s cannot be used in %<asm%> here",
2519 : reg_names[eliminables[i].from]);
2520 : else
2521 0 : df_set_regs_ever_live (eliminables[i].from, true);
2522 : }
2523 : if (!HARD_FRAME_POINTER_IS_FRAME_POINTER)
2524 : {
2525 3069428 : for (i = 0; i < fp_reg_count; i++)
2526 1534714 : if (global_regs[HARD_FRAME_POINTER_REGNUM + i])
2527 : /* Nothing to do: the register is already treated as live
2528 : where appropriate, and cannot be eliminated. */
2529 : ;
2530 1534693 : else if (!TEST_HARD_REG_BIT (crtl->asm_clobbers,
2531 : HARD_FRAME_POINTER_REGNUM + i))
2532 : {
2533 1533402 : SET_HARD_REG_BIT (eliminable_regset,
2534 : HARD_FRAME_POINTER_REGNUM + i);
2535 1533402 : if (frame_pointer_needed)
2536 520458 : SET_HARD_REG_BIT (ira_no_alloc_regs,
2537 : HARD_FRAME_POINTER_REGNUM + i);
2538 : }
2539 1291 : else if (frame_pointer_needed)
2540 0 : error ("%s cannot be used in %<asm%> here",
2541 : reg_names[HARD_FRAME_POINTER_REGNUM + i]);
2542 : else
2543 1291 : df_set_regs_ever_live (HARD_FRAME_POINTER_REGNUM + i, true);
2544 : }
2545 1534714 : }
2546 :
2547 :
2548 :
2549 : /* Vector of substitutions of register numbers,
2550 : used to map pseudo regs into hardware regs.
2551 : This is set up as a result of register allocation.
2552 : Element N is the hard reg assigned to pseudo reg N,
2553 : or is -1 if no hard reg was assigned.
2554 : If N is a hard reg number, element N is N. */
2555 : short *reg_renumber;
2556 :
2557 : /* Set up REG_RENUMBER and CALLER_SAVE_NEEDED (used by reload) from
2558 : the allocation found by IRA. */
2559 : static void
2560 1504950 : setup_reg_renumber (void)
2561 : {
2562 1504950 : int regno, hard_regno;
2563 1504950 : ira_allocno_t a;
2564 1504950 : ira_allocno_iterator ai;
2565 :
2566 1504950 : caller_save_needed = 0;
2567 38442707 : FOR_EACH_ALLOCNO (a, ai)
2568 : {
2569 36937757 : if (ira_use_lra_p && ALLOCNO_CAP_MEMBER (a) != NULL)
2570 3642487 : continue;
2571 : /* There are no caps at this point. */
2572 33295270 : ira_assert (ALLOCNO_CAP_MEMBER (a) == NULL);
2573 33295270 : if (! ALLOCNO_ASSIGNED_P (a))
2574 : /* It can happen if A is not referenced but partially anticipated
2575 : somewhere in a region. */
2576 0 : ALLOCNO_ASSIGNED_P (a) = true;
2577 33295270 : ira_free_allocno_updated_costs (a);
2578 33295270 : hard_regno = ALLOCNO_HARD_REGNO (a);
2579 33295270 : regno = ALLOCNO_REGNO (a);
2580 33295270 : reg_renumber[regno] = (hard_regno < 0 ? -1 : hard_regno);
2581 33295270 : if (hard_regno >= 0)
2582 : {
2583 29962637 : int i, nwords;
2584 29962637 : enum reg_class pclass;
2585 29962637 : ira_object_t obj;
2586 :
2587 29962637 : pclass = ira_pressure_class_translate[REGNO_REG_CLASS (hard_regno)];
2588 29962637 : nwords = ALLOCNO_NUM_OBJECTS (a);
2589 61049675 : for (i = 0; i < nwords; i++)
2590 : {
2591 31087038 : obj = ALLOCNO_OBJECT (a, i);
2592 31087038 : OBJECT_TOTAL_CONFLICT_HARD_REGS (obj)
2593 62174076 : |= ~reg_class_contents[pclass];
2594 : }
2595 29962637 : if (ira_need_caller_save_p (a, hard_regno))
2596 : {
2597 439329 : ira_assert (!optimize || flag_caller_saves
2598 : || (ALLOCNO_CALLS_CROSSED_NUM (a)
2599 : == ALLOCNO_CHEAP_CALLS_CROSSED_NUM (a))
2600 : || regno >= ira_reg_equiv_len
2601 : || ira_equiv_no_lvalue_p (regno));
2602 439329 : caller_save_needed = 1;
2603 : }
2604 : }
2605 : }
2606 1504950 : }
2607 :
2608 : /* Set up allocno assignment flags for further allocation
2609 : improvements. */
2610 : static void
2611 0 : setup_allocno_assignment_flags (void)
2612 : {
2613 0 : int hard_regno;
2614 0 : ira_allocno_t a;
2615 0 : ira_allocno_iterator ai;
2616 :
2617 0 : FOR_EACH_ALLOCNO (a, ai)
2618 : {
2619 0 : if (! ALLOCNO_ASSIGNED_P (a))
2620 : /* It can happen if A is not referenced but partially anticipated
2621 : somewhere in a region. */
2622 0 : ira_free_allocno_updated_costs (a);
2623 0 : hard_regno = ALLOCNO_HARD_REGNO (a);
2624 : /* Don't assign hard registers to allocnos which are destination
2625 : of removed store at the end of loop. It has no sense to keep
2626 : the same value in different hard registers. It is also
2627 : impossible to assign hard registers correctly to such
2628 : allocnos because the cost info and info about intersected
2629 : calls are incorrect for them. */
2630 0 : ALLOCNO_ASSIGNED_P (a) = (hard_regno >= 0
2631 0 : || ALLOCNO_EMIT_DATA (a)->mem_optimized_dest_p
2632 0 : || (ALLOCNO_MEMORY_COST (a)
2633 0 : - ALLOCNO_CLASS_COST (a)) < 0);
2634 0 : ira_assert
2635 : (hard_regno < 0
2636 : || ira_hard_reg_in_set_p (hard_regno, ALLOCNO_MODE (a),
2637 : reg_class_contents[ALLOCNO_CLASS (a)]));
2638 : }
2639 0 : }
2640 :
2641 : /* Evaluate overall allocation cost and the costs for using hard
2642 : registers and memory for allocnos. */
2643 : static void
2644 1504950 : calculate_allocation_cost (void)
2645 : {
2646 1504950 : int hard_regno, cost;
2647 1504950 : ira_allocno_t a;
2648 1504950 : ira_allocno_iterator ai;
2649 :
2650 1504950 : ira_overall_cost = ira_reg_cost = ira_mem_cost = 0;
2651 38442707 : FOR_EACH_ALLOCNO (a, ai)
2652 : {
2653 36937757 : hard_regno = ALLOCNO_HARD_REGNO (a);
2654 36937757 : ira_assert (hard_regno < 0
2655 : || (ira_hard_reg_in_set_p
2656 : (hard_regno, ALLOCNO_MODE (a),
2657 : reg_class_contents[ALLOCNO_CLASS (a)])));
2658 36937757 : if (hard_regno < 0)
2659 : {
2660 3732687 : cost = ALLOCNO_MEMORY_COST (a);
2661 3732687 : ira_mem_cost += cost;
2662 : }
2663 33205070 : else if (ALLOCNO_HARD_REG_COSTS (a) != NULL)
2664 : {
2665 8742300 : cost = (ALLOCNO_HARD_REG_COSTS (a)
2666 : [ira_class_hard_reg_index
2667 8742300 : [ALLOCNO_CLASS (a)][hard_regno]]);
2668 8742300 : ira_reg_cost += cost;
2669 : }
2670 : else
2671 : {
2672 24462770 : cost = ALLOCNO_CLASS_COST (a);
2673 24462770 : ira_reg_cost += cost;
2674 : }
2675 36937757 : ira_overall_cost += cost;
2676 : }
2677 :
2678 1504950 : if (internal_flag_ira_verbose > 0 && ira_dump_file != NULL)
2679 : {
2680 95 : fprintf (ira_dump_file,
2681 : "+++Costs: overall %" PRId64
2682 : ", reg %" PRId64
2683 : ", mem %" PRId64
2684 : ", ld %" PRId64
2685 : ", st %" PRId64
2686 : ", move %" PRId64,
2687 : ira_overall_cost, ira_reg_cost, ira_mem_cost,
2688 : ira_load_cost, ira_store_cost, ira_shuffle_cost);
2689 95 : fprintf (ira_dump_file, "\n+++ move loops %d, new jumps %d\n",
2690 : ira_move_loops_num, ira_additional_jumps_num);
2691 : }
2692 :
2693 1504950 : }
2694 :
2695 : #ifdef ENABLE_IRA_CHECKING
2696 : /* Check the correctness of the allocation. We do need this because
2697 : of complicated code to transform more one region internal
2698 : representation into one region representation. */
2699 : static void
2700 0 : check_allocation (void)
2701 : {
2702 0 : ira_allocno_t a;
2703 0 : int hard_regno, nregs, conflict_nregs;
2704 0 : ira_allocno_iterator ai;
2705 :
2706 0 : FOR_EACH_ALLOCNO (a, ai)
2707 : {
2708 0 : int n = ALLOCNO_NUM_OBJECTS (a);
2709 0 : int i;
2710 :
2711 0 : if (ALLOCNO_CAP_MEMBER (a) != NULL
2712 0 : || (hard_regno = ALLOCNO_HARD_REGNO (a)) < 0)
2713 0 : continue;
2714 0 : nregs = hard_regno_nregs (hard_regno, ALLOCNO_MODE (a));
2715 0 : if (nregs == 1)
2716 : /* We allocated a single hard register. */
2717 : n = 1;
2718 0 : else if (n > 1)
2719 : /* We allocated multiple hard registers, and we will test
2720 : conflicts in a granularity of single hard regs. */
2721 0 : nregs = 1;
2722 :
2723 0 : for (i = 0; i < n; i++)
2724 : {
2725 0 : ira_object_t obj = ALLOCNO_OBJECT (a, i);
2726 0 : ira_object_t conflict_obj;
2727 0 : ira_object_conflict_iterator oci;
2728 0 : int this_regno = hard_regno;
2729 0 : if (n > 1)
2730 : {
2731 0 : if (REG_WORDS_BIG_ENDIAN)
2732 : this_regno += n - i - 1;
2733 : else
2734 0 : this_regno += i;
2735 : }
2736 0 : FOR_EACH_OBJECT_CONFLICT (obj, conflict_obj, oci)
2737 : {
2738 0 : ira_allocno_t conflict_a = OBJECT_ALLOCNO (conflict_obj);
2739 0 : int conflict_hard_regno = ALLOCNO_HARD_REGNO (conflict_a);
2740 0 : if (conflict_hard_regno < 0)
2741 0 : continue;
2742 0 : if (ira_soft_conflict (a, conflict_a))
2743 0 : continue;
2744 :
2745 0 : conflict_nregs = hard_regno_nregs (conflict_hard_regno,
2746 0 : ALLOCNO_MODE (conflict_a));
2747 :
2748 0 : if (ALLOCNO_NUM_OBJECTS (conflict_a) > 1
2749 0 : && conflict_nregs == ALLOCNO_NUM_OBJECTS (conflict_a))
2750 : {
2751 0 : if (REG_WORDS_BIG_ENDIAN)
2752 : conflict_hard_regno += (ALLOCNO_NUM_OBJECTS (conflict_a)
2753 : - OBJECT_SUBWORD (conflict_obj) - 1);
2754 : else
2755 0 : conflict_hard_regno += OBJECT_SUBWORD (conflict_obj);
2756 0 : conflict_nregs = 1;
2757 : }
2758 :
2759 0 : if ((conflict_hard_regno <= this_regno
2760 0 : && this_regno < conflict_hard_regno + conflict_nregs)
2761 0 : || (this_regno <= conflict_hard_regno
2762 0 : && conflict_hard_regno < this_regno + nregs))
2763 : {
2764 0 : fprintf (stderr, "bad allocation for %d and %d\n",
2765 : ALLOCNO_REGNO (a), ALLOCNO_REGNO (conflict_a));
2766 0 : gcc_unreachable ();
2767 : }
2768 : }
2769 : }
2770 : }
2771 0 : }
2772 : #endif
2773 :
2774 : /* Allocate REG_EQUIV_INIT. Set up it from IRA_REG_EQUIV which should
2775 : be already calculated. */
2776 : static void
2777 1504950 : setup_reg_equiv_init (void)
2778 : {
2779 1504950 : int i;
2780 1504950 : int max_regno = max_reg_num ();
2781 :
2782 208362348 : for (i = 0; i < max_regno; i++)
2783 205352448 : reg_equiv_init (i) = ira_reg_equiv[i].init_insns;
2784 1504950 : }
2785 :
2786 : /* Update equiv regno from movement of FROM_REGNO to TO_REGNO. INSNS
2787 : are insns which were generated for such movement. It is assumed
2788 : that FROM_REGNO and TO_REGNO always have the same value at the
2789 : point of any move containing such registers. This function is used
2790 : to update equiv info for register shuffles on the region borders
2791 : and for caller save/restore insns. */
2792 : void
2793 2137193 : ira_update_equiv_info_by_shuffle_insn (int to_regno, int from_regno, rtx_insn *insns)
2794 : {
2795 2137193 : rtx_insn *insn;
2796 2137193 : rtx x, note;
2797 :
2798 2137193 : if (! ira_reg_equiv[from_regno].defined_p
2799 2137193 : && (! ira_reg_equiv[to_regno].defined_p
2800 924 : || ((x = ira_reg_equiv[to_regno].memory) != NULL_RTX
2801 923 : && ! MEM_READONLY_P (x))))
2802 : return;
2803 39734 : insn = insns;
2804 39734 : if (NEXT_INSN (insn) != NULL_RTX)
2805 : {
2806 0 : if (! ira_reg_equiv[to_regno].defined_p)
2807 : {
2808 0 : ira_assert (ira_reg_equiv[to_regno].init_insns == NULL_RTX);
2809 : return;
2810 : }
2811 0 : ira_reg_equiv[to_regno].defined_p = false;
2812 0 : ira_reg_equiv[to_regno].caller_save_p = false;
2813 0 : ira_reg_equiv[to_regno].memory
2814 0 : = ira_reg_equiv[to_regno].constant
2815 0 : = ira_reg_equiv[to_regno].invariant
2816 0 : = ira_reg_equiv[to_regno].init_insns = NULL;
2817 0 : if (internal_flag_ira_verbose > 3 && ira_dump_file != NULL)
2818 0 : fprintf (ira_dump_file,
2819 : " Invalidating equiv info for reg %d\n", to_regno);
2820 0 : return;
2821 : }
2822 : /* It is possible that FROM_REGNO still has no equivalence because
2823 : in shuffles to_regno<-from_regno and from_regno<-to_regno the 2nd
2824 : insn was not processed yet. */
2825 39734 : if (ira_reg_equiv[from_regno].defined_p)
2826 : {
2827 39733 : ira_reg_equiv[to_regno].defined_p = true;
2828 39733 : if ((x = ira_reg_equiv[from_regno].memory) != NULL_RTX)
2829 : {
2830 39586 : ira_assert (ira_reg_equiv[from_regno].invariant == NULL_RTX
2831 : && ira_reg_equiv[from_regno].constant == NULL_RTX);
2832 39586 : ira_assert (ira_reg_equiv[to_regno].memory == NULL_RTX
2833 : || rtx_equal_p (ira_reg_equiv[to_regno].memory, x));
2834 39586 : ira_reg_equiv[to_regno].memory = x;
2835 39586 : if (! MEM_READONLY_P (x))
2836 : /* We don't add the insn to insn init list because memory
2837 : equivalence is just to say what memory is better to use
2838 : when the pseudo is spilled. */
2839 : return;
2840 : }
2841 147 : else if ((x = ira_reg_equiv[from_regno].constant) != NULL_RTX)
2842 : {
2843 37 : ira_assert (ira_reg_equiv[from_regno].invariant == NULL_RTX);
2844 37 : ira_assert (ira_reg_equiv[to_regno].constant == NULL_RTX
2845 : || rtx_equal_p (ira_reg_equiv[to_regno].constant, x));
2846 37 : ira_reg_equiv[to_regno].constant = x;
2847 : }
2848 : else
2849 : {
2850 110 : x = ira_reg_equiv[from_regno].invariant;
2851 110 : ira_assert (x != NULL_RTX);
2852 110 : ira_assert (ira_reg_equiv[to_regno].invariant == NULL_RTX
2853 : || rtx_equal_p (ira_reg_equiv[to_regno].invariant, x));
2854 110 : ira_reg_equiv[to_regno].invariant = x;
2855 : }
2856 164 : if (find_reg_note (insn, REG_EQUIV, x) == NULL_RTX)
2857 : {
2858 164 : note = set_unique_reg_note (insn, REG_EQUIV, copy_rtx (x));
2859 164 : gcc_assert (note != NULL_RTX);
2860 164 : if (internal_flag_ira_verbose > 3 && ira_dump_file != NULL)
2861 : {
2862 0 : fprintf (ira_dump_file,
2863 : " Adding equiv note to insn %u for reg %d ",
2864 0 : INSN_UID (insn), to_regno);
2865 0 : dump_value_slim (ira_dump_file, x, 1);
2866 0 : fprintf (ira_dump_file, "\n");
2867 : }
2868 : }
2869 : }
2870 165 : ira_reg_equiv[to_regno].init_insns
2871 330 : = gen_rtx_INSN_LIST (VOIDmode, insn,
2872 165 : ira_reg_equiv[to_regno].init_insns);
2873 165 : if (internal_flag_ira_verbose > 3 && ira_dump_file != NULL)
2874 0 : fprintf (ira_dump_file,
2875 : " Adding equiv init move insn %u to reg %d\n",
2876 0 : INSN_UID (insn), to_regno);
2877 : }
2878 :
2879 : /* Fix values of array REG_EQUIV_INIT after live range splitting done
2880 : by IRA. */
2881 : static void
2882 2114118 : fix_reg_equiv_init (void)
2883 : {
2884 2114118 : int max_regno = max_reg_num ();
2885 2114118 : int i, new_regno, max;
2886 2114118 : rtx set;
2887 2114118 : rtx_insn_list *x, *next, *prev;
2888 2114118 : rtx_insn *insn;
2889 :
2890 2114118 : if (max_regno_before_ira < max_regno)
2891 : {
2892 514304 : max = vec_safe_length (reg_equivs);
2893 514304 : grow_reg_equivs ();
2894 48510396 : for (i = FIRST_PSEUDO_REGISTER; i < max; i++)
2895 47996092 : for (prev = NULL, x = reg_equiv_init (i);
2896 52545870 : x != NULL_RTX;
2897 : x = next)
2898 : {
2899 4549778 : next = x->next ();
2900 4549778 : insn = x->insn ();
2901 4549778 : set = single_set (insn);
2902 4549778 : ira_assert (set != NULL_RTX
2903 : && (REG_P (SET_DEST (set)) || REG_P (SET_SRC (set))));
2904 4549778 : if (REG_P (SET_DEST (set))
2905 4549778 : && ((int) REGNO (SET_DEST (set)) == i
2906 0 : || (int) ORIGINAL_REGNO (SET_DEST (set)) == i))
2907 : new_regno = REGNO (SET_DEST (set));
2908 495109 : else if (REG_P (SET_SRC (set))
2909 495109 : && ((int) REGNO (SET_SRC (set)) == i
2910 0 : || (int) ORIGINAL_REGNO (SET_SRC (set)) == i))
2911 : new_regno = REGNO (SET_SRC (set));
2912 : else
2913 0 : gcc_unreachable ();
2914 4549778 : if (new_regno == i)
2915 : prev = x;
2916 : else
2917 : {
2918 : /* Remove the wrong list element. */
2919 0 : if (prev == NULL_RTX)
2920 0 : reg_equiv_init (i) = next;
2921 : else
2922 0 : XEXP (prev, 1) = next;
2923 0 : XEXP (x, 1) = reg_equiv_init (new_regno);
2924 0 : reg_equiv_init (new_regno) = x;
2925 : }
2926 : }
2927 : }
2928 2114118 : }
2929 :
2930 : #ifdef ENABLE_IRA_CHECKING
2931 : /* Print redundant memory-memory copies. */
2932 : static void
2933 1057059 : print_redundant_copies (void)
2934 : {
2935 1057059 : int hard_regno;
2936 1057059 : ira_allocno_t a;
2937 1057059 : ira_copy_t cp, next_cp;
2938 1057059 : ira_allocno_iterator ai;
2939 :
2940 26364964 : FOR_EACH_ALLOCNO (a, ai)
2941 : {
2942 25307905 : if (ALLOCNO_CAP_MEMBER (a) != NULL)
2943 : /* It is a cap. */
2944 3642487 : continue;
2945 21665418 : hard_regno = ALLOCNO_HARD_REGNO (a);
2946 21665418 : if (hard_regno >= 0)
2947 18422020 : continue;
2948 4212350 : for (cp = ALLOCNO_COPIES (a); cp != NULL; cp = next_cp)
2949 968952 : if (cp->first == a)
2950 374216 : next_cp = cp->next_first_allocno_copy;
2951 : else
2952 : {
2953 594736 : next_cp = cp->next_second_allocno_copy;
2954 594736 : if (internal_flag_ira_verbose > 4 && ira_dump_file != NULL
2955 1 : && cp->insn != NULL_RTX
2956 0 : && ALLOCNO_HARD_REGNO (cp->first) == hard_regno)
2957 0 : fprintf (ira_dump_file,
2958 : " Redundant move from %d(freq %d):%d\n",
2959 0 : INSN_UID (cp->insn), cp->freq, hard_regno);
2960 : }
2961 : }
2962 1057059 : }
2963 : #endif
2964 :
2965 : /* Setup preferred and alternative classes for new pseudo-registers
2966 : created by IRA starting with START. */
2967 : static void
2968 1092570 : setup_preferred_alternate_classes_for_new_pseudos (int start)
2969 : {
2970 1092570 : int i, old_regno;
2971 1092570 : int max_regno = max_reg_num ();
2972 :
2973 2264789 : for (i = start; i < max_regno; i++)
2974 : {
2975 1172219 : old_regno = ORIGINAL_REGNO (regno_reg_rtx[i]);
2976 1172219 : ira_assert (i != old_regno);
2977 1172219 : setup_reg_classes (i, reg_preferred_class (old_regno),
2978 : reg_alternate_class (old_regno),
2979 : reg_allocno_class (old_regno));
2980 1172219 : if (internal_flag_ira_verbose > 2 && ira_dump_file != NULL)
2981 0 : fprintf (ira_dump_file,
2982 : " New r%d: setting preferred %s, alternative %s\n",
2983 0 : i, reg_class_names[reg_preferred_class (old_regno)],
2984 0 : reg_class_names[reg_alternate_class (old_regno)]);
2985 : }
2986 1092570 : }
2987 :
2988 :
2989 : /* The number of entries allocated in reg_info. */
2990 : static int allocated_reg_info_size;
2991 :
2992 : /* Regional allocation can create new pseudo-registers. This function
2993 : expands some arrays for pseudo-registers. */
2994 : static void
2995 1092570 : expand_reg_info (void)
2996 : {
2997 1092570 : int i;
2998 1092570 : int size = max_reg_num ();
2999 :
3000 1092570 : resize_reg_info ();
3001 2264789 : for (i = allocated_reg_info_size; i < size; i++)
3002 1172219 : setup_reg_classes (i, GENERAL_REGS, ALL_REGS, GENERAL_REGS);
3003 1092570 : setup_preferred_alternate_classes_for_new_pseudos (allocated_reg_info_size);
3004 1092570 : allocated_reg_info_size = size;
3005 1092570 : }
3006 :
3007 : /* Return TRUE if there is too high register pressure in the function.
3008 : It is used to decide when stack slot sharing is worth to do. */
3009 : static bool
3010 1504950 : too_high_register_pressure_p (void)
3011 : {
3012 1504950 : int i;
3013 1504950 : enum reg_class pclass;
3014 :
3015 7561726 : for (i = 0; i < ira_pressure_classes_num; i++)
3016 : {
3017 6056778 : pclass = ira_pressure_classes[i];
3018 6056778 : if (ira_loop_tree_root->reg_pressure[pclass] > 10000)
3019 : return true;
3020 : }
3021 : return false;
3022 : }
3023 :
3024 :
3025 :
3026 : /* Indicate that hard register number FROM was eliminated and replaced with
3027 : an offset from hard register number TO. The status of hard registers live
3028 : at the start of a basic block is updated by replacing a use of FROM with
3029 : a use of TO. */
3030 :
3031 : void
3032 0 : mark_elimination (int from, int to)
3033 : {
3034 0 : basic_block bb;
3035 0 : bitmap r;
3036 :
3037 0 : FOR_EACH_BB_FN (bb, cfun)
3038 : {
3039 0 : r = DF_LR_IN (bb);
3040 0 : if (bitmap_bit_p (r, from))
3041 : {
3042 0 : bitmap_clear_bit (r, from);
3043 0 : bitmap_set_bit (r, to);
3044 : }
3045 0 : if (! df_live)
3046 0 : continue;
3047 0 : r = DF_LIVE_IN (bb);
3048 0 : if (bitmap_bit_p (r, from))
3049 : {
3050 0 : bitmap_clear_bit (r, from);
3051 0 : bitmap_set_bit (r, to);
3052 : }
3053 : }
3054 0 : }
3055 :
3056 :
3057 :
3058 : /* The length of the following array. */
3059 : int ira_reg_equiv_len;
3060 :
3061 : /* Info about equiv. info for each register. */
3062 : struct ira_reg_equiv_s *ira_reg_equiv;
3063 :
3064 : /* Expand ira_reg_equiv if necessary. */
3065 : void
3066 15156555 : ira_expand_reg_equiv (void)
3067 : {
3068 15156555 : int old = ira_reg_equiv_len;
3069 :
3070 15156555 : if (ira_reg_equiv_len > max_reg_num ())
3071 : return;
3072 1508137 : ira_reg_equiv_len = max_reg_num () * 3 / 2 + 1;
3073 1508137 : ira_reg_equiv
3074 3016274 : = (struct ira_reg_equiv_s *) xrealloc (ira_reg_equiv,
3075 1508137 : ira_reg_equiv_len
3076 : * sizeof (struct ira_reg_equiv_s));
3077 1508137 : gcc_assert (old < ira_reg_equiv_len);
3078 1508137 : memset (ira_reg_equiv + old, 0,
3079 1508137 : sizeof (struct ira_reg_equiv_s) * (ira_reg_equiv_len - old));
3080 : }
3081 :
3082 : static void
3083 1504950 : init_reg_equiv (void)
3084 : {
3085 1504950 : ira_reg_equiv_len = 0;
3086 1504950 : ira_reg_equiv = NULL;
3087 0 : ira_expand_reg_equiv ();
3088 0 : }
3089 :
3090 : static void
3091 1504950 : finish_reg_equiv (void)
3092 : {
3093 1504950 : free (ira_reg_equiv);
3094 0 : }
3095 :
3096 :
3097 :
3098 : struct equivalence
3099 : {
3100 : /* Set when a REG_EQUIV note is found or created. Use to
3101 : keep track of what memory accesses might be created later,
3102 : e.g. by reload. */
3103 : rtx replacement;
3104 : rtx *src_p;
3105 :
3106 : /* The list of each instruction which initializes this register.
3107 :
3108 : NULL indicates we know nothing about this register's equivalence
3109 : properties.
3110 :
3111 : An INSN_LIST with a NULL insn indicates this pseudo is already
3112 : known to not have a valid equivalence. */
3113 : rtx_insn_list *init_insns;
3114 :
3115 : /* Loop depth is used to recognize equivalences which appear
3116 : to be present within the same loop (or in an inner loop). */
3117 : short loop_depth;
3118 : /* Nonzero if this had a preexisting REG_EQUIV note. */
3119 : unsigned char is_arg_equivalence : 1;
3120 : /* Set when an attempt should be made to replace a register
3121 : with the associated src_p entry. */
3122 : unsigned char replace : 1;
3123 : /* Set if this register has no known equivalence. */
3124 : unsigned char no_equiv : 1;
3125 : /* Set if this register is mentioned in a paradoxical subreg. */
3126 : unsigned char pdx_subregs : 1;
3127 : };
3128 :
3129 : /* reg_equiv[N] (where N is a pseudo reg number) is the equivalence
3130 : structure for that register. */
3131 : static struct equivalence *reg_equiv;
3132 :
3133 : /* Used for communication between the following two functions. */
3134 : struct equiv_mem_data
3135 : {
3136 : /* A MEM that we wish to ensure remains unchanged. */
3137 : rtx equiv_mem;
3138 :
3139 : /* Set true if EQUIV_MEM is modified. */
3140 : bool equiv_mem_modified;
3141 : };
3142 :
3143 : /* If EQUIV_MEM is modified by modifying DEST, indicate that it is modified.
3144 : Called via note_stores. */
3145 : static void
3146 14479972 : validate_equiv_mem_from_store (rtx dest, const_rtx set ATTRIBUTE_UNUSED,
3147 : void *data)
3148 : {
3149 14479972 : struct equiv_mem_data *info = (struct equiv_mem_data *) data;
3150 :
3151 14479972 : if ((REG_P (dest)
3152 10690339 : && reg_overlap_mentioned_p (dest, info->equiv_mem))
3153 25155442 : || (MEM_P (dest)
3154 3755833 : && anti_dependence (info->equiv_mem, dest)))
3155 362346 : info->equiv_mem_modified = true;
3156 14479972 : }
3157 :
3158 : static bool equiv_init_varies_p (rtx x);
3159 :
3160 : enum valid_equiv { valid_none, valid_combine, valid_reload };
3161 :
3162 : /* Verify that no store between START and the death of REG invalidates
3163 : MEMREF. MEMREF is invalidated by modifying a register used in MEMREF,
3164 : by storing into an overlapping memory location, or with a non-const
3165 : CALL_INSN.
3166 :
3167 : Return VALID_RELOAD if MEMREF remains valid for both reload and
3168 : combine_and_move insns, VALID_COMBINE if only valid for
3169 : combine_and_move_insns, and VALID_NONE otherwise. */
3170 : static enum valid_equiv
3171 4045237 : validate_equiv_mem (rtx_insn *start, rtx reg, rtx memref)
3172 : {
3173 4045237 : rtx_insn *insn;
3174 4045237 : rtx note;
3175 4045237 : struct equiv_mem_data info = { memref, false };
3176 4045237 : enum valid_equiv ret = valid_reload;
3177 :
3178 : /* If the memory reference has side effects or is volatile, it isn't a
3179 : valid equivalence. */
3180 4045237 : if (side_effects_p (memref))
3181 : return valid_none;
3182 :
3183 21542451 : for (insn = start; insn; insn = NEXT_INSN (insn))
3184 : {
3185 21542221 : if (!INSN_P (insn))
3186 1438273 : continue;
3187 :
3188 20103948 : if (find_reg_note (insn, REG_DEAD, reg))
3189 : return ret;
3190 :
3191 17320770 : if (CALL_P (insn))
3192 : {
3193 : /* We can combine a reg def from one insn into a reg use in
3194 : another over a call if the memory is readonly or the call
3195 : const/pure. However, we can't set reg_equiv notes up for
3196 : reload over any call. The problem is the equivalent form
3197 : may reference a pseudo which gets assigned a call
3198 : clobbered hard reg. When we later replace REG with its
3199 : equivalent form, the value in the call-clobbered reg has
3200 : been changed and all hell breaks loose. */
3201 94139 : ret = valid_combine;
3202 94139 : if (!MEM_READONLY_P (memref)
3203 94139 : && (!RTL_CONST_OR_PURE_CALL_P (insn)
3204 8390 : || equiv_init_varies_p (XEXP (memref, 0))))
3205 88308 : return valid_none;
3206 : }
3207 :
3208 17232462 : note_stores (insn, validate_equiv_mem_from_store, &info);
3209 17232462 : if (info.equiv_mem_modified)
3210 : return valid_none;
3211 :
3212 : /* If a register mentioned in MEMREF is modified via an
3213 : auto-increment, we lose the equivalence. Do the same if one
3214 : dies; although we could extend the life, it doesn't seem worth
3215 : the trouble. */
3216 :
3217 23501318 : for (note = REG_NOTES (insn); note; note = XEXP (note, 1))
3218 7265797 : if ((REG_NOTE_KIND (note) == REG_INC
3219 7265797 : || REG_NOTE_KIND (note) == REG_DEAD)
3220 5450431 : && REG_P (XEXP (note, 0))
3221 12716228 : && reg_overlap_mentioned_p (XEXP (note, 0), memref))
3222 : return valid_none;
3223 : }
3224 :
3225 : return valid_none;
3226 : }
3227 :
3228 : /* Returns false if X is known to be invariant. */
3229 : static bool
3230 840598 : equiv_init_varies_p (rtx x)
3231 : {
3232 840598 : RTX_CODE code = GET_CODE (x);
3233 840598 : int i;
3234 840598 : const char *fmt;
3235 :
3236 840598 : switch (code)
3237 : {
3238 223730 : case MEM:
3239 223730 : return !MEM_READONLY_P (x) || equiv_init_varies_p (XEXP (x, 0));
3240 :
3241 : case CONST:
3242 : CASE_CONST_ANY:
3243 : case SYMBOL_REF:
3244 : case LABEL_REF:
3245 : return false;
3246 :
3247 200111 : case REG:
3248 200111 : return reg_equiv[REGNO (x)].replace == 0 && rtx_varies_p (x, 0);
3249 :
3250 0 : case ASM_OPERANDS:
3251 0 : if (MEM_VOLATILE_P (x))
3252 : return true;
3253 :
3254 : /* Fall through. */
3255 :
3256 134207 : default:
3257 134207 : break;
3258 : }
3259 :
3260 134207 : fmt = GET_RTX_FORMAT (code);
3261 328812 : for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
3262 221201 : if (fmt[i] == 'e')
3263 : {
3264 219016 : if (equiv_init_varies_p (XEXP (x, i)))
3265 : return true;
3266 : }
3267 2185 : else if (fmt[i] == 'E')
3268 : {
3269 : int j;
3270 3323 : for (j = 0; j < XVECLEN (x, i); j++)
3271 2965 : if (equiv_init_varies_p (XVECEXP (x, i, j)))
3272 : return true;
3273 : }
3274 :
3275 : return false;
3276 : }
3277 :
3278 : /* Returns true if X (used to initialize register REGNO) is movable.
3279 : X is only movable if the registers it uses have equivalent initializations
3280 : which appear to be within the same loop (or in an inner loop) and movable
3281 : or if they are not candidates for local_alloc and don't vary. */
3282 : static bool
3283 10353941 : equiv_init_movable_p (rtx x, int regno)
3284 : {
3285 13272635 : int i, j;
3286 13272635 : const char *fmt;
3287 13272635 : enum rtx_code code = GET_CODE (x);
3288 :
3289 13272635 : switch (code)
3290 : {
3291 2918694 : case SET:
3292 2918694 : return equiv_init_movable_p (SET_SRC (x), regno);
3293 :
3294 : case CLOBBER:
3295 : return false;
3296 :
3297 : case PRE_INC:
3298 : case PRE_DEC:
3299 : case POST_INC:
3300 : case POST_DEC:
3301 : case PRE_MODIFY:
3302 : case POST_MODIFY:
3303 : return false;
3304 :
3305 1742564 : case REG:
3306 1742564 : return ((reg_equiv[REGNO (x)].loop_depth >= reg_equiv[regno].loop_depth
3307 1294016 : && reg_equiv[REGNO (x)].replace)
3308 2965549 : || (REG_BASIC_BLOCK (REGNO (x)) < NUM_FIXED_BLOCKS
3309 1564223 : && ! rtx_varies_p (x, 0)));
3310 :
3311 : case UNSPEC_VOLATILE:
3312 : return false;
3313 :
3314 0 : case ASM_OPERANDS:
3315 0 : if (MEM_VOLATILE_P (x))
3316 : return false;
3317 :
3318 : /* Fall through. */
3319 :
3320 7949968 : default:
3321 7949968 : break;
3322 : }
3323 :
3324 7949968 : fmt = GET_RTX_FORMAT (code);
3325 18852415 : for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
3326 12864104 : switch (fmt[i])
3327 : {
3328 5882608 : case 'e':
3329 5882608 : if (! equiv_init_movable_p (XEXP (x, i), regno))
3330 : return false;
3331 : break;
3332 779507 : case 'E':
3333 1009322 : for (j = XVECLEN (x, i) - 1; j >= 0; j--)
3334 891230 : if (! equiv_init_movable_p (XVECEXP (x, i, j), regno))
3335 : return false;
3336 : break;
3337 : }
3338 :
3339 : return true;
3340 : }
3341 :
3342 : static bool memref_referenced_p (rtx memref, rtx x, bool read_p);
3343 :
3344 : /* Auxiliary function for memref_referenced_p. Process setting X for
3345 : MEMREF store. */
3346 : static bool
3347 849767 : process_set_for_memref_referenced_p (rtx memref, rtx x)
3348 : {
3349 : /* If we are setting a MEM, it doesn't count (its address does), but any
3350 : other SET_DEST that has a MEM in it is referencing the MEM. */
3351 849767 : if (MEM_P (x))
3352 : {
3353 680655 : if (memref_referenced_p (memref, XEXP (x, 0), true))
3354 : return true;
3355 : }
3356 169112 : else if (memref_referenced_p (memref, x, false))
3357 : return true;
3358 :
3359 : return false;
3360 : }
3361 :
3362 : /* TRUE if X references a memory location (as a read if READ_P) that
3363 : would be affected by a store to MEMREF. */
3364 : static bool
3365 4009669 : memref_referenced_p (rtx memref, rtx x, bool read_p)
3366 : {
3367 4009669 : int i, j;
3368 4009669 : const char *fmt;
3369 4009669 : enum rtx_code code = GET_CODE (x);
3370 :
3371 4009669 : switch (code)
3372 : {
3373 : case CONST:
3374 : case LABEL_REF:
3375 : case SYMBOL_REF:
3376 : CASE_CONST_ANY:
3377 : case PC:
3378 : case HIGH:
3379 : case LO_SUM:
3380 : return false;
3381 :
3382 1607116 : case REG:
3383 1607116 : return (reg_equiv[REGNO (x)].replacement
3384 1673879 : && memref_referenced_p (memref,
3385 66763 : reg_equiv[REGNO (x)].replacement, read_p));
3386 :
3387 122808 : case MEM:
3388 : /* Memory X might have another effective type than MEMREF. */
3389 122808 : if (read_p || true_dependence (memref, VOIDmode, x))
3390 110510 : return true;
3391 : break;
3392 :
3393 832291 : case SET:
3394 832291 : if (process_set_for_memref_referenced_p (memref, SET_DEST (x)))
3395 : return true;
3396 :
3397 817742 : return memref_referenced_p (memref, SET_SRC (x), true);
3398 :
3399 17476 : case CLOBBER:
3400 17476 : if (process_set_for_memref_referenced_p (memref, XEXP (x, 0)))
3401 : return true;
3402 :
3403 : return false;
3404 :
3405 0 : case PRE_DEC:
3406 0 : case POST_DEC:
3407 0 : case PRE_INC:
3408 0 : case POST_INC:
3409 0 : if (process_set_for_memref_referenced_p (memref, XEXP (x, 0)))
3410 : return true;
3411 :
3412 0 : return memref_referenced_p (memref, XEXP (x, 0), true);
3413 :
3414 0 : case POST_MODIFY:
3415 0 : case PRE_MODIFY:
3416 : /* op0 = op0 + op1 */
3417 0 : if (process_set_for_memref_referenced_p (memref, XEXP (x, 0)))
3418 : return true;
3419 :
3420 0 : if (memref_referenced_p (memref, XEXP (x, 0), true))
3421 : return true;
3422 :
3423 0 : return memref_referenced_p (memref, XEXP (x, 1), true);
3424 :
3425 : default:
3426 : break;
3427 : }
3428 :
3429 750008 : fmt = GET_RTX_FORMAT (code);
3430 2170657 : for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
3431 1452241 : switch (fmt[i])
3432 : {
3433 1400414 : case 'e':
3434 1400414 : if (memref_referenced_p (memref, XEXP (x, i), read_p))
3435 : return true;
3436 : break;
3437 21359 : case 'E':
3438 61071 : for (j = XVECLEN (x, i) - 1; j >= 0; j--)
3439 42717 : if (memref_referenced_p (memref, XVECEXP (x, i, j), read_p))
3440 : return true;
3441 : break;
3442 : }
3443 :
3444 : return false;
3445 : }
3446 :
3447 : /* TRUE if some insn in the range (START, END] references a memory location
3448 : that would be affected by a store to MEMREF.
3449 :
3450 : Callers should not call this routine if START is after END in the
3451 : RTL chain. */
3452 :
3453 : static bool
3454 622785 : memref_used_between_p (rtx memref, rtx_insn *start, rtx_insn *end)
3455 : {
3456 622785 : rtx_insn *insn;
3457 :
3458 2134722 : for (insn = NEXT_INSN (start);
3459 4253240 : insn && insn != NEXT_INSN (end);
3460 1511937 : insn = NEXT_INSN (insn))
3461 : {
3462 1622447 : if (!NONDEBUG_INSN_P (insn))
3463 790181 : continue;
3464 :
3465 832266 : if (memref_referenced_p (memref, PATTERN (insn), false))
3466 : return true;
3467 :
3468 : /* Nonconst functions may access memory. */
3469 721756 : if (CALL_P (insn) && (! RTL_CONST_CALL_P (insn)))
3470 : return true;
3471 : }
3472 :
3473 512275 : gcc_assert (insn == NEXT_INSN (end));
3474 : return false;
3475 : }
3476 :
3477 : /* Mark REG as having no known equivalence.
3478 : Some instructions might have been processed before and furnished
3479 : with REG_EQUIV notes for this register; these notes will have to be
3480 : removed.
3481 : STORE is the piece of RTL that does the non-constant / conflicting
3482 : assignment - a SET, CLOBBER or REG_INC note. It is currently not used,
3483 : but needs to be there because this function is called from note_stores. */
3484 : static void
3485 51747132 : no_equiv (rtx reg, const_rtx store ATTRIBUTE_UNUSED,
3486 : void *data ATTRIBUTE_UNUSED)
3487 : {
3488 51747132 : int regno;
3489 51747132 : rtx_insn_list *list;
3490 :
3491 51747132 : if (!REG_P (reg))
3492 : return;
3493 35827488 : regno = REGNO (reg);
3494 35827488 : reg_equiv[regno].no_equiv = 1;
3495 35827488 : list = reg_equiv[regno].init_insns;
3496 65217807 : if (list && list->insn () == NULL)
3497 : return;
3498 7208736 : reg_equiv[regno].init_insns = gen_rtx_INSN_LIST (VOIDmode, NULL_RTX, NULL);
3499 7208736 : reg_equiv[regno].replacement = NULL_RTX;
3500 : /* This doesn't matter for equivalences made for argument registers, we
3501 : should keep their initialization insns. */
3502 7208736 : if (reg_equiv[regno].is_arg_equivalence)
3503 : return;
3504 7203476 : ira_reg_equiv[regno].defined_p = false;
3505 7203476 : ira_reg_equiv[regno].caller_save_p = false;
3506 7203476 : ira_reg_equiv[regno].init_insns = NULL;
3507 8011901 : for (; list; list = list->next ())
3508 : {
3509 808425 : rtx_insn *insn = list->insn ();
3510 808425 : remove_note (insn, find_reg_note (insn, REG_EQUIV, NULL_RTX));
3511 : }
3512 : }
3513 :
3514 : /* Check whether the SUBREG is a paradoxical subreg and set the result
3515 : in PDX_SUBREGS. */
3516 :
3517 : static void
3518 85023315 : set_paradoxical_subreg (rtx_insn *insn)
3519 : {
3520 85023315 : subrtx_iterator::array_type array;
3521 538084585 : FOR_EACH_SUBRTX (iter, array, PATTERN (insn), NONCONST)
3522 : {
3523 453061270 : const_rtx subreg = *iter;
3524 453061270 : if (GET_CODE (subreg) == SUBREG)
3525 : {
3526 3048889 : const_rtx reg = SUBREG_REG (subreg);
3527 3048889 : if (REG_P (reg) && paradoxical_subreg_p (subreg))
3528 830394 : reg_equiv[REGNO (reg)].pdx_subregs = true;
3529 : }
3530 : }
3531 85023315 : }
3532 :
3533 : /* In DEBUG_INSN location adjust REGs from CLEARED_REGS bitmap to the
3534 : equivalent replacement. */
3535 :
3536 : static rtx
3537 44246735 : adjust_cleared_regs (rtx loc, const_rtx old_rtx ATTRIBUTE_UNUSED, void *data)
3538 : {
3539 44246735 : if (REG_P (loc))
3540 : {
3541 6660139 : bitmap cleared_regs = (bitmap) data;
3542 6660139 : if (bitmap_bit_p (cleared_regs, REGNO (loc)))
3543 17460 : return simplify_replace_fn_rtx (copy_rtx (*reg_equiv[REGNO (loc)].src_p),
3544 17460 : NULL_RTX, adjust_cleared_regs, data);
3545 : }
3546 : return NULL_RTX;
3547 : }
3548 :
3549 : /* Given register REGNO is set only once, return true if the defining
3550 : insn dominates all uses. */
3551 :
3552 : static bool
3553 50314 : def_dominates_uses (int regno)
3554 : {
3555 50314 : df_ref def = DF_REG_DEF_CHAIN (regno);
3556 :
3557 50314 : struct df_insn_info *def_info = DF_REF_INSN_INFO (def);
3558 : /* If this is an artificial def (eh handler regs, hard frame pointer
3559 : for non-local goto, regs defined on function entry) then def_info
3560 : is NULL and the reg is always live before any use. We might
3561 : reasonably return true in that case, but since the only call
3562 : of this function is currently here in ira.cc when we are looking
3563 : at a defining insn we can't have an artificial def as that would
3564 : bump DF_REG_DEF_COUNT. */
3565 50314 : gcc_assert (DF_REG_DEF_COUNT (regno) == 1 && def_info != NULL);
3566 :
3567 50314 : rtx_insn *def_insn = DF_REF_INSN (def);
3568 50314 : basic_block def_bb = BLOCK_FOR_INSN (def_insn);
3569 :
3570 50314 : for (df_ref use = DF_REG_USE_CHAIN (regno);
3571 144692 : use;
3572 94378 : use = DF_REF_NEXT_REG (use))
3573 : {
3574 94378 : struct df_insn_info *use_info = DF_REF_INSN_INFO (use);
3575 : /* Only check real uses, not artificial ones. */
3576 94378 : if (use_info)
3577 : {
3578 94378 : rtx_insn *use_insn = DF_REF_INSN (use);
3579 94378 : if (!DEBUG_INSN_P (use_insn))
3580 : {
3581 94128 : basic_block use_bb = BLOCK_FOR_INSN (use_insn);
3582 94128 : if (use_bb != def_bb
3583 94128 : ? !dominated_by_p (CDI_DOMINATORS, use_bb, def_bb)
3584 55430 : : DF_INSN_INFO_LUID (use_info) < DF_INSN_INFO_LUID (def_info))
3585 : return false;
3586 : }
3587 : }
3588 : }
3589 : return true;
3590 : }
3591 :
3592 : /* Scan the instructions before update_equiv_regs. Record which registers
3593 : are referenced as paradoxical subregs. Also check for cases in which
3594 : the current function needs to save a register that one of its call
3595 : instructions clobbers.
3596 :
3597 : These things are logically unrelated, but it's more efficient to do
3598 : them together. */
3599 :
3600 : static void
3601 1504950 : update_equiv_regs_prescan (void)
3602 : {
3603 1504950 : basic_block bb;
3604 1504950 : rtx_insn *insn;
3605 1504950 : function_abi_aggregator callee_abis;
3606 :
3607 16134410 : FOR_EACH_BB_FN (bb, cfun)
3608 178419051 : FOR_BB_INSNS (bb, insn)
3609 163789591 : if (NONDEBUG_INSN_P (insn))
3610 : {
3611 85023315 : set_paradoxical_subreg (insn);
3612 85023315 : if (CALL_P (insn))
3613 6110291 : callee_abis.note_callee_abi (insn_callee_abi (insn));
3614 : }
3615 :
3616 1504950 : HARD_REG_SET extra_caller_saves = callee_abis.caller_save_regs (*crtl->abi);
3617 :
3618 1504950 : hard_reg_set_iterator hrsi;
3619 1504950 : unsigned int regno = 0;
3620 3009900 : if (!hard_reg_set_empty_p (extra_caller_saves))
3621 : {
3622 526035 : EXECUTE_IF_SET_IN_HARD_REG_SET (extra_caller_saves, 0, regno, hrsi)
3623 484908 : df_set_regs_ever_live (regno, true);
3624 : }
3625 1504950 : }
3626 :
3627 : /* Find registers that are equivalent to a single value throughout the
3628 : compilation (either because they can be referenced in memory or are
3629 : set once from a single constant). Lower their priority for a
3630 : register.
3631 :
3632 : If such a register is only referenced once, try substituting its
3633 : value into the using insn. If it succeeds, we can eliminate the
3634 : register completely.
3635 :
3636 : Initialize init_insns in ira_reg_equiv array. */
3637 : static void
3638 1504950 : update_equiv_regs (void)
3639 : {
3640 1504950 : rtx_insn *insn;
3641 1504950 : basic_block bb;
3642 :
3643 : /* Scan the insns and find which registers have equivalences. Do this
3644 : in a separate scan of the insns because (due to -fcse-follow-jumps)
3645 : a register can be set below its use. */
3646 1504950 : bitmap setjmp_crosses = regstat_get_setjmp_crosses ();
3647 16134410 : FOR_EACH_BB_FN (bb, cfun)
3648 : {
3649 14629460 : int loop_depth = bb_loop_depth (bb);
3650 :
3651 178419051 : for (insn = BB_HEAD (bb);
3652 178419051 : insn != NEXT_INSN (BB_END (bb));
3653 163789591 : insn = NEXT_INSN (insn))
3654 : {
3655 163789591 : rtx note;
3656 163789591 : rtx set;
3657 163789591 : rtx dest, src;
3658 163789591 : int regno;
3659 :
3660 163789591 : if (! INSN_P (insn))
3661 26985256 : continue;
3662 :
3663 223675394 : for (note = REG_NOTES (insn); note; note = XEXP (note, 1))
3664 86871059 : if (REG_NOTE_KIND (note) == REG_INC)
3665 0 : no_equiv (XEXP (note, 0), note, NULL);
3666 :
3667 136804335 : set = single_set (insn);
3668 :
3669 : /* If this insn contains more (or less) than a single SET,
3670 : only mark all destinations as having no known equivalence. */
3671 197026184 : if (set == NULL_RTX
3672 136804335 : || side_effects_p (SET_SRC (set)))
3673 : {
3674 60221849 : note_pattern_stores (PATTERN (insn), no_equiv, NULL);
3675 60221849 : continue;
3676 : }
3677 76582486 : else if (GET_CODE (PATTERN (insn)) == PARALLEL)
3678 : {
3679 10719301 : int i;
3680 :
3681 32378610 : for (i = XVECLEN (PATTERN (insn), 0) - 1; i >= 0; i--)
3682 : {
3683 21659309 : rtx part = XVECEXP (PATTERN (insn), 0, i);
3684 21659309 : if (part != set)
3685 10940008 : note_pattern_stores (part, no_equiv, NULL);
3686 : }
3687 : }
3688 :
3689 76582486 : dest = SET_DEST (set);
3690 76582486 : src = SET_SRC (set);
3691 :
3692 : /* See if this is setting up the equivalence between an argument
3693 : register and its stack slot. */
3694 76582486 : note = find_reg_note (insn, REG_EQUIV, NULL_RTX);
3695 76582486 : if (note)
3696 : {
3697 231616 : gcc_assert (REG_P (dest));
3698 231616 : regno = REGNO (dest);
3699 :
3700 : /* Note that we don't want to clear init_insns in
3701 : ira_reg_equiv even if there are multiple sets of this
3702 : register. */
3703 231616 : reg_equiv[regno].is_arg_equivalence = 1;
3704 :
3705 : /* The insn result can have equivalence memory although
3706 : the equivalence is not set up by the insn. We add
3707 : this insn to init insns as it is a flag for now that
3708 : regno has an equivalence. We will remove the insn
3709 : from init insn list later. */
3710 231616 : if (rtx_equal_p (src, XEXP (note, 0)) || MEM_P (XEXP (note, 0)))
3711 231616 : ira_reg_equiv[regno].init_insns
3712 231616 : = gen_rtx_INSN_LIST (VOIDmode, insn,
3713 231616 : ira_reg_equiv[regno].init_insns);
3714 :
3715 : /* Continue normally in case this is a candidate for
3716 : replacements. */
3717 : }
3718 :
3719 76582486 : if (!optimize)
3720 23247528 : continue;
3721 :
3722 : /* We only handle the case of a pseudo register being set
3723 : once, or always to the same value. */
3724 : /* ??? The mn10200 port breaks if we add equivalences for
3725 : values that need an ADDRESS_REGS register and set them equivalent
3726 : to a MEM of a pseudo. The actual problem is in the over-conservative
3727 : handling of INPADDR_ADDRESS / INPUT_ADDRESS / INPUT triples in
3728 : calculate_needs, but we traditionally work around this problem
3729 : here by rejecting equivalences when the destination is in a register
3730 : that's likely spilled. This is fragile, of course, since the
3731 : preferred class of a pseudo depends on all instructions that set
3732 : or use it. */
3733 :
3734 88150114 : if (!REG_P (dest)
3735 37034385 : || (regno = REGNO (dest)) < FIRST_PSEUDO_REGISTER
3736 21138961 : || (reg_equiv[regno].init_insns
3737 3440564 : && reg_equiv[regno].init_insns->insn () == NULL)
3738 71854823 : || (targetm.class_likely_spilled_p (reg_preferred_class (regno))
3739 350 : && MEM_P (src) && ! reg_equiv[regno].is_arg_equivalence))
3740 : {
3741 : /* This might be setting a SUBREG of a pseudo, a pseudo that is
3742 : also set somewhere else to a constant. */
3743 34815156 : note_pattern_stores (set, no_equiv, NULL);
3744 34815156 : continue;
3745 : }
3746 :
3747 : /* Don't set reg mentioned in a paradoxical subreg
3748 : equivalent to a mem. */
3749 18519802 : if (MEM_P (src) && reg_equiv[regno].pdx_subregs)
3750 : {
3751 17672 : note_pattern_stores (set, no_equiv, NULL);
3752 17672 : continue;
3753 : }
3754 :
3755 18502130 : note = find_reg_note (insn, REG_EQUAL, NULL_RTX);
3756 :
3757 : /* cse sometimes generates function invariants, but doesn't put a
3758 : REG_EQUAL note on the insn. Since this note would be redundant,
3759 : there's no point creating it earlier than here. */
3760 18502130 : if (! note && ! rtx_varies_p (src, 0))
3761 2751644 : note = set_unique_reg_note (insn, REG_EQUAL, copy_rtx (src));
3762 :
3763 : /* Don't bother considering a REG_EQUAL note containing an EXPR_LIST
3764 : since it represents a function call. */
3765 18502130 : if (note && GET_CODE (XEXP (note, 0)) == EXPR_LIST)
3766 14713060 : note = NULL_RTX;
3767 :
3768 18502130 : if (DF_REG_DEF_COUNT (regno) != 1)
3769 : {
3770 2875090 : bool equal_p = true;
3771 2875090 : rtx_insn_list *list;
3772 :
3773 : /* If we have already processed this pseudo and determined it
3774 : cannot have an equivalence, then honor that decision. */
3775 2875090 : if (reg_equiv[regno].no_equiv)
3776 0 : continue;
3777 :
3778 4681667 : if (! note
3779 1101983 : || rtx_varies_p (XEXP (note, 0), 0)
3780 3943603 : || (reg_equiv[regno].replacement
3781 0 : && ! rtx_equal_p (XEXP (note, 0),
3782 : reg_equiv[regno].replacement)))
3783 : {
3784 1806577 : no_equiv (dest, set, NULL);
3785 1806577 : continue;
3786 : }
3787 :
3788 1068513 : list = reg_equiv[regno].init_insns;
3789 2032886 : for (; list; list = list->next ())
3790 : {
3791 1044942 : rtx note_tmp;
3792 1044942 : rtx_insn *insn_tmp;
3793 :
3794 1044942 : insn_tmp = list->insn ();
3795 1044942 : note_tmp = find_reg_note (insn_tmp, REG_EQUAL, NULL_RTX);
3796 1044942 : gcc_assert (note_tmp);
3797 1044942 : if (! rtx_equal_p (XEXP (note, 0), XEXP (note_tmp, 0)))
3798 : {
3799 : equal_p = false;
3800 : break;
3801 : }
3802 : }
3803 :
3804 1068513 : if (! equal_p)
3805 : {
3806 80569 : no_equiv (dest, set, NULL);
3807 80569 : continue;
3808 : }
3809 : }
3810 :
3811 : /* Record this insn as initializing this register. */
3812 16614984 : reg_equiv[regno].init_insns
3813 16614984 : = gen_rtx_INSN_LIST (VOIDmode, insn, reg_equiv[regno].init_insns);
3814 :
3815 : /* If this register is known to be equal to a constant, record that
3816 : it is always equivalent to the constant.
3817 : Note that it is possible to have a register use before
3818 : the def in loops (see gcc.c-torture/execute/pr79286.c)
3819 : where the reg is undefined on first use. If the def insn
3820 : won't trap we can use it as an equivalence, effectively
3821 : choosing the "undefined" value for the reg to be the
3822 : same as the value set by the def. */
3823 16614984 : if (DF_REG_DEF_COUNT (regno) == 1
3824 15627040 : && note
3825 2687087 : && !rtx_varies_p (XEXP (note, 0), 0)
3826 18855059 : && (!may_trap_or_fault_p (XEXP (note, 0))
3827 50314 : || def_dominates_uses (regno)))
3828 : {
3829 2240075 : rtx note_value = XEXP (note, 0);
3830 2240075 : remove_note (insn, note);
3831 2240075 : set_unique_reg_note (insn, REG_EQUIV, note_value);
3832 : }
3833 :
3834 : /* If this insn introduces a "constant" register, decrease the priority
3835 : of that register. Record this insn if the register is only used once
3836 : more and the equivalence value is the same as our source.
3837 :
3838 : The latter condition is checked for two reasons: First, it is an
3839 : indication that it may be more efficient to actually emit the insn
3840 : as written (if no registers are available, reload will substitute
3841 : the equivalence). Secondly, it avoids problems with any registers
3842 : dying in this insn whose death notes would be missed.
3843 :
3844 : If we don't have a REG_EQUIV note, see if this insn is loading
3845 : a register used only in one basic block from a MEM. If so, and the
3846 : MEM remains unchanged for the life of the register, add a REG_EQUIV
3847 : note. */
3848 16614984 : note = find_reg_note (insn, REG_EQUIV, NULL_RTX);
3849 :
3850 16614984 : rtx replacement = NULL_RTX;
3851 16614984 : if (note)
3852 2464273 : replacement = XEXP (note, 0);
3853 14150711 : else if (REG_BASIC_BLOCK (regno) >= NUM_FIXED_BLOCKS
3854 10135584 : && MEM_P (SET_SRC (set)))
3855 : {
3856 2944643 : enum valid_equiv validity;
3857 2944643 : validity = validate_equiv_mem (insn, dest, SET_SRC (set));
3858 2944643 : if (validity != valid_none)
3859 : {
3860 2160250 : replacement = copy_rtx (SET_SRC (set));
3861 2160250 : if (validity == valid_reload)
3862 : {
3863 2159439 : note = set_unique_reg_note (insn, REG_EQUIV, replacement);
3864 : }
3865 811 : else if (ira_use_lra_p)
3866 : {
3867 : /* We still can use this equivalence for caller save
3868 : optimization in LRA. Mark this. */
3869 811 : ira_reg_equiv[regno].caller_save_p = true;
3870 811 : ira_reg_equiv[regno].init_insns
3871 811 : = gen_rtx_INSN_LIST (VOIDmode, insn,
3872 811 : ira_reg_equiv[regno].init_insns);
3873 : }
3874 : }
3875 : }
3876 :
3877 : /* If we haven't done so, record for reload that this is an
3878 : equivalencing insn. */
3879 16614984 : if (note && !reg_equiv[regno].is_arg_equivalence)
3880 4399514 : ira_reg_equiv[regno].init_insns
3881 4399514 : = gen_rtx_INSN_LIST (VOIDmode, insn,
3882 4399514 : ira_reg_equiv[regno].init_insns);
3883 :
3884 16614984 : if (replacement)
3885 : {
3886 4624523 : reg_equiv[regno].replacement = replacement;
3887 4624523 : reg_equiv[regno].src_p = &SET_SRC (set);
3888 4624523 : reg_equiv[regno].loop_depth = (short) loop_depth;
3889 :
3890 : /* Don't mess with things live during setjmp. */
3891 4624523 : if (optimize && !bitmap_bit_p (setjmp_crosses, regno))
3892 : {
3893 : /* If the register is referenced exactly twice, meaning it is
3894 : set once and used once, indicate that the reference may be
3895 : replaced by the equivalence we computed above. Do this
3896 : even if the register is only used in one block so that
3897 : dependencies can be handled where the last register is
3898 : used in a different block (i.e. HIGH / LO_SUM sequences)
3899 : and to reduce the number of registers alive across
3900 : calls. */
3901 :
3902 4624472 : if (REG_N_REFS (regno) == 2
3903 3654011 : && (rtx_equal_p (replacement, src)
3904 386501 : || ! equiv_init_varies_p (src))
3905 3580103 : && NONJUMP_INSN_P (insn)
3906 8204575 : && equiv_init_movable_p (PATTERN (insn), regno))
3907 2179697 : reg_equiv[regno].replace = 1;
3908 : }
3909 : }
3910 : }
3911 : }
3912 1504950 : }
3913 :
3914 : /* For insns that set a MEM to the contents of a REG that is only used
3915 : in a single basic block, see if the register is always equivalent
3916 : to that memory location and if moving the store from INSN to the
3917 : insn that sets REG is safe. If so, put a REG_EQUIV note on the
3918 : initializing insn. */
3919 : static void
3920 976443 : add_store_equivs (void)
3921 : {
3922 976443 : auto_sbitmap seen_insns (get_max_uid () + 1);
3923 976443 : bitmap_clear (seen_insns);
3924 :
3925 132310712 : for (rtx_insn *insn = get_insns (); insn; insn = NEXT_INSN (insn))
3926 : {
3927 131334269 : rtx set, src, dest;
3928 131334269 : unsigned regno;
3929 131334269 : rtx_insn *init_insn;
3930 :
3931 131334269 : bitmap_set_bit (seen_insns, INSN_UID (insn));
3932 :
3933 131334269 : if (! INSN_P (insn))
3934 24100170 : continue;
3935 :
3936 107234099 : set = single_set (insn);
3937 107234099 : if (! set)
3938 55300347 : continue;
3939 :
3940 51933752 : dest = SET_DEST (set);
3941 51933752 : src = SET_SRC (set);
3942 :
3943 : /* Don't add a REG_EQUIV note if the insn already has one. The existing
3944 : REG_EQUIV is likely more useful than the one we are adding. */
3945 7840910 : if (MEM_P (dest) && REG_P (src)
3946 5242769 : && (regno = REGNO (src)) >= FIRST_PSEUDO_REGISTER
3947 5175995 : && REG_BASIC_BLOCK (regno) >= NUM_FIXED_BLOCKS
3948 2946802 : && DF_REG_DEF_COUNT (regno) == 1
3949 2881318 : && ! reg_equiv[regno].pdx_subregs
3950 2737766 : && reg_equiv[regno].init_insns != NULL
3951 2737766 : && (init_insn = reg_equiv[regno].init_insns->insn ()) != 0
3952 2684191 : && bitmap_bit_p (seen_insns, INSN_UID (init_insn))
3953 2684191 : && ! find_reg_note (init_insn, REG_EQUIV, NULL_RTX)
3954 1100594 : && validate_equiv_mem (init_insn, src, dest) == valid_reload
3955 622785 : && ! memref_used_between_p (dest, init_insn, insn)
3956 : /* Attaching a REG_EQUIV note will fail if INIT_INSN has
3957 : multiple sets. */
3958 52446027 : && set_unique_reg_note (init_insn, REG_EQUIV, copy_rtx (dest)))
3959 : {
3960 : /* This insn makes the equivalence, not the one initializing
3961 : the register. */
3962 511785 : ira_reg_equiv[regno].init_insns
3963 511785 : = gen_rtx_INSN_LIST (VOIDmode, insn, NULL_RTX);
3964 511785 : df_notes_rescan (init_insn);
3965 511785 : if (dump_file)
3966 88 : fprintf (dump_file,
3967 : "Adding REG_EQUIV to insn %d for source of insn %d\n",
3968 88 : INSN_UID (init_insn),
3969 88 : INSN_UID (insn));
3970 : }
3971 : }
3972 976443 : }
3973 :
3974 : /* Scan all regs killed in an insn to see if any of them are registers
3975 : only used that once. If so, see if we can replace the reference
3976 : with the equivalent form. If we can, delete the initializing
3977 : reference and this register will go away. If we can't replace the
3978 : reference, and the initializing reference is within the same loop
3979 : (or in an inner loop), then move the register initialization just
3980 : before the use, so that they are in the same basic block. */
3981 : static void
3982 1057006 : combine_and_move_insns (void)
3983 : {
3984 1057006 : auto_bitmap cleared_regs;
3985 1057006 : int max = max_reg_num ();
3986 :
3987 52747409 : for (int regno = FIRST_PSEUDO_REGISTER; regno < max; regno++)
3988 : {
3989 51690403 : if (!reg_equiv[regno].replace)
3990 49510919 : continue;
3991 :
3992 2179484 : rtx_insn *use_insn = 0;
3993 2179484 : bool multiple_insns = false;
3994 2179484 : for (df_ref use = DF_REG_USE_CHAIN (regno);
3995 4379559 : use;
3996 2200075 : use = DF_REF_NEXT_REG (use))
3997 2200075 : if (DF_REF_INSN_INFO (use))
3998 : {
3999 2200075 : if (DEBUG_INSN_P (DF_REF_INSN (use)))
4000 20589 : continue;
4001 2179486 : if (use_insn && DF_REF_INSN (use) != use_insn)
4002 : {
4003 : multiple_insns = true;
4004 : break;
4005 : }
4006 : use_insn = DF_REF_INSN (use);
4007 : }
4008 2179484 : gcc_assert (use_insn);
4009 :
4010 : /* If a register is used by more than one insn, we cannot trivially move
4011 : or delete the definition anymore. */
4012 2179484 : if (multiple_insns)
4013 0 : continue;
4014 :
4015 : /* Don't substitute into jumps. indirect_jump_optimize does
4016 : this for anything we are prepared to handle. */
4017 2179484 : if (JUMP_P (use_insn))
4018 400 : continue;
4019 :
4020 : /* Also don't substitute into a conditional trap insn -- it can become
4021 : an unconditional trap, and that is a flow control insn. */
4022 2179084 : if (GET_CODE (PATTERN (use_insn)) == TRAP_IF)
4023 0 : continue;
4024 :
4025 2179084 : df_ref def = DF_REG_DEF_CHAIN (regno);
4026 2179084 : gcc_assert (DF_REG_DEF_COUNT (regno) == 1 && DF_REF_INSN_INFO (def));
4027 2179084 : rtx_insn *def_insn = DF_REF_INSN (def);
4028 :
4029 : /* We may not move instructions that can throw, since that
4030 : changes basic block boundaries and we are not prepared to
4031 : adjust the CFG to match. */
4032 2179084 : if (can_throw_internal (def_insn))
4033 0 : continue;
4034 :
4035 : /* Instructions with multiple sets can only be moved if DF analysis is
4036 : performed for all of the registers set. See PR91052. */
4037 2179084 : if (multiple_sets (def_insn))
4038 0 : continue;
4039 :
4040 2179084 : basic_block use_bb = BLOCK_FOR_INSN (use_insn);
4041 2179084 : basic_block def_bb = BLOCK_FOR_INSN (def_insn);
4042 2179084 : if (bb_loop_depth (use_bb) > bb_loop_depth (def_bb))
4043 135238 : continue;
4044 :
4045 2043846 : if (asm_noperands (PATTERN (def_insn)) < 0
4046 4087692 : && validate_replace_rtx (regno_reg_rtx[regno],
4047 2043846 : *reg_equiv[regno].src_p, use_insn))
4048 : {
4049 384765 : rtx link;
4050 : /* Append the REG_DEAD notes from def_insn. */
4051 770429 : for (rtx *p = ®_NOTES (def_insn); (link = *p) != 0; )
4052 : {
4053 385664 : if (REG_NOTE_KIND (link) == REG_DEAD)
4054 : {
4055 344 : *p = XEXP (link, 1);
4056 344 : XEXP (link, 1) = REG_NOTES (use_insn);
4057 344 : REG_NOTES (use_insn) = link;
4058 : }
4059 : else
4060 385320 : p = &XEXP (link, 1);
4061 : }
4062 :
4063 384765 : remove_death (regno, use_insn);
4064 384765 : SET_REG_N_REFS (regno, 0);
4065 384765 : REG_FREQ (regno) = 0;
4066 384765 : df_ref use;
4067 463823 : FOR_EACH_INSN_USE (use, def_insn)
4068 : {
4069 79058 : unsigned int use_regno = DF_REF_REGNO (use);
4070 79058 : if (!HARD_REGISTER_NUM_P (use_regno))
4071 1188 : reg_equiv[use_regno].replace = 0;
4072 : }
4073 :
4074 384765 : delete_insn (def_insn);
4075 :
4076 384765 : reg_equiv[regno].init_insns = NULL;
4077 384765 : ira_reg_equiv[regno].init_insns = NULL;
4078 384765 : bitmap_set_bit (cleared_regs, regno);
4079 : }
4080 :
4081 : /* Move the initialization of the register to just before
4082 : USE_INSN. Update the flow information. */
4083 1659081 : else if (prev_nondebug_insn (use_insn) != def_insn)
4084 : {
4085 341973 : rtx_insn *new_insn;
4086 :
4087 341973 : new_insn = emit_insn_before (PATTERN (def_insn), use_insn);
4088 341973 : REG_NOTES (new_insn) = REG_NOTES (def_insn);
4089 341973 : REG_NOTES (def_insn) = 0;
4090 : /* Rescan it to process the notes. */
4091 341973 : df_insn_rescan (new_insn);
4092 :
4093 : /* Make sure this insn is recognized before reload begins,
4094 : otherwise eliminate_regs_in_insn will die. */
4095 341973 : INSN_CODE (new_insn) = INSN_CODE (def_insn);
4096 :
4097 341973 : delete_insn (def_insn);
4098 :
4099 341973 : XEXP (reg_equiv[regno].init_insns, 0) = new_insn;
4100 :
4101 341973 : REG_BASIC_BLOCK (regno) = use_bb->index;
4102 341973 : REG_N_CALLS_CROSSED (regno) = 0;
4103 :
4104 341973 : if (use_insn == BB_HEAD (use_bb))
4105 0 : BB_HEAD (use_bb) = new_insn;
4106 :
4107 : /* We know regno dies in use_insn, but inside a loop
4108 : REG_DEAD notes might be missing when def_insn was in
4109 : another basic block. However, when we move def_insn into
4110 : this bb we'll definitely get a REG_DEAD note and reload
4111 : will see the death. It's possible that update_equiv_regs
4112 : set up an equivalence referencing regno for a reg set by
4113 : use_insn, when regno was seen as non-local. Now that
4114 : regno is local to this block, and dies, such an
4115 : equivalence is invalid. */
4116 341973 : if (find_reg_note (use_insn, REG_EQUIV, regno_reg_rtx[regno]))
4117 : {
4118 0 : rtx set = single_set (use_insn);
4119 0 : if (set && REG_P (SET_DEST (set)))
4120 0 : no_equiv (SET_DEST (set), set, NULL);
4121 : }
4122 :
4123 341973 : ira_reg_equiv[regno].init_insns
4124 341973 : = gen_rtx_INSN_LIST (VOIDmode, new_insn, NULL_RTX);
4125 341973 : bitmap_set_bit (cleared_regs, regno);
4126 : }
4127 : }
4128 :
4129 1057006 : if (!bitmap_empty_p (cleared_regs))
4130 : {
4131 226226 : basic_block bb;
4132 :
4133 5897212 : FOR_EACH_BB_FN (bb, cfun)
4134 : {
4135 11341972 : bitmap_and_compl_into (DF_LR_IN (bb), cleared_regs);
4136 11341972 : bitmap_and_compl_into (DF_LR_OUT (bb), cleared_regs);
4137 5670986 : if (!df_live)
4138 5670986 : continue;
4139 0 : bitmap_and_compl_into (DF_LIVE_IN (bb), cleared_regs);
4140 0 : bitmap_and_compl_into (DF_LIVE_OUT (bb), cleared_regs);
4141 : }
4142 :
4143 : /* Last pass - adjust debug insns referencing cleared regs. */
4144 226226 : if (MAY_HAVE_DEBUG_BIND_INSNS)
4145 63870738 : for (rtx_insn *insn = get_insns (); insn; insn = NEXT_INSN (insn))
4146 63745300 : if (DEBUG_BIND_INSN_P (insn))
4147 : {
4148 24282496 : rtx old_loc = INSN_VAR_LOCATION_LOC (insn);
4149 24282496 : INSN_VAR_LOCATION_LOC (insn)
4150 48564992 : = simplify_replace_fn_rtx (old_loc, NULL_RTX,
4151 : adjust_cleared_regs,
4152 24282496 : (void *) cleared_regs);
4153 24282496 : if (old_loc != INSN_VAR_LOCATION_LOC (insn))
4154 17121 : df_insn_rescan (insn);
4155 : }
4156 : }
4157 1057006 : }
4158 :
4159 : /* A pass over indirect jumps, converting simple cases to direct jumps.
4160 : Combine does this optimization too, but only within a basic block. */
4161 : static void
4162 1504950 : indirect_jump_optimize (void)
4163 : {
4164 1504950 : basic_block bb;
4165 1504950 : bool rebuild_p = false;
4166 :
4167 16134412 : FOR_EACH_BB_REVERSE_FN (bb, cfun)
4168 : {
4169 14629462 : rtx_insn *insn = BB_END (bb);
4170 20388045 : if (!JUMP_P (insn)
4171 14629462 : || find_reg_note (insn, REG_NON_LOCAL_GOTO, NULL_RTX))
4172 5758583 : continue;
4173 :
4174 8870879 : rtx x = pc_set (insn);
4175 8870879 : if (!x || !REG_P (SET_SRC (x)))
4176 8869481 : continue;
4177 :
4178 1398 : int regno = REGNO (SET_SRC (x));
4179 1398 : if (DF_REG_DEF_COUNT (regno) == 1)
4180 : {
4181 1287 : df_ref def = DF_REG_DEF_CHAIN (regno);
4182 1287 : if (!DF_REF_IS_ARTIFICIAL (def))
4183 : {
4184 1287 : rtx_insn *def_insn = DF_REF_INSN (def);
4185 1287 : rtx lab = NULL_RTX;
4186 1287 : rtx set = single_set (def_insn);
4187 1287 : if (set && GET_CODE (SET_SRC (set)) == LABEL_REF)
4188 : lab = SET_SRC (set);
4189 : else
4190 : {
4191 1286 : rtx eqnote = find_reg_note (def_insn, REG_EQUAL, NULL_RTX);
4192 1286 : if (eqnote && GET_CODE (XEXP (eqnote, 0)) == LABEL_REF)
4193 : lab = XEXP (eqnote, 0);
4194 : }
4195 1 : if (lab && validate_replace_rtx (SET_SRC (x), lab, insn))
4196 : rebuild_p = true;
4197 : }
4198 : }
4199 : }
4200 :
4201 1504950 : if (rebuild_p)
4202 : {
4203 1 : timevar_push (TV_JUMP);
4204 1 : rebuild_jump_labels (get_insns ());
4205 1 : if (purge_all_dead_edges ())
4206 1 : delete_unreachable_blocks ();
4207 1 : timevar_pop (TV_JUMP);
4208 : }
4209 1504950 : }
4210 :
4211 : /* Set up fields memory, constant, and invariant from init_insns in
4212 : the structures of array ira_reg_equiv. */
4213 : static void
4214 1504950 : setup_reg_equiv (void)
4215 : {
4216 1504950 : int i;
4217 1504950 : rtx_insn_list *elem, *prev_elem, *next_elem;
4218 1504950 : rtx_insn *insn;
4219 1504950 : rtx set, x;
4220 :
4221 172008771 : for (i = FIRST_PSEUDO_REGISTER; i < ira_reg_equiv_len; i++)
4222 170503821 : for (prev_elem = NULL, elem = ira_reg_equiv[i].init_insns;
4223 175138219 : elem;
4224 : prev_elem = elem, elem = next_elem)
4225 : {
4226 4758879 : next_elem = elem->next ();
4227 4758879 : insn = elem->insn ();
4228 4758879 : set = single_set (insn);
4229 :
4230 : /* Init insns can set up equivalence when the reg is a destination or
4231 : a source (in this case the destination is memory). */
4232 4758879 : if (set != 0 && (REG_P (SET_DEST (set)) || REG_P (SET_SRC (set))))
4233 : {
4234 4758879 : if ((x = find_reg_note (insn, REG_EQUIV, NULL_RTX)) != NULL)
4235 : {
4236 4246356 : x = XEXP (x, 0);
4237 4246356 : if (REG_P (SET_DEST (set))
4238 4246356 : && REGNO (SET_DEST (set)) == (unsigned int) i
4239 8492712 : && ! rtx_equal_p (SET_SRC (set), x) && MEM_P (x))
4240 : {
4241 : /* This insn reporting the equivalence but
4242 : actually not setting it. Remove it from the
4243 : list. */
4244 30411 : if (prev_elem == NULL)
4245 30411 : ira_reg_equiv[i].init_insns = next_elem;
4246 : else
4247 0 : XEXP (prev_elem, 1) = next_elem;
4248 : elem = prev_elem;
4249 : }
4250 : }
4251 512523 : else if (REG_P (SET_DEST (set))
4252 512523 : && REGNO (SET_DEST (set)) == (unsigned int) i)
4253 738 : x = SET_SRC (set);
4254 : else
4255 : {
4256 511785 : gcc_assert (REG_P (SET_SRC (set))
4257 : && REGNO (SET_SRC (set)) == (unsigned int) i);
4258 : x = SET_DEST (set);
4259 : }
4260 : /* If PIC is enabled and the equiv is not a LEGITIMATE_PIC_OPERAND,
4261 : we can't use it. */
4262 4758879 : if (! CONSTANT_P (x)
4263 1013746 : || ! flag_pic
4264 : /* A function invariant is often CONSTANT_P but may
4265 : include a register. We promise to only pass
4266 : CONSTANT_P objects to LEGITIMATE_PIC_OPERAND_P. */
4267 4883386 : || LEGITIMATE_PIC_OPERAND_P (x))
4268 : {
4269 : /* It can happen that a REG_EQUIV note contains a MEM
4270 : that is not a legitimate memory operand. As later
4271 : stages of reload assume that all addresses found in
4272 : the lra_regno_equiv_* arrays were originally
4273 : legitimate, we ignore such REG_EQUIV notes. */
4274 4714375 : if (memory_operand (x, VOIDmode))
4275 : {
4276 2762050 : ira_reg_equiv[i].defined_p = !ira_reg_equiv[i].caller_save_p;
4277 2762050 : ira_reg_equiv[i].memory = x;
4278 2762050 : continue;
4279 : }
4280 1952325 : else if (function_invariant_p (x))
4281 : {
4282 1872854 : machine_mode mode;
4283 :
4284 1872854 : mode = GET_MODE (SET_DEST (set));
4285 1872854 : if (GET_CODE (x) == PLUS
4286 970436 : || x == frame_pointer_rtx || x == arg_pointer_rtx)
4287 : /* This is PLUS of frame pointer and a constant,
4288 : or fp, or argp. */
4289 903612 : ira_reg_equiv[i].invariant = x;
4290 969242 : else if (targetm.legitimate_constant_p (mode, x))
4291 721869 : ira_reg_equiv[i].constant = x;
4292 : else
4293 : {
4294 247373 : ira_reg_equiv[i].memory = force_const_mem (mode, x);
4295 247373 : if (ira_reg_equiv[i].memory == NULL_RTX)
4296 : {
4297 506 : ira_reg_equiv[i].defined_p = false;
4298 506 : ira_reg_equiv[i].caller_save_p = false;
4299 506 : ira_reg_equiv[i].init_insns = NULL;
4300 506 : break;
4301 : }
4302 : }
4303 1872348 : ira_reg_equiv[i].defined_p = true;
4304 1872348 : continue;
4305 1872348 : }
4306 : }
4307 : }
4308 123975 : ira_reg_equiv[i].defined_p = false;
4309 123975 : ira_reg_equiv[i].caller_save_p = false;
4310 123975 : ira_reg_equiv[i].init_insns = NULL;
4311 123975 : break;
4312 : }
4313 1504950 : }
4314 :
4315 :
4316 :
4317 : /* Print chain C to FILE. */
4318 : static void
4319 0 : print_insn_chain (FILE *file, class insn_chain *c)
4320 : {
4321 0 : fprintf (file, "insn=%d, ", INSN_UID (c->insn));
4322 0 : bitmap_print (file, &c->live_throughout, "live_throughout: ", ", ");
4323 0 : bitmap_print (file, &c->dead_or_set, "dead_or_set: ", "\n");
4324 0 : }
4325 :
4326 :
4327 : /* Print all reload_insn_chains to FILE. */
4328 : static void
4329 0 : print_insn_chains (FILE *file)
4330 : {
4331 0 : class insn_chain *c;
4332 0 : for (c = reload_insn_chain; c ; c = c->next)
4333 0 : print_insn_chain (file, c);
4334 0 : }
4335 :
4336 : /* Return true if pseudo REGNO should be added to set live_throughout
4337 : or dead_or_set of the insn chains for reload consideration. */
4338 : static bool
4339 0 : pseudo_for_reload_consideration_p (int regno)
4340 : {
4341 : /* Consider spilled pseudos too for IRA because they still have a
4342 : chance to get hard-registers in the reload when IRA is used. */
4343 0 : return (reg_renumber[regno] >= 0 || ira_conflicts_p);
4344 : }
4345 :
4346 : /* Return true if we can track the individual bytes of subreg X.
4347 : When returning true, set *OUTER_SIZE to the number of bytes in
4348 : X itself, *INNER_SIZE to the number of bytes in the inner register
4349 : and *START to the offset of the first byte. */
4350 : static bool
4351 0 : get_subreg_tracking_sizes (rtx x, HOST_WIDE_INT *outer_size,
4352 : HOST_WIDE_INT *inner_size, HOST_WIDE_INT *start)
4353 : {
4354 0 : rtx reg = regno_reg_rtx[REGNO (SUBREG_REG (x))];
4355 0 : return (GET_MODE_SIZE (GET_MODE (x)).is_constant (outer_size)
4356 0 : && GET_MODE_SIZE (GET_MODE (reg)).is_constant (inner_size)
4357 0 : && SUBREG_BYTE (x).is_constant (start));
4358 : }
4359 :
4360 : /* Init LIVE_SUBREGS[ALLOCNUM] and LIVE_SUBREGS_USED[ALLOCNUM] for
4361 : a register with SIZE bytes, making the register live if INIT_VALUE. */
4362 : static void
4363 0 : init_live_subregs (bool init_value, sbitmap *live_subregs,
4364 : bitmap live_subregs_used, int allocnum, int size)
4365 : {
4366 0 : gcc_assert (size > 0);
4367 :
4368 : /* Been there, done that. */
4369 0 : if (bitmap_bit_p (live_subregs_used, allocnum))
4370 : return;
4371 :
4372 : /* Create a new one. */
4373 0 : if (live_subregs[allocnum] == NULL)
4374 0 : live_subregs[allocnum] = sbitmap_alloc (size);
4375 :
4376 : /* If the entire reg was live before blasting into subregs, we need
4377 : to init all of the subregs to ones else init to 0. */
4378 0 : if (init_value)
4379 0 : bitmap_ones (live_subregs[allocnum]);
4380 : else
4381 0 : bitmap_clear (live_subregs[allocnum]);
4382 :
4383 0 : bitmap_set_bit (live_subregs_used, allocnum);
4384 : }
4385 :
4386 : /* Walk the insns of the current function and build reload_insn_chain,
4387 : and record register life information. */
4388 : static void
4389 0 : build_insn_chain (void)
4390 : {
4391 0 : unsigned int i;
4392 0 : class insn_chain **p = &reload_insn_chain;
4393 0 : basic_block bb;
4394 0 : class insn_chain *c = NULL;
4395 0 : class insn_chain *next = NULL;
4396 0 : auto_bitmap live_relevant_regs;
4397 0 : auto_bitmap elim_regset;
4398 : /* live_subregs is a vector used to keep accurate information about
4399 : which hardregs are live in multiword pseudos. live_subregs and
4400 : live_subregs_used are indexed by pseudo number. The live_subreg
4401 : entry for a particular pseudo is only used if the corresponding
4402 : element is non zero in live_subregs_used. The sbitmap size of
4403 : live_subreg[allocno] is number of bytes that the pseudo can
4404 : occupy. */
4405 0 : sbitmap *live_subregs = XCNEWVEC (sbitmap, max_regno);
4406 0 : auto_bitmap live_subregs_used;
4407 :
4408 0 : hard_reg_set_iterator hrsi;
4409 0 : EXECUTE_IF_SET_IN_HARD_REG_SET (eliminable_regset, 0, i, hrsi)
4410 0 : bitmap_set_bit (elim_regset, i);
4411 0 : FOR_EACH_BB_REVERSE_FN (bb, cfun)
4412 : {
4413 0 : bitmap_iterator bi;
4414 0 : rtx_insn *insn;
4415 :
4416 0 : CLEAR_REG_SET (live_relevant_regs);
4417 0 : bitmap_clear (live_subregs_used);
4418 :
4419 0 : EXECUTE_IF_SET_IN_BITMAP (df_get_live_out (bb), 0, i, bi)
4420 : {
4421 0 : if (i >= FIRST_PSEUDO_REGISTER)
4422 : break;
4423 0 : bitmap_set_bit (live_relevant_regs, i);
4424 : }
4425 :
4426 0 : EXECUTE_IF_SET_IN_BITMAP (df_get_live_out (bb),
4427 : FIRST_PSEUDO_REGISTER, i, bi)
4428 : {
4429 0 : if (pseudo_for_reload_consideration_p (i))
4430 0 : bitmap_set_bit (live_relevant_regs, i);
4431 : }
4432 :
4433 0 : FOR_BB_INSNS_REVERSE (bb, insn)
4434 : {
4435 0 : if (!NOTE_P (insn) && !BARRIER_P (insn))
4436 : {
4437 0 : struct df_insn_info *insn_info = DF_INSN_INFO_GET (insn);
4438 0 : df_ref def, use;
4439 :
4440 0 : c = new_insn_chain ();
4441 0 : c->next = next;
4442 0 : next = c;
4443 0 : *p = c;
4444 0 : p = &c->prev;
4445 :
4446 0 : c->insn = insn;
4447 0 : c->block = bb->index;
4448 :
4449 0 : if (NONDEBUG_INSN_P (insn))
4450 0 : FOR_EACH_INSN_INFO_DEF (def, insn_info)
4451 : {
4452 0 : unsigned int regno = DF_REF_REGNO (def);
4453 :
4454 : /* Ignore may clobbers because these are generated
4455 : from calls. However, every other kind of def is
4456 : added to dead_or_set. */
4457 0 : if (!DF_REF_FLAGS_IS_SET (def, DF_REF_MAY_CLOBBER))
4458 : {
4459 0 : if (regno < FIRST_PSEUDO_REGISTER)
4460 : {
4461 0 : if (!fixed_regs[regno])
4462 0 : bitmap_set_bit (&c->dead_or_set, regno);
4463 : }
4464 0 : else if (pseudo_for_reload_consideration_p (regno))
4465 0 : bitmap_set_bit (&c->dead_or_set, regno);
4466 : }
4467 :
4468 0 : if ((regno < FIRST_PSEUDO_REGISTER
4469 0 : || reg_renumber[regno] >= 0
4470 0 : || ira_conflicts_p)
4471 0 : && (!DF_REF_FLAGS_IS_SET (def, DF_REF_CONDITIONAL)))
4472 : {
4473 0 : rtx reg = DF_REF_REG (def);
4474 0 : HOST_WIDE_INT outer_size, inner_size, start;
4475 :
4476 : /* We can usually track the liveness of individual
4477 : bytes within a subreg. The only exceptions are
4478 : subregs wrapped in ZERO_EXTRACTs and subregs whose
4479 : size is not known; in those cases we need to be
4480 : conservative and treat the definition as a partial
4481 : definition of the full register rather than a full
4482 : definition of a specific part of the register. */
4483 0 : if (GET_CODE (reg) == SUBREG
4484 0 : && !DF_REF_FLAGS_IS_SET (def, DF_REF_ZERO_EXTRACT)
4485 0 : && get_subreg_tracking_sizes (reg, &outer_size,
4486 : &inner_size, &start))
4487 : {
4488 0 : HOST_WIDE_INT last = start + outer_size;
4489 :
4490 0 : init_live_subregs
4491 0 : (bitmap_bit_p (live_relevant_regs, regno),
4492 : live_subregs, live_subregs_used, regno,
4493 : inner_size);
4494 :
4495 0 : if (!DF_REF_FLAGS_IS_SET
4496 : (def, DF_REF_STRICT_LOW_PART))
4497 : {
4498 : /* Expand the range to cover entire words.
4499 : Bytes added here are "don't care". */
4500 0 : start
4501 0 : = start / UNITS_PER_WORD * UNITS_PER_WORD;
4502 0 : last = ((last + UNITS_PER_WORD - 1)
4503 0 : / UNITS_PER_WORD * UNITS_PER_WORD);
4504 : }
4505 :
4506 : /* Ignore the paradoxical bits. */
4507 0 : if (last > SBITMAP_SIZE (live_subregs[regno]))
4508 : last = SBITMAP_SIZE (live_subregs[regno]);
4509 :
4510 0 : while (start < last)
4511 : {
4512 0 : bitmap_clear_bit (live_subregs[regno], start);
4513 0 : start++;
4514 : }
4515 :
4516 0 : if (bitmap_empty_p (live_subregs[regno]))
4517 : {
4518 0 : bitmap_clear_bit (live_subregs_used, regno);
4519 0 : bitmap_clear_bit (live_relevant_regs, regno);
4520 : }
4521 : else
4522 : /* Set live_relevant_regs here because
4523 : that bit has to be true to get us to
4524 : look at the live_subregs fields. */
4525 0 : bitmap_set_bit (live_relevant_regs, regno);
4526 : }
4527 : else
4528 : {
4529 : /* DF_REF_PARTIAL is generated for
4530 : subregs, STRICT_LOW_PART, and
4531 : ZERO_EXTRACT. We handle the subreg
4532 : case above so here we have to keep from
4533 : modeling the def as a killing def. */
4534 0 : if (!DF_REF_FLAGS_IS_SET (def, DF_REF_PARTIAL))
4535 : {
4536 0 : bitmap_clear_bit (live_subregs_used, regno);
4537 0 : bitmap_clear_bit (live_relevant_regs, regno);
4538 : }
4539 : }
4540 : }
4541 : }
4542 :
4543 0 : bitmap_and_compl_into (live_relevant_regs, elim_regset);
4544 0 : bitmap_copy (&c->live_throughout, live_relevant_regs);
4545 :
4546 0 : if (NONDEBUG_INSN_P (insn))
4547 0 : FOR_EACH_INSN_INFO_USE (use, insn_info)
4548 : {
4549 0 : unsigned int regno = DF_REF_REGNO (use);
4550 0 : rtx reg = DF_REF_REG (use);
4551 :
4552 : /* DF_REF_READ_WRITE on a use means that this use
4553 : is fabricated from a def that is a partial set
4554 : to a multiword reg. Here, we only model the
4555 : subreg case that is not wrapped in ZERO_EXTRACT
4556 : precisely so we do not need to look at the
4557 : fabricated use. */
4558 0 : if (DF_REF_FLAGS_IS_SET (use, DF_REF_READ_WRITE)
4559 0 : && !DF_REF_FLAGS_IS_SET (use, DF_REF_ZERO_EXTRACT)
4560 0 : && DF_REF_FLAGS_IS_SET (use, DF_REF_SUBREG))
4561 0 : continue;
4562 :
4563 : /* Add the last use of each var to dead_or_set. */
4564 0 : if (!bitmap_bit_p (live_relevant_regs, regno))
4565 : {
4566 0 : if (regno < FIRST_PSEUDO_REGISTER)
4567 : {
4568 0 : if (!fixed_regs[regno])
4569 0 : bitmap_set_bit (&c->dead_or_set, regno);
4570 : }
4571 0 : else if (pseudo_for_reload_consideration_p (regno))
4572 0 : bitmap_set_bit (&c->dead_or_set, regno);
4573 : }
4574 :
4575 0 : if (regno < FIRST_PSEUDO_REGISTER
4576 0 : || pseudo_for_reload_consideration_p (regno))
4577 : {
4578 0 : HOST_WIDE_INT outer_size, inner_size, start;
4579 0 : if (GET_CODE (reg) == SUBREG
4580 0 : && !DF_REF_FLAGS_IS_SET (use,
4581 : DF_REF_SIGN_EXTRACT
4582 : | DF_REF_ZERO_EXTRACT)
4583 0 : && get_subreg_tracking_sizes (reg, &outer_size,
4584 : &inner_size, &start))
4585 : {
4586 0 : HOST_WIDE_INT last = start + outer_size;
4587 :
4588 0 : init_live_subregs
4589 0 : (bitmap_bit_p (live_relevant_regs, regno),
4590 : live_subregs, live_subregs_used, regno,
4591 : inner_size);
4592 :
4593 : /* Ignore the paradoxical bits. */
4594 0 : if (last > SBITMAP_SIZE (live_subregs[regno]))
4595 : last = SBITMAP_SIZE (live_subregs[regno]);
4596 :
4597 0 : while (start < last)
4598 : {
4599 0 : bitmap_set_bit (live_subregs[regno], start);
4600 0 : start++;
4601 : }
4602 : }
4603 : else
4604 : /* Resetting the live_subregs_used is
4605 : effectively saying do not use the subregs
4606 : because we are reading the whole
4607 : pseudo. */
4608 0 : bitmap_clear_bit (live_subregs_used, regno);
4609 0 : bitmap_set_bit (live_relevant_regs, regno);
4610 : }
4611 : }
4612 : }
4613 : }
4614 :
4615 : /* FIXME!! The following code is a disaster. Reload needs to see the
4616 : labels and jump tables that are just hanging out in between
4617 : the basic blocks. See pr33676. */
4618 0 : insn = BB_HEAD (bb);
4619 :
4620 : /* Skip over the barriers and cruft. */
4621 0 : while (insn && (BARRIER_P (insn) || NOTE_P (insn)
4622 0 : || BLOCK_FOR_INSN (insn) == bb))
4623 0 : insn = PREV_INSN (insn);
4624 :
4625 : /* While we add anything except barriers and notes, the focus is
4626 : to get the labels and jump tables into the
4627 : reload_insn_chain. */
4628 0 : while (insn)
4629 : {
4630 0 : if (!NOTE_P (insn) && !BARRIER_P (insn))
4631 : {
4632 0 : if (BLOCK_FOR_INSN (insn))
4633 : break;
4634 :
4635 0 : c = new_insn_chain ();
4636 0 : c->next = next;
4637 0 : next = c;
4638 0 : *p = c;
4639 0 : p = &c->prev;
4640 :
4641 : /* The block makes no sense here, but it is what the old
4642 : code did. */
4643 0 : c->block = bb->index;
4644 0 : c->insn = insn;
4645 0 : bitmap_copy (&c->live_throughout, live_relevant_regs);
4646 : }
4647 0 : insn = PREV_INSN (insn);
4648 : }
4649 : }
4650 :
4651 0 : reload_insn_chain = c;
4652 0 : *p = NULL;
4653 :
4654 0 : for (i = 0; i < (unsigned int) max_regno; i++)
4655 0 : if (live_subregs[i] != NULL)
4656 0 : sbitmap_free (live_subregs[i]);
4657 0 : free (live_subregs);
4658 :
4659 0 : if (dump_file)
4660 0 : print_insn_chains (dump_file);
4661 0 : }
4662 :
4663 : /* Examine the rtx found in *LOC, which is read or written to as determined
4664 : by TYPE. Return false if we find a reason why an insn containing this
4665 : rtx should not be moved (such as accesses to non-constant memory), true
4666 : otherwise. */
4667 : static bool
4668 6792731 : rtx_moveable_p (rtx *loc, enum op_type type)
4669 : {
4670 6801794 : const char *fmt;
4671 6801794 : rtx x = *loc;
4672 6801794 : int i, j;
4673 :
4674 6801794 : enum rtx_code code = GET_CODE (x);
4675 6801794 : switch (code)
4676 : {
4677 : case CONST:
4678 : CASE_CONST_ANY:
4679 : case SYMBOL_REF:
4680 : case LABEL_REF:
4681 : return true;
4682 :
4683 0 : case PC:
4684 0 : return type == OP_IN;
4685 :
4686 2378901 : case REG:
4687 2378901 : if (x == frame_pointer_rtx)
4688 : return true;
4689 2377889 : if (HARD_REGISTER_P (x))
4690 : return false;
4691 :
4692 : return true;
4693 :
4694 652284 : case MEM:
4695 652284 : if (type == OP_IN && MEM_READONLY_P (x))
4696 8988 : return rtx_moveable_p (&XEXP (x, 0), OP_IN);
4697 : return false;
4698 :
4699 2099559 : case SET:
4700 2099559 : return (rtx_moveable_p (&SET_SRC (x), OP_IN)
4701 2099559 : && rtx_moveable_p (&SET_DEST (x), OP_OUT));
4702 :
4703 10 : case STRICT_LOW_PART:
4704 10 : return rtx_moveable_p (&XEXP (x, 0), OP_OUT);
4705 :
4706 493 : case ZERO_EXTRACT:
4707 493 : case SIGN_EXTRACT:
4708 493 : return (rtx_moveable_p (&XEXP (x, 0), type)
4709 493 : && rtx_moveable_p (&XEXP (x, 1), OP_IN)
4710 986 : && rtx_moveable_p (&XEXP (x, 2), OP_IN));
4711 :
4712 65 : case CLOBBER:
4713 65 : return rtx_moveable_p (&SET_DEST (x), OP_OUT);
4714 :
4715 : case UNSPEC_VOLATILE:
4716 : /* It is a bad idea to consider insns with such rtl
4717 : as moveable ones. The insn scheduler also considers them as barrier
4718 : for a reason. */
4719 : return false;
4720 :
4721 0 : case ASM_OPERANDS:
4722 : /* The same is true for volatile asm: it has unknown side effects, it
4723 : cannot be moved at will. */
4724 0 : if (MEM_VOLATILE_P (x))
4725 : return false;
4726 :
4727 1112963 : default:
4728 1112963 : break;
4729 : }
4730 :
4731 1112963 : fmt = GET_RTX_FORMAT (code);
4732 2865382 : for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
4733 : {
4734 1932877 : if (fmt[i] == 'e')
4735 : {
4736 1534855 : if (!rtx_moveable_p (&XEXP (x, i), type))
4737 : return false;
4738 : }
4739 398022 : else if (fmt[i] == 'E')
4740 526649 : for (j = XVECLEN (x, i) - 1; j >= 0; j--)
4741 : {
4742 404485 : if (!rtx_moveable_p (&XVECEXP (x, i, j), type))
4743 : return false;
4744 : }
4745 : }
4746 : return true;
4747 : }
4748 :
4749 : /* A wrapper around dominated_by_p, which uses the information in UID_LUID
4750 : to give dominance relationships between two insns I1 and I2. */
4751 : static bool
4752 21596269 : insn_dominated_by_p (rtx i1, rtx i2, int *uid_luid)
4753 : {
4754 21596269 : basic_block bb1 = BLOCK_FOR_INSN (i1);
4755 21596269 : basic_block bb2 = BLOCK_FOR_INSN (i2);
4756 :
4757 21596269 : if (bb1 == bb2)
4758 11431774 : return uid_luid[INSN_UID (i2)] < uid_luid[INSN_UID (i1)];
4759 10164495 : return dominated_by_p (CDI_DOMINATORS, bb1, bb2);
4760 : }
4761 :
4762 : /* Record the range of register numbers added by find_moveable_pseudos. */
4763 : int first_moveable_pseudo, last_moveable_pseudo;
4764 :
4765 : /* These two vectors hold data for every register added by
4766 : find_movable_pseudos, with index 0 holding data for the
4767 : first_moveable_pseudo. */
4768 : /* The original home register. */
4769 : static vec<rtx> pseudo_replaced_reg;
4770 :
4771 : /* Look for instances where we have an instruction that is known to increase
4772 : register pressure, and whose result is not used immediately. If it is
4773 : possible to move the instruction downwards to just before its first use,
4774 : split its lifetime into two ranges. We create a new pseudo to compute the
4775 : value, and emit a move instruction just before the first use. If, after
4776 : register allocation, the new pseudo remains unallocated, the function
4777 : move_unallocated_pseudos then deletes the move instruction and places
4778 : the computation just before the first use.
4779 :
4780 : Such a move is safe and profitable if all the input registers remain live
4781 : and unchanged between the original computation and its first use. In such
4782 : a situation, the computation is known to increase register pressure, and
4783 : moving it is known to at least not worsen it.
4784 :
4785 : We restrict moves to only those cases where a register remains unallocated,
4786 : in order to avoid interfering too much with the instruction schedule. As
4787 : an exception, we may move insns which only modify their input register
4788 : (typically induction variables), as this increases the freedom for our
4789 : intended transformation, and does not limit the second instruction
4790 : scheduler pass. */
4791 :
4792 : static void
4793 1057059 : find_moveable_pseudos (void)
4794 : {
4795 1057059 : unsigned i;
4796 1057059 : int max_regs = max_reg_num ();
4797 1057059 : int max_uid = get_max_uid ();
4798 1057059 : basic_block bb;
4799 1057059 : int *uid_luid = XNEWVEC (int, max_uid);
4800 1057059 : rtx_insn **closest_uses = XNEWVEC (rtx_insn *, max_regs);
4801 : /* A set of registers which are live but not modified throughout a block. */
4802 1057059 : bitmap_head *bb_transp_live = XNEWVEC (bitmap_head,
4803 : last_basic_block_for_fn (cfun));
4804 : /* A set of registers which only exist in a given basic block. */
4805 1057059 : bitmap_head *bb_local = XNEWVEC (bitmap_head,
4806 : last_basic_block_for_fn (cfun));
4807 : /* A set of registers which are set once, in an instruction that can be
4808 : moved freely downwards, but are otherwise transparent to a block. */
4809 1057059 : bitmap_head *bb_moveable_reg_sets = XNEWVEC (bitmap_head,
4810 : last_basic_block_for_fn (cfun));
4811 1057059 : auto_bitmap live, used, set, interesting, unusable_as_input;
4812 1057059 : bitmap_iterator bi;
4813 :
4814 1057059 : first_moveable_pseudo = max_regs;
4815 1057059 : pseudo_replaced_reg.release ();
4816 1057059 : pseudo_replaced_reg.safe_grow_cleared (max_regs, true);
4817 :
4818 1057059 : df_analyze ();
4819 1057059 : calculate_dominance_info (CDI_DOMINATORS);
4820 :
4821 1057059 : i = 0;
4822 12184562 : FOR_EACH_BB_FN (bb, cfun)
4823 : {
4824 11127503 : rtx_insn *insn;
4825 11127503 : bitmap transp = bb_transp_live + bb->index;
4826 11127503 : bitmap moveable = bb_moveable_reg_sets + bb->index;
4827 11127503 : bitmap local = bb_local + bb->index;
4828 :
4829 11127503 : bitmap_initialize (local, 0);
4830 11127503 : bitmap_initialize (transp, 0);
4831 11127503 : bitmap_initialize (moveable, 0);
4832 11127503 : bitmap_copy (live, df_get_live_out (bb));
4833 11127503 : bitmap_and_into (live, df_get_live_in (bb));
4834 11127503 : bitmap_copy (transp, live);
4835 11127503 : bitmap_clear (moveable);
4836 11127503 : bitmap_clear (live);
4837 11127503 : bitmap_clear (used);
4838 11127503 : bitmap_clear (set);
4839 143765403 : FOR_BB_INSNS (bb, insn)
4840 132637900 : if (NONDEBUG_INSN_P (insn))
4841 : {
4842 59359741 : df_insn_info *insn_info = DF_INSN_INFO_GET (insn);
4843 59359741 : df_ref def, use;
4844 :
4845 59359741 : uid_luid[INSN_UID (insn)] = i++;
4846 :
4847 59359741 : def = df_single_def (insn_info);
4848 59359741 : use = df_single_use (insn_info);
4849 59359741 : if (use
4850 59359741 : && def
4851 19584306 : && DF_REF_REGNO (use) == DF_REF_REGNO (def)
4852 624837 : && !bitmap_bit_p (set, DF_REF_REGNO (use))
4853 59443615 : && rtx_moveable_p (&PATTERN (insn), OP_IN))
4854 : {
4855 35176 : unsigned regno = DF_REF_REGNO (use);
4856 35176 : bitmap_set_bit (moveable, regno);
4857 35176 : bitmap_set_bit (set, regno);
4858 35176 : bitmap_set_bit (used, regno);
4859 35176 : bitmap_clear_bit (transp, regno);
4860 35176 : continue;
4861 35176 : }
4862 132980923 : FOR_EACH_INSN_INFO_USE (use, insn_info)
4863 : {
4864 73656358 : unsigned regno = DF_REF_REGNO (use);
4865 73656358 : bitmap_set_bit (used, regno);
4866 73656358 : if (bitmap_clear_bit (moveable, regno))
4867 15910 : bitmap_clear_bit (transp, regno);
4868 : }
4869 :
4870 494175062 : FOR_EACH_INSN_INFO_DEF (def, insn_info)
4871 : {
4872 434850497 : unsigned regno = DF_REF_REGNO (def);
4873 434850497 : bitmap_set_bit (set, regno);
4874 434850497 : bitmap_clear_bit (transp, regno);
4875 434850497 : bitmap_clear_bit (moveable, regno);
4876 : }
4877 : }
4878 : }
4879 :
4880 12184562 : FOR_EACH_BB_FN (bb, cfun)
4881 : {
4882 11127503 : bitmap local = bb_local + bb->index;
4883 11127503 : rtx_insn *insn;
4884 :
4885 143765403 : FOR_BB_INSNS (bb, insn)
4886 132637900 : if (NONDEBUG_INSN_P (insn))
4887 : {
4888 59359741 : df_insn_info *insn_info = DF_INSN_INFO_GET (insn);
4889 59359741 : rtx_insn *def_insn;
4890 59359741 : rtx closest_use, note;
4891 59359741 : df_ref def, use;
4892 59359741 : unsigned regno;
4893 59359741 : bool all_dominated, all_local;
4894 59359741 : machine_mode mode;
4895 :
4896 59359741 : def = df_single_def (insn_info);
4897 : /* There must be exactly one def in this insn. */
4898 32410087 : if (!def || !single_set (insn))
4899 27034248 : continue;
4900 : /* This must be the only definition of the reg. We also limit
4901 : which modes we deal with so that we can assume we can generate
4902 : move instructions. */
4903 32325493 : regno = DF_REF_REGNO (def);
4904 32325493 : mode = GET_MODE (DF_REF_REG (def));
4905 32325493 : if (DF_REG_DEF_COUNT (regno) != 1
4906 12289783 : || !DF_REF_INSN_INFO (def)
4907 12289783 : || HARD_REGISTER_NUM_P (regno)
4908 12256745 : || DF_REG_EQ_USE_COUNT (regno) > 0
4909 11739276 : || (!INTEGRAL_MODE_P (mode)
4910 : && !FLOAT_MODE_P (mode)
4911 : && !OPAQUE_MODE_P (mode)))
4912 20586217 : continue;
4913 11739276 : def_insn = DF_REF_INSN (def);
4914 :
4915 19902184 : for (note = REG_NOTES (def_insn); note; note = XEXP (note, 1))
4916 10733818 : if (REG_NOTE_KIND (note) == REG_EQUIV && MEM_P (XEXP (note, 0)))
4917 : break;
4918 :
4919 11739276 : if (note)
4920 : {
4921 2570910 : if (dump_file)
4922 68 : fprintf (dump_file, "Ignoring reg %d, has equiv memory\n",
4923 : regno);
4924 2570910 : bitmap_set_bit (unusable_as_input, regno);
4925 2570910 : continue;
4926 : }
4927 :
4928 9168366 : use = DF_REG_USE_CHAIN (regno);
4929 9168366 : all_dominated = true;
4930 9168366 : all_local = true;
4931 9168366 : closest_use = NULL_RTX;
4932 27112550 : for (; use; use = DF_REF_NEXT_REG (use))
4933 : {
4934 17944184 : rtx_insn *insn;
4935 17944184 : if (!DF_REF_INSN_INFO (use))
4936 : {
4937 : all_dominated = false;
4938 : all_local = false;
4939 : break;
4940 : }
4941 17944184 : insn = DF_REF_INSN (use);
4942 17944184 : if (DEBUG_INSN_P (insn))
4943 2344396 : continue;
4944 15599788 : if (BLOCK_FOR_INSN (insn) != BLOCK_FOR_INSN (def_insn))
4945 6174877 : all_local = false;
4946 15599788 : if (!insn_dominated_by_p (insn, def_insn, uid_luid))
4947 9816 : all_dominated = false;
4948 15599788 : if (closest_use != insn && closest_use != const0_rtx)
4949 : {
4950 13748747 : if (closest_use == NULL_RTX)
4951 : closest_use = insn;
4952 4643614 : else if (insn_dominated_by_p (closest_use, insn, uid_luid))
4953 : closest_use = insn;
4954 1352867 : else if (!insn_dominated_by_p (insn, closest_use, uid_luid))
4955 637356 : closest_use = const0_rtx;
4956 : }
4957 : }
4958 9168366 : if (!all_dominated)
4959 : {
4960 4933 : if (dump_file)
4961 0 : fprintf (dump_file, "Reg %d not all uses dominated by set\n",
4962 : regno);
4963 4933 : continue;
4964 : }
4965 9163433 : if (all_local)
4966 6534138 : bitmap_set_bit (local, regno);
4967 8527794 : if (closest_use == const0_rtx || closest_use == NULL
4968 17627994 : || next_nonnote_nondebug_insn (def_insn) == closest_use)
4969 : {
4970 5782737 : if (dump_file)
4971 99 : fprintf (dump_file, "Reg %d uninteresting%s\n", regno,
4972 99 : closest_use == const0_rtx || closest_use == NULL
4973 : ? " (no unique first use)" : "");
4974 5782737 : continue;
4975 : }
4976 :
4977 3380696 : bitmap_set_bit (interesting, regno);
4978 : /* If we get here, we know closest_use is a non-NULL insn
4979 : (as opposed to const_0_rtx). */
4980 3380696 : closest_uses[regno] = as_a <rtx_insn *> (closest_use);
4981 :
4982 3380696 : if (dump_file && (all_local || all_dominated))
4983 : {
4984 78 : fprintf (dump_file, "Reg %u:", regno);
4985 78 : if (all_local)
4986 14 : fprintf (dump_file, " local to bb %d", bb->index);
4987 78 : if (all_dominated)
4988 78 : fprintf (dump_file, " def dominates all uses");
4989 78 : if (closest_use != const0_rtx)
4990 78 : fprintf (dump_file, " has unique first use");
4991 78 : fputs ("\n", dump_file);
4992 : }
4993 : }
4994 : }
4995 :
4996 4437755 : EXECUTE_IF_SET_IN_BITMAP (interesting, 0, i, bi)
4997 : {
4998 3380696 : df_ref def = DF_REG_DEF_CHAIN (i);
4999 3380696 : rtx_insn *def_insn = DF_REF_INSN (def);
5000 3380696 : basic_block def_block = BLOCK_FOR_INSN (def_insn);
5001 3380696 : bitmap def_bb_local = bb_local + def_block->index;
5002 3380696 : bitmap def_bb_moveable = bb_moveable_reg_sets + def_block->index;
5003 3380696 : bitmap def_bb_transp = bb_transp_live + def_block->index;
5004 3380696 : bool local_to_bb_p = bitmap_bit_p (def_bb_local, i);
5005 3380696 : rtx_insn *use_insn = closest_uses[i];
5006 3380696 : df_ref use;
5007 3380696 : bool all_ok = true;
5008 3380696 : bool all_transp = true;
5009 :
5010 3380696 : if (!REG_P (DF_REF_REG (def)))
5011 49584 : continue;
5012 :
5013 3331112 : if (!local_to_bb_p)
5014 : {
5015 1267384 : if (dump_file)
5016 64 : fprintf (dump_file, "Reg %u not local to one basic block\n",
5017 : i);
5018 1267384 : continue;
5019 : }
5020 2063728 : if (reg_equiv_init (i) != NULL_RTX)
5021 : {
5022 48014 : if (dump_file)
5023 0 : fprintf (dump_file, "Ignoring reg %u with equiv init insn\n",
5024 : i);
5025 48014 : continue;
5026 : }
5027 2015714 : if (!rtx_moveable_p (&PATTERN (def_insn), OP_IN))
5028 : {
5029 1412770 : if (dump_file)
5030 14 : fprintf (dump_file, "Found def insn %d for %d to be not moveable\n",
5031 14 : INSN_UID (def_insn), i);
5032 1412770 : continue;
5033 : }
5034 602944 : if (dump_file)
5035 0 : fprintf (dump_file, "Examining insn %d, def for %d\n",
5036 0 : INSN_UID (def_insn), i);
5037 1357383 : FOR_EACH_INSN_USE (use, def_insn)
5038 : {
5039 813427 : unsigned regno = DF_REF_REGNO (use);
5040 813427 : if (bitmap_bit_p (unusable_as_input, regno))
5041 : {
5042 58988 : all_ok = false;
5043 58988 : if (dump_file)
5044 0 : fprintf (dump_file, " found unusable input reg %u.\n", regno);
5045 : break;
5046 : }
5047 754439 : if (!bitmap_bit_p (def_bb_transp, regno))
5048 : {
5049 703089 : if (bitmap_bit_p (def_bb_moveable, regno)
5050 703089 : && !control_flow_insn_p (use_insn))
5051 : {
5052 45 : if (modified_between_p (DF_REF_REG (use), def_insn, use_insn))
5053 : {
5054 0 : rtx_insn *x = NEXT_INSN (def_insn);
5055 0 : while (!modified_in_p (DF_REF_REG (use), x))
5056 : {
5057 0 : gcc_assert (x != use_insn);
5058 0 : x = NEXT_INSN (x);
5059 : }
5060 0 : if (dump_file)
5061 0 : fprintf (dump_file, " input reg %u modified but insn %d moveable\n",
5062 0 : regno, INSN_UID (x));
5063 0 : emit_insn_after (PATTERN (x), use_insn);
5064 0 : set_insn_deleted (x);
5065 : }
5066 : else
5067 : {
5068 45 : if (dump_file)
5069 0 : fprintf (dump_file, " input reg %u modified between def and use\n",
5070 : regno);
5071 : all_transp = false;
5072 : }
5073 : }
5074 : else
5075 : all_transp = false;
5076 : }
5077 : }
5078 0 : if (!all_ok)
5079 58988 : continue;
5080 543956 : if (!dbg_cnt (ira_move))
5081 : break;
5082 543956 : if (dump_file)
5083 0 : fprintf (dump_file, " all ok%s\n", all_transp ? " and transp" : "");
5084 :
5085 543956 : if (all_transp)
5086 : {
5087 14109 : rtx def_reg = DF_REF_REG (def);
5088 14109 : rtx newreg = ira_create_new_reg (def_reg);
5089 14109 : if (validate_change (def_insn, DF_REF_REAL_LOC (def), newreg, 0))
5090 : {
5091 14109 : unsigned nregno = REGNO (newreg);
5092 14109 : emit_insn_before (gen_move_insn (def_reg, newreg), use_insn);
5093 14109 : nregno -= max_regs;
5094 14109 : pseudo_replaced_reg[nregno] = def_reg;
5095 : }
5096 : }
5097 : }
5098 :
5099 12184562 : FOR_EACH_BB_FN (bb, cfun)
5100 : {
5101 11127503 : bitmap_clear (bb_local + bb->index);
5102 11127503 : bitmap_clear (bb_transp_live + bb->index);
5103 11127503 : bitmap_clear (bb_moveable_reg_sets + bb->index);
5104 : }
5105 1057059 : free (uid_luid);
5106 1057059 : free (closest_uses);
5107 1057059 : free (bb_local);
5108 1057059 : free (bb_transp_live);
5109 1057059 : free (bb_moveable_reg_sets);
5110 :
5111 1057059 : last_moveable_pseudo = max_reg_num ();
5112 :
5113 1057059 : fix_reg_equiv_init ();
5114 1057059 : expand_reg_info ();
5115 1057059 : regstat_free_n_sets_and_refs ();
5116 1057059 : regstat_free_ri ();
5117 1057059 : regstat_init_n_sets_and_refs ();
5118 1057059 : regstat_compute_ri ();
5119 1057059 : free_dominance_info (CDI_DOMINATORS);
5120 1057059 : }
5121 :
5122 : /* If SET pattern SET is an assignment from a hard register to a pseudo which
5123 : is live at CALL_DOM (if non-NULL, otherwise this check is omitted), return
5124 : the destination. Otherwise return NULL. */
5125 :
5126 : static rtx
5127 2140935 : interesting_dest_for_shprep_1 (rtx set, basic_block call_dom)
5128 : {
5129 2140935 : rtx src = SET_SRC (set);
5130 2140935 : rtx dest = SET_DEST (set);
5131 721646 : if (!REG_P (src) || !HARD_REGISTER_P (src)
5132 562358 : || !REG_P (dest) || HARD_REGISTER_P (dest)
5133 2682437 : || (call_dom && !bitmap_bit_p (df_get_live_in (call_dom), REGNO (dest))))
5134 1689071 : return NULL;
5135 : return dest;
5136 : }
5137 :
5138 : /* If insn is interesting for parameter range-splitting shrink-wrapping
5139 : preparation, i.e. it is a single set from a hard register to a pseudo, which
5140 : is live at CALL_DOM (if non-NULL, otherwise this check is omitted), or a
5141 : parallel statement with only one such statement, return the destination.
5142 : Otherwise return NULL. */
5143 :
5144 : static rtx
5145 4448548 : interesting_dest_for_shprep (rtx_insn *insn, basic_block call_dom)
5146 : {
5147 4448548 : if (!INSN_P (insn))
5148 : return NULL;
5149 3568871 : rtx pat = PATTERN (insn);
5150 3568871 : if (GET_CODE (pat) == SET)
5151 1902675 : return interesting_dest_for_shprep_1 (pat, call_dom);
5152 :
5153 1666196 : if (GET_CODE (pat) != PARALLEL)
5154 : return NULL;
5155 : rtx ret = NULL;
5156 624269 : for (int i = 0; i < XVECLEN (pat, 0); i++)
5157 : {
5158 421965 : rtx sub = XVECEXP (pat, 0, i);
5159 421965 : if (GET_CODE (sub) == USE || GET_CODE (sub) == CLOBBER)
5160 175629 : continue;
5161 246336 : if (GET_CODE (sub) != SET
5162 246336 : || side_effects_p (sub))
5163 8076 : return NULL;
5164 238260 : rtx dest = interesting_dest_for_shprep_1 (sub, call_dom);
5165 238260 : if (dest && ret)
5166 : return NULL;
5167 238260 : if (dest)
5168 413889 : ret = dest;
5169 : }
5170 : return ret;
5171 : }
5172 :
5173 : /* Split live ranges of pseudos that are loaded from hard registers in the
5174 : first BB in a BB that dominates all non-sibling call if such a BB can be
5175 : found and is not in a loop. Return true if the function has made any
5176 : changes. */
5177 :
5178 : static bool
5179 1057059 : split_live_ranges_for_shrink_wrap (void)
5180 : {
5181 1057059 : basic_block bb, call_dom = NULL;
5182 1057059 : basic_block first = single_succ (ENTRY_BLOCK_PTR_FOR_FN (cfun));
5183 1057059 : rtx_insn *insn, *last_interesting_insn = NULL;
5184 1057059 : auto_bitmap need_new, reachable;
5185 1057059 : vec<basic_block> queue;
5186 :
5187 1057059 : if (!SHRINK_WRAPPING_ENABLED)
5188 246 : return false;
5189 :
5190 1056813 : queue.create (n_basic_blocks_for_fn (cfun));
5191 :
5192 7538424 : FOR_EACH_BB_FN (bb, cfun)
5193 76324610 : FOR_BB_INSNS (bb, insn)
5194 71233485 : if (CALL_P (insn) && !SIBLING_CALL_P (insn))
5195 : {
5196 1804536 : if (bb == first)
5197 : {
5198 414050 : queue.release ();
5199 414050 : return false;
5200 : }
5201 :
5202 1390486 : bitmap_set_bit (need_new, bb->index);
5203 1390486 : bitmap_set_bit (reachable, bb->index);
5204 1390486 : queue.quick_push (bb);
5205 1390486 : break;
5206 : }
5207 :
5208 642763 : if (queue.is_empty ())
5209 : {
5210 385188 : queue.release ();
5211 385188 : return false;
5212 : }
5213 :
5214 4216495 : while (!queue.is_empty ())
5215 : {
5216 3958920 : edge e;
5217 3958920 : edge_iterator ei;
5218 :
5219 3958920 : bb = queue.pop ();
5220 9489503 : FOR_EACH_EDGE (e, ei, bb->succs)
5221 5530583 : if (e->dest != EXIT_BLOCK_PTR_FOR_FN (cfun)
5222 5530583 : && bitmap_set_bit (reachable, e->dest->index))
5223 2568434 : queue.quick_push (e->dest);
5224 : }
5225 257575 : queue.release ();
5226 :
5227 4006614 : FOR_BB_INSNS (first, insn)
5228 : {
5229 3749459 : rtx dest = interesting_dest_for_shprep (insn, NULL);
5230 3749459 : if (!dest)
5231 3353043 : continue;
5232 :
5233 396416 : if (DF_REG_DEF_COUNT (REGNO (dest)) > 1)
5234 : return false;
5235 :
5236 395996 : for (df_ref use = DF_REG_USE_CHAIN (REGNO(dest));
5237 2858334 : use;
5238 2462338 : use = DF_REF_NEXT_REG (use))
5239 : {
5240 2462338 : int ubbi = DF_REF_BB (use)->index;
5241 :
5242 : /* Only non debug insns should be taken into account. */
5243 2462338 : if (NONDEBUG_INSN_P (DF_REF_INSN (use))
5244 2462338 : && bitmap_bit_p (reachable, ubbi))
5245 1111398 : bitmap_set_bit (need_new, ubbi);
5246 : }
5247 : last_interesting_insn = insn;
5248 : }
5249 :
5250 257155 : if (!last_interesting_insn)
5251 : return false;
5252 :
5253 184751 : call_dom = nearest_common_dominator_for_set (CDI_DOMINATORS, need_new);
5254 184751 : if (call_dom == first)
5255 : return false;
5256 :
5257 96326 : loop_optimizer_init (AVOID_CFG_MODIFICATIONS);
5258 215974 : while (bb_loop_depth (call_dom) > 0)
5259 23322 : call_dom = get_immediate_dominator (CDI_DOMINATORS, call_dom);
5260 96326 : loop_optimizer_finalize ();
5261 :
5262 96326 : if (call_dom == first)
5263 : return false;
5264 :
5265 84274 : calculate_dominance_info (CDI_POST_DOMINATORS);
5266 84274 : if (dominated_by_p (CDI_POST_DOMINATORS, first, call_dom))
5267 : {
5268 7797 : free_dominance_info (CDI_POST_DOMINATORS);
5269 7797 : return false;
5270 : }
5271 76477 : free_dominance_info (CDI_POST_DOMINATORS);
5272 :
5273 76477 : if (dump_file)
5274 2 : fprintf (dump_file, "Will split live ranges of parameters at BB %i\n",
5275 : call_dom->index);
5276 :
5277 76477 : bool ret = false;
5278 751248 : FOR_BB_INSNS (first, insn)
5279 : {
5280 699089 : rtx dest = interesting_dest_for_shprep (insn, call_dom);
5281 699089 : if (!dest || dest == pic_offset_table_rtx)
5282 643641 : continue;
5283 :
5284 55448 : bool need_newreg = false;
5285 55448 : df_ref use, next;
5286 68920 : for (use = DF_REG_USE_CHAIN (REGNO (dest)); use; use = next)
5287 : {
5288 68850 : rtx_insn *uin = DF_REF_INSN (use);
5289 68850 : next = DF_REF_NEXT_REG (use);
5290 :
5291 68850 : if (DEBUG_INSN_P (uin))
5292 516 : continue;
5293 :
5294 68334 : basic_block ubb = BLOCK_FOR_INSN (uin);
5295 68334 : if (ubb == call_dom
5296 68334 : || dominated_by_p (CDI_DOMINATORS, ubb, call_dom))
5297 : {
5298 : need_newreg = true;
5299 : break;
5300 : }
5301 : }
5302 :
5303 55448 : if (need_newreg)
5304 : {
5305 55378 : rtx newreg = ira_create_new_reg (dest);
5306 :
5307 437129 : for (use = DF_REG_USE_CHAIN (REGNO (dest)); use; use = next)
5308 : {
5309 381751 : rtx_insn *uin = DF_REF_INSN (use);
5310 381751 : next = DF_REF_NEXT_REG (use);
5311 :
5312 381751 : basic_block ubb = BLOCK_FOR_INSN (uin);
5313 381751 : if (ubb == call_dom
5314 381751 : || dominated_by_p (CDI_DOMINATORS, ubb, call_dom))
5315 279704 : validate_change (uin, DF_REF_REAL_LOC (use), newreg, true);
5316 : }
5317 :
5318 55378 : rtx_insn *new_move = gen_move_insn (newreg, dest);
5319 55378 : emit_insn_after (new_move, bb_note (call_dom));
5320 55378 : if (dump_file)
5321 : {
5322 2 : fprintf (dump_file, "Split live-range of register ");
5323 2 : print_rtl_single (dump_file, dest);
5324 : }
5325 : ret = true;
5326 : }
5327 :
5328 55448 : if (insn == last_interesting_insn)
5329 : break;
5330 : }
5331 76477 : apply_change_group ();
5332 76477 : return ret;
5333 1057059 : }
5334 :
5335 : /* Perform the second half of the transformation started in
5336 : find_moveable_pseudos. We look for instances where the newly introduced
5337 : pseudo remains unallocated, and remove it by moving the definition to
5338 : just before its use, replacing the move instruction generated by
5339 : find_moveable_pseudos. */
5340 : static void
5341 1057059 : move_unallocated_pseudos (void)
5342 : {
5343 1057059 : int i;
5344 1071168 : for (i = first_moveable_pseudo; i < last_moveable_pseudo; i++)
5345 14109 : if (reg_renumber[i] < 0)
5346 : {
5347 3599 : int idx = i - first_moveable_pseudo;
5348 3599 : rtx other_reg = pseudo_replaced_reg[idx];
5349 : /* The iterating range [first_moveable_pseudo, last_moveable_pseudo)
5350 : covers every new pseudo created in find_moveable_pseudos,
5351 : regardless of the validation with it is successful or not.
5352 : So we need to skip the pseudos which were used in those failed
5353 : validations to avoid unexpected DF info and consequent ICE.
5354 : We only set pseudo_replaced_reg[] when the validation is successful
5355 : in find_moveable_pseudos, it's enough to check it here. */
5356 3599 : if (!other_reg)
5357 0 : continue;
5358 3599 : rtx_insn *def_insn = DF_REF_INSN (DF_REG_DEF_CHAIN (i));
5359 : /* The use must follow all definitions of OTHER_REG, so we can
5360 : insert the new definition immediately after any of them. */
5361 3599 : df_ref other_def = DF_REG_DEF_CHAIN (REGNO (other_reg));
5362 3599 : rtx_insn *move_insn = DF_REF_INSN (other_def);
5363 3599 : rtx_insn *newinsn = emit_insn_after (PATTERN (def_insn), move_insn);
5364 3599 : rtx set;
5365 3599 : int success;
5366 :
5367 3599 : if (dump_file)
5368 0 : fprintf (dump_file, "moving def of %d (insn %d now) ",
5369 0 : REGNO (other_reg), INSN_UID (def_insn));
5370 :
5371 3599 : delete_insn (move_insn);
5372 7198 : while ((other_def = DF_REG_DEF_CHAIN (REGNO (other_reg))))
5373 0 : delete_insn (DF_REF_INSN (other_def));
5374 3599 : delete_insn (def_insn);
5375 :
5376 3599 : set = single_set (newinsn);
5377 3599 : success = validate_change (newinsn, &SET_DEST (set), other_reg, 0);
5378 3599 : gcc_assert (success);
5379 3599 : if (dump_file)
5380 0 : fprintf (dump_file, " %d) rather than keep unallocated replacement %d\n",
5381 0 : INSN_UID (newinsn), i);
5382 3599 : SET_REG_N_REFS (i, 0);
5383 : }
5384 :
5385 1057059 : first_moveable_pseudo = last_moveable_pseudo = 0;
5386 1057059 : }
5387 :
5388 :
5389 :
5390 : /* Code dealing with scratches (changing them onto
5391 : pseudos and restoring them from the pseudos).
5392 :
5393 : We change scratches into pseudos at the beginning of IRA to
5394 : simplify dealing with them (conflicts, hard register assignments).
5395 :
5396 : If the pseudo denoting scratch was spilled it means that we do not
5397 : need a hard register for it. Such pseudos are transformed back to
5398 : scratches at the end of LRA. */
5399 :
5400 : /* Description of location of a former scratch operand. */
5401 : struct sloc
5402 : {
5403 : rtx_insn *insn; /* Insn where the scratch was. */
5404 : int nop; /* Number of the operand which was a scratch. */
5405 : unsigned regno; /* regno generated instead of scratch */
5406 : int icode; /* Original icode from which scratch was removed. */
5407 : };
5408 :
5409 : typedef struct sloc *sloc_t;
5410 :
5411 : /* Locations of the former scratches. */
5412 : static vec<sloc_t> scratches;
5413 :
5414 : /* Bitmap of scratch regnos. */
5415 : static bitmap_head scratch_bitmap;
5416 :
5417 : /* Bitmap of scratch operands. */
5418 : static bitmap_head scratch_operand_bitmap;
5419 :
5420 : /* Return true if pseudo REGNO is made of SCRATCH. */
5421 : bool
5422 376655633 : ira_former_scratch_p (int regno)
5423 : {
5424 376655633 : return bitmap_bit_p (&scratch_bitmap, regno);
5425 : }
5426 :
5427 : /* Return true if the operand NOP of INSN is a former scratch. */
5428 : bool
5429 0 : ira_former_scratch_operand_p (rtx_insn *insn, int nop)
5430 : {
5431 0 : return bitmap_bit_p (&scratch_operand_bitmap,
5432 0 : INSN_UID (insn) * MAX_RECOG_OPERANDS + nop) != 0;
5433 : }
5434 :
5435 : /* Register operand NOP in INSN as a former scratch. It will be
5436 : changed to scratch back, if it is necessary, at the LRA end. */
5437 : void
5438 93474 : ira_register_new_scratch_op (rtx_insn *insn, int nop, int icode)
5439 : {
5440 93474 : rtx op = *recog_data.operand_loc[nop];
5441 93474 : sloc_t loc = XNEW (struct sloc);
5442 93474 : ira_assert (REG_P (op));
5443 93474 : loc->insn = insn;
5444 93474 : loc->nop = nop;
5445 93474 : loc->regno = REGNO (op);
5446 93474 : loc->icode = icode;
5447 93474 : scratches.safe_push (loc);
5448 93474 : bitmap_set_bit (&scratch_bitmap, REGNO (op));
5449 186948 : bitmap_set_bit (&scratch_operand_bitmap,
5450 93474 : INSN_UID (insn) * MAX_RECOG_OPERANDS + nop);
5451 93474 : add_reg_note (insn, REG_UNUSED, op);
5452 93474 : }
5453 :
5454 : /* Return true if string STR contains constraint 'X'. */
5455 : static bool
5456 93474 : contains_X_constraint_p (const char *str)
5457 : {
5458 93474 : int c;
5459 :
5460 385585 : while ((c = *str))
5461 : {
5462 301139 : str += CONSTRAINT_LEN (c, str);
5463 301139 : if (c == 'X') return true;
5464 : }
5465 : return false;
5466 : }
5467 :
5468 : /* Change INSN's scratches into pseudos and save their location.
5469 : Return true if we changed any scratch. */
5470 : bool
5471 282868068 : ira_remove_insn_scratches (rtx_insn *insn, bool all_p, FILE *dump_file,
5472 : rtx (*get_reg) (rtx original))
5473 : {
5474 282868068 : int i;
5475 282868068 : bool insn_changed_p;
5476 282868068 : rtx reg, *loc;
5477 :
5478 282868068 : extract_insn (insn);
5479 282868068 : insn_changed_p = false;
5480 948312874 : for (i = 0; i < recog_data.n_operands; i++)
5481 : {
5482 382576738 : loc = recog_data.operand_loc[i];
5483 382576738 : if (GET_CODE (*loc) == SCRATCH && GET_MODE (*loc) != VOIDmode)
5484 : {
5485 102502 : if (! all_p && contains_X_constraint_p (recog_data.constraints[i]))
5486 9028 : continue;
5487 93474 : insn_changed_p = true;
5488 93474 : *loc = reg = get_reg (*loc);
5489 93474 : ira_register_new_scratch_op (insn, i, INSN_CODE (insn));
5490 93474 : if (dump_file != NULL)
5491 0 : fprintf (dump_file,
5492 : "Removing SCRATCH to p%u in insn #%u (nop %d)\n",
5493 0 : REGNO (reg), INSN_UID (insn), i);
5494 : }
5495 : }
5496 282868068 : return insn_changed_p;
5497 : }
5498 :
5499 : /* Return new register of the same mode as ORIGINAL. Used in
5500 : remove_scratches. */
5501 : static rtx
5502 84446 : get_scratch_reg (rtx original)
5503 : {
5504 84446 : return gen_reg_rtx (GET_MODE (original));
5505 : }
5506 :
5507 : /* Change scratches into pseudos and save their location. Return true
5508 : if we changed any scratch. */
5509 : static bool
5510 1504950 : remove_scratches (void)
5511 : {
5512 1504950 : bool change_p = false;
5513 1504950 : basic_block bb;
5514 1504950 : rtx_insn *insn;
5515 :
5516 1504950 : scratches.create (get_max_uid ());
5517 1504950 : bitmap_initialize (&scratch_bitmap, ®_obstack);
5518 1504950 : bitmap_initialize (&scratch_operand_bitmap, ®_obstack);
5519 16134410 : FOR_EACH_BB_FN (bb, cfun)
5520 178034286 : FOR_BB_INSNS (bb, insn)
5521 163404826 : if (INSN_P (insn)
5522 163404826 : && ira_remove_insn_scratches (insn, false, ira_dump_file, get_scratch_reg))
5523 : {
5524 : /* Because we might use DF, we need to keep DF info up to date. */
5525 83286 : df_insn_rescan (insn);
5526 83286 : change_p = true;
5527 : }
5528 1504950 : return change_p;
5529 : }
5530 :
5531 : /* Changes pseudos created by function remove_scratches onto scratches. */
5532 : void
5533 1504950 : ira_restore_scratches (FILE *dump_file)
5534 : {
5535 1504950 : int regno, n;
5536 1504950 : unsigned i;
5537 1504950 : rtx *op_loc;
5538 1504950 : sloc_t loc;
5539 :
5540 1598424 : for (i = 0; scratches.iterate (i, &loc); i++)
5541 : {
5542 : /* Ignore already deleted insns. */
5543 93474 : if (NOTE_P (loc->insn)
5544 0 : && NOTE_KIND (loc->insn) == NOTE_INSN_DELETED)
5545 0 : continue;
5546 93474 : extract_insn (loc->insn);
5547 93474 : if (loc->icode != INSN_CODE (loc->insn))
5548 : {
5549 : /* The icode doesn't match, which means the insn has been modified
5550 : (e.g. register elimination). The scratch cannot be restored. */
5551 0 : continue;
5552 : }
5553 93474 : op_loc = recog_data.operand_loc[loc->nop];
5554 93474 : if (REG_P (*op_loc)
5555 93474 : && ((regno = REGNO (*op_loc)) >= FIRST_PSEUDO_REGISTER)
5556 186948 : && reg_renumber[regno] < 0)
5557 : {
5558 : /* It should be only case when scratch register with chosen
5559 : constraint 'X' did not get memory or hard register. */
5560 5334 : ira_assert (ira_former_scratch_p (regno));
5561 5334 : *op_loc = gen_rtx_SCRATCH (GET_MODE (*op_loc));
5562 5334 : for (n = 0; n < recog_data.n_dups; n++)
5563 0 : *recog_data.dup_loc[n]
5564 0 : = *recog_data.operand_loc[(int) recog_data.dup_num[n]];
5565 5334 : if (dump_file != NULL)
5566 0 : fprintf (dump_file, "Restoring SCRATCH in insn #%u(nop %d)\n",
5567 0 : INSN_UID (loc->insn), loc->nop);
5568 : }
5569 : }
5570 1598424 : for (i = 0; scratches.iterate (i, &loc); i++)
5571 93474 : free (loc);
5572 1504950 : scratches.release ();
5573 1504950 : bitmap_clear (&scratch_bitmap);
5574 1504950 : bitmap_clear (&scratch_operand_bitmap);
5575 1504950 : }
5576 :
5577 :
5578 :
5579 : /* If the backend knows where to allocate pseudos for hard
5580 : register initial values, register these allocations now. */
5581 : static void
5582 1504950 : allocate_initial_values (void)
5583 : {
5584 1504950 : if (targetm.allocate_initial_value)
5585 : {
5586 : rtx hreg, preg, x;
5587 : int i, regno;
5588 :
5589 0 : for (i = 0; HARD_REGISTER_NUM_P (i); i++)
5590 : {
5591 0 : if (! initial_value_entry (i, &hreg, &preg))
5592 : break;
5593 :
5594 0 : x = targetm.allocate_initial_value (hreg);
5595 0 : regno = REGNO (preg);
5596 0 : if (x && REG_N_SETS (regno) <= 1)
5597 : {
5598 0 : if (MEM_P (x))
5599 0 : reg_equiv_memory_loc (regno) = x;
5600 : else
5601 : {
5602 0 : basic_block bb;
5603 0 : int new_regno;
5604 :
5605 0 : gcc_assert (REG_P (x));
5606 0 : new_regno = REGNO (x);
5607 0 : reg_renumber[regno] = new_regno;
5608 : /* Poke the regno right into regno_reg_rtx so that even
5609 : fixed regs are accepted. */
5610 0 : SET_REGNO (preg, new_regno);
5611 : /* Update global register liveness information. */
5612 0 : FOR_EACH_BB_FN (bb, cfun)
5613 : {
5614 0 : if (REGNO_REG_SET_P (df_get_live_in (bb), regno))
5615 0 : SET_REGNO_REG_SET (df_get_live_in (bb), new_regno);
5616 0 : if (REGNO_REG_SET_P (df_get_live_out (bb), regno))
5617 0 : SET_REGNO_REG_SET (df_get_live_out (bb), new_regno);
5618 : }
5619 : }
5620 : }
5621 : }
5622 :
5623 0 : gcc_checking_assert (! initial_value_entry (FIRST_PSEUDO_REGISTER,
5624 : &hreg, &preg));
5625 : }
5626 1504950 : }
5627 :
5628 :
5629 :
5630 :
5631 : /* True when we use LRA instead of reload pass for the current
5632 : function. */
5633 : bool ira_use_lra_p;
5634 :
5635 : /* True if we have allocno conflicts. It is false for non-optimized
5636 : mode or when the conflict table is too big. */
5637 : bool ira_conflicts_p;
5638 :
5639 : /* Saved between IRA and reload. */
5640 : static int saved_flag_ira_share_spill_slots;
5641 :
5642 : /* Set to true while in IRA. */
5643 : bool ira_in_progress = false;
5644 :
5645 : /* Set up array ira_hard_regno_nrefs. */
5646 : static void
5647 1504950 : setup_hard_regno_nrefs (void)
5648 : {
5649 1504950 : int i;
5650 :
5651 139960350 : for (i = 0; i < FIRST_PSEUDO_REGISTER; i++)
5652 : {
5653 138455400 : ira_hard_regno_nrefs[i] = 0;
5654 138455400 : for (df_ref use = DF_REG_USE_CHAIN (i);
5655 256741696 : use != NULL;
5656 118286296 : use = DF_REF_NEXT_REG (use))
5657 118286296 : if (DF_REF_CLASS (use) != DF_REF_ARTIFICIAL
5658 54424306 : && !(DF_REF_INSN_INFO (use) && DEBUG_INSN_P (DF_REF_INSN (use))))
5659 50955624 : ira_hard_regno_nrefs[i]++;
5660 138455400 : for (df_ref def = DF_REG_DEF_CHAIN (i);
5661 700841608 : def != NULL;
5662 562386208 : def = DF_REF_NEXT_REG (def))
5663 562386208 : if (DF_REF_CLASS (def) != DF_REF_ARTIFICIAL
5664 534405264 : && !(DF_REF_INSN_INFO (def) && DEBUG_INSN_P (DF_REF_INSN (def))))
5665 534405264 : ira_hard_regno_nrefs[i]++;
5666 : }
5667 1504950 : }
5668 :
5669 : /* This is the main entry of IRA. */
5670 : static void
5671 1504950 : ira (FILE *f)
5672 : {
5673 1504950 : bool loops_p;
5674 1504950 : int ira_max_point_before_emit;
5675 1504950 : bool saved_flag_caller_saves = flag_caller_saves;
5676 1504950 : enum ira_region saved_flag_ira_region = flag_ira_region;
5677 1504950 : basic_block bb;
5678 1504950 : edge_iterator ei;
5679 1504950 : edge e;
5680 1504950 : bool output_jump_reload_p = false;
5681 :
5682 1504950 : setup_hard_regno_nrefs ();
5683 1504950 : lra_reset_dependent_filters ();
5684 1504950 : if (ira_use_lra_p)
5685 : {
5686 : /* First put potential jump output reloads on the output edges
5687 : as USE which will be removed at the end of LRA. The major
5688 : goal is actually to create BBs for critical edges for LRA and
5689 : populate them later by live info. In LRA it will be
5690 : difficult to do this. */
5691 16134401 : FOR_EACH_BB_FN (bb, cfun)
5692 : {
5693 14629451 : rtx_insn *end = BB_END (bb);
5694 14629451 : if (!JUMP_P (end))
5695 5757694 : continue;
5696 8871757 : extract_insn (end);
5697 24038511 : for (int i = 0; i < recog_data.n_operands; i++)
5698 15166906 : if (recog_data.operand_type[i] != OP_IN)
5699 : {
5700 152 : bool skip_p = false;
5701 511 : FOR_EACH_EDGE (e, ei, bb->succs)
5702 729 : if (EDGE_CRITICAL_P (e)
5703 11 : && e->dest != EXIT_BLOCK_PTR_FOR_FN (cfun)
5704 370 : && (e->flags & EDGE_ABNORMAL))
5705 : {
5706 : skip_p = true;
5707 : break;
5708 : }
5709 152 : if (skip_p)
5710 : break;
5711 152 : output_jump_reload_p = true;
5712 511 : FOR_EACH_EDGE (e, ei, bb->succs)
5713 729 : if (EDGE_CRITICAL_P (e)
5714 370 : && e->dest != EXIT_BLOCK_PTR_FOR_FN (cfun))
5715 : {
5716 11 : start_sequence ();
5717 : /* We need to put some no-op insn here. We can
5718 : not put a note as commit_edges insertion will
5719 : fail. */
5720 11 : emit_insn (gen_rtx_USE (VOIDmode, const1_rtx));
5721 11 : rtx_insn *insns = end_sequence ();
5722 11 : insert_insn_on_edge (insns, e);
5723 : }
5724 : break;
5725 : }
5726 : }
5727 1504950 : if (output_jump_reload_p)
5728 146 : commit_edge_insertions ();
5729 : }
5730 :
5731 1504950 : if (flag_ira_verbose < 10)
5732 : {
5733 1504950 : internal_flag_ira_verbose = flag_ira_verbose;
5734 1504950 : ira_dump_file = f;
5735 : }
5736 : else
5737 : {
5738 0 : internal_flag_ira_verbose = flag_ira_verbose - 10;
5739 0 : ira_dump_file = stderr;
5740 : }
5741 :
5742 1504950 : clear_bb_flags ();
5743 :
5744 : /* Determine if the current function is a leaf before running IRA
5745 : since this can impact optimizations done by the prologue and
5746 : epilogue thus changing register elimination offsets.
5747 : Other target callbacks may use crtl->is_leaf too, including
5748 : SHRINK_WRAPPING_ENABLED, so initialize as early as possible. */
5749 1504950 : crtl->is_leaf = leaf_function_p ();
5750 :
5751 : /* Perform target specific PIC register initialization. */
5752 1504950 : targetm.init_pic_reg ();
5753 :
5754 1504950 : ira_conflicts_p = optimize > 0;
5755 :
5756 : /* Determine the number of pseudos actually requiring coloring. */
5757 1504950 : unsigned int num_used_regs = 0;
5758 68260575 : for (unsigned int i = FIRST_PSEUDO_REGISTER; i < DF_REG_SIZE (df); i++)
5759 66755625 : if (DF_REG_DEF_COUNT (i) || DF_REG_USE_COUNT (i))
5760 30411576 : num_used_regs++;
5761 :
5762 : /* If there are too many pseudos and/or basic blocks (e.g. 10K pseudos and
5763 : 10K blocks or 100K pseudos and 1K blocks) or we have too many function
5764 : insns, we will use simplified and faster algorithms in LRA. */
5765 1504950 : lra_simple_p
5766 1504950 : = (ira_use_lra_p
5767 1504950 : && (num_used_regs >= (1U << 26) / last_basic_block_for_fn (cfun)
5768 : /* max uid is a good evaluation of the number of insns as most
5769 : optimizations are done on tree-SSA level. */
5770 1504945 : || ((uint64_t) get_max_uid ()
5771 1504945 : > (uint64_t) param_ira_simple_lra_insn_threshold * 1000)));
5772 :
5773 1504950 : if (lra_simple_p)
5774 : {
5775 : /* It permits to skip live range splitting in LRA. */
5776 5 : flag_caller_saves = false;
5777 : /* There is no sense to do regional allocation when we use
5778 : simplified LRA. */
5779 5 : flag_ira_region = IRA_REGION_ONE;
5780 5 : ira_conflicts_p = false;
5781 : }
5782 :
5783 : #ifndef IRA_NO_OBSTACK
5784 : gcc_obstack_init (&ira_obstack);
5785 : #endif
5786 1504950 : bitmap_obstack_initialize (&ira_bitmap_obstack);
5787 :
5788 : /* LRA uses its own infrastructure to handle caller save registers. */
5789 1504950 : if (flag_caller_saves && !ira_use_lra_p)
5790 0 : init_caller_save ();
5791 :
5792 1504950 : setup_prohibited_mode_move_regs ();
5793 1504950 : decrease_live_ranges_number ();
5794 1504950 : df_note_add_problem ();
5795 :
5796 : /* DF_LIVE can't be used in the register allocator, too many other
5797 : parts of the compiler depend on using the "classic" liveness
5798 : interpretation of the DF_LR problem. See PR38711.
5799 : Remove the problem, so that we don't spend time updating it in
5800 : any of the df_analyze() calls during IRA/LRA. */
5801 1504950 : if (optimize > 1)
5802 976481 : df_remove_problem (df_live);
5803 1504950 : gcc_checking_assert (df_live == NULL);
5804 :
5805 1504950 : if (flag_checking)
5806 1504930 : df->changeable_flags |= DF_VERIFY_SCHEDULED;
5807 :
5808 1504950 : df_analyze ();
5809 :
5810 1504950 : init_reg_equiv ();
5811 1504950 : if (ira_conflicts_p)
5812 : {
5813 1057059 : calculate_dominance_info (CDI_DOMINATORS);
5814 :
5815 1057059 : if (split_live_ranges_for_shrink_wrap ())
5816 27711 : df_analyze ();
5817 :
5818 1057059 : free_dominance_info (CDI_DOMINATORS);
5819 : }
5820 :
5821 1504950 : df_clear_flags (DF_NO_INSN_RESCAN);
5822 :
5823 1504950 : indirect_jump_optimize ();
5824 1504950 : if (delete_trivially_dead_insns (get_insns (), max_reg_num ()))
5825 5539 : df_analyze ();
5826 :
5827 1504950 : regstat_init_n_sets_and_refs ();
5828 1504950 : regstat_compute_ri ();
5829 :
5830 : /* If we are not optimizing, then this is the only place before
5831 : register allocation where dataflow is done. And that is needed
5832 : to generate these warnings. */
5833 1504950 : if (warn_clobbered)
5834 135509 : generate_setjmp_warnings ();
5835 :
5836 : /* update_equiv_regs can use reg classes of pseudos and they are set up in
5837 : register pressure sensitive scheduling and loop invariant motion and in
5838 : live range shrinking. This info can become obsolete if we add new pseudos
5839 : since the last set up. Recalculate it again if the new pseudos were
5840 : added. */
5841 1504950 : if (resize_reg_info () && (flag_sched_pressure || flag_live_range_shrinkage
5842 1504855 : || flag_ira_loop_pressure))
5843 43 : ira_set_pseudo_classes (true, ira_dump_file);
5844 :
5845 1504950 : init_alias_analysis ();
5846 1504950 : loop_optimizer_init (AVOID_CFG_MODIFICATIONS);
5847 1504950 : reg_equiv = XCNEWVEC (struct equivalence, max_reg_num ());
5848 1504950 : update_equiv_regs_prescan ();
5849 1504950 : update_equiv_regs ();
5850 :
5851 : /* Don't move insns if live range shrinkage or register
5852 : pressure-sensitive scheduling were done because it will not
5853 : improve allocation but likely worsen insn scheduling. */
5854 1504950 : if (optimize
5855 1057059 : && !flag_live_range_shrinkage
5856 1057028 : && !(flag_sched_pressure && flag_schedule_insns))
5857 1057006 : combine_and_move_insns ();
5858 :
5859 : /* Gather additional equivalences with memory. */
5860 1504950 : if (optimize && flag_expensive_optimizations)
5861 976443 : add_store_equivs ();
5862 :
5863 1504950 : loop_optimizer_finalize ();
5864 1504950 : free_dominance_info (CDI_DOMINATORS);
5865 1504950 : end_alias_analysis ();
5866 1504950 : free (reg_equiv);
5867 :
5868 : /* Once max_regno changes, we need to free and re-init/re-compute
5869 : some data structures like regstat_n_sets_and_refs and reg_info_p. */
5870 1579691 : auto regstat_recompute_for_max_regno = []() {
5871 74741 : regstat_free_n_sets_and_refs ();
5872 74741 : regstat_free_ri ();
5873 74741 : regstat_init_n_sets_and_refs ();
5874 74741 : regstat_compute_ri ();
5875 74741 : resize_reg_info ();
5876 74741 : };
5877 :
5878 1504950 : int max_regno_before_rm = max_reg_num ();
5879 1504950 : if (ira_use_lra_p && remove_scratches ())
5880 : {
5881 39646 : ira_expand_reg_equiv ();
5882 : /* For now remove_scatches is supposed to create pseudos when it
5883 : succeeds, assert this happens all the time. Once it doesn't
5884 : hold, we should guard the regstat recompute for the case
5885 : max_regno changes. */
5886 39646 : gcc_assert (max_regno_before_rm != max_reg_num ());
5887 39646 : regstat_recompute_for_max_regno ();
5888 : }
5889 :
5890 1504950 : setup_reg_equiv ();
5891 1504950 : grow_reg_equivs ();
5892 1504950 : setup_reg_equiv_init ();
5893 :
5894 1504950 : allocated_reg_info_size = max_reg_num ();
5895 :
5896 : /* It is not worth to do such improvement when we use a simple
5897 : allocation because of -O0 usage or because the function is too
5898 : big. */
5899 1504950 : if (ira_conflicts_p)
5900 1057059 : find_moveable_pseudos ();
5901 :
5902 1504950 : max_regno_before_ira = max_reg_num ();
5903 1504950 : ira_setup_eliminable_regset ();
5904 :
5905 1504950 : ira_overall_cost = ira_reg_cost = ira_mem_cost = 0;
5906 1504950 : ira_load_cost = ira_store_cost = ira_shuffle_cost = 0;
5907 1504950 : ira_move_loops_num = ira_additional_jumps_num = 0;
5908 :
5909 1504950 : ira_assert (current_loops == NULL);
5910 1504950 : if (flag_ira_region == IRA_REGION_ALL || flag_ira_region == IRA_REGION_MIXED)
5911 1010795 : loop_optimizer_init (AVOID_CFG_MODIFICATIONS | LOOPS_HAVE_RECORDED_EXITS);
5912 :
5913 1504950 : if (internal_flag_ira_verbose > 0 && ira_dump_file != NULL)
5914 95 : fprintf (ira_dump_file, "Building IRA IR\n");
5915 1504950 : loops_p = ira_build ();
5916 :
5917 1504950 : ira_assert (ira_conflicts_p || !loops_p);
5918 :
5919 1504950 : saved_flag_ira_share_spill_slots = flag_ira_share_spill_slots;
5920 1504950 : if (too_high_register_pressure_p () || cfun->calls_setjmp)
5921 : /* It is just wasting compiler's time to pack spilled pseudos into
5922 : stack slots in this case -- prohibit it. We also do this if
5923 : there is setjmp call because a variable not modified between
5924 : setjmp and longjmp the compiler is required to preserve its
5925 : value and sharing slots does not guarantee it. */
5926 1365 : flag_ira_share_spill_slots = false;
5927 :
5928 1504950 : ira_color ();
5929 :
5930 1504950 : ira_max_point_before_emit = ira_max_point;
5931 :
5932 1504950 : ira_initiate_emit_data ();
5933 :
5934 1504950 : ira_emit (loops_p);
5935 :
5936 1504950 : max_regno = max_reg_num ();
5937 1504950 : if (ira_conflicts_p)
5938 : {
5939 1057059 : if (! loops_p)
5940 : {
5941 1021548 : if (! ira_use_lra_p)
5942 0 : ira_initiate_assign ();
5943 : }
5944 : else
5945 : {
5946 35511 : expand_reg_info ();
5947 :
5948 35511 : if (ira_use_lra_p)
5949 : {
5950 35511 : ira_allocno_t a;
5951 35511 : ira_allocno_iterator ai;
5952 :
5953 11623828 : FOR_EACH_ALLOCNO (a, ai)
5954 : {
5955 11552806 : int old_regno = ALLOCNO_REGNO (a);
5956 11552806 : int new_regno = REGNO (ALLOCNO_EMIT_DATA (a)->reg);
5957 :
5958 11552806 : ALLOCNO_REGNO (a) = new_regno;
5959 :
5960 11552806 : if (old_regno != new_regno)
5961 1272237 : setup_reg_classes (new_regno, reg_preferred_class (old_regno),
5962 : reg_alternate_class (old_regno),
5963 : reg_allocno_class (old_regno));
5964 : }
5965 : }
5966 : else
5967 : {
5968 0 : if (internal_flag_ira_verbose > 0 && ira_dump_file != NULL)
5969 0 : fprintf (ira_dump_file, "Flattening IR\n");
5970 0 : ira_flattening (max_regno_before_ira, ira_max_point_before_emit);
5971 : }
5972 : /* New insns were generated: add notes and recalculate live
5973 : info. */
5974 35511 : df_analyze ();
5975 :
5976 : /* ??? Rebuild the loop tree, but why? Does the loop tree
5977 : change if new insns were generated? Can that be handled
5978 : by updating the loop tree incrementally? */
5979 35511 : loop_optimizer_finalize ();
5980 35511 : free_dominance_info (CDI_DOMINATORS);
5981 35511 : loop_optimizer_init (AVOID_CFG_MODIFICATIONS
5982 : | LOOPS_HAVE_RECORDED_EXITS);
5983 :
5984 35511 : if (! ira_use_lra_p)
5985 : {
5986 0 : setup_allocno_assignment_flags ();
5987 0 : ira_initiate_assign ();
5988 0 : ira_reassign_conflict_allocnos (max_regno);
5989 : }
5990 : }
5991 : }
5992 :
5993 1504950 : ira_finish_emit_data ();
5994 :
5995 1504950 : setup_reg_renumber ();
5996 :
5997 1504950 : calculate_allocation_cost ();
5998 :
5999 : #ifdef ENABLE_IRA_CHECKING
6000 1504950 : if (ira_conflicts_p && ! ira_use_lra_p)
6001 : /* Opposite to reload pass, LRA does not use any conflict info
6002 : from IRA. We don't rebuild conflict info for LRA (through
6003 : ira_flattening call) and cannot use the check here. We could
6004 : rebuild this info for LRA in the check mode but there is a risk
6005 : that code generated with the check and without it will be a bit
6006 : different. Calling ira_flattening in any mode would be a
6007 : wasting CPU time. So do not check the allocation for LRA. */
6008 0 : check_allocation ();
6009 : #endif
6010 :
6011 1504950 : if (max_regno != max_regno_before_ira)
6012 35095 : regstat_recompute_for_max_regno ();
6013 :
6014 1504950 : overall_cost_before = ira_overall_cost;
6015 1504950 : if (! ira_conflicts_p)
6016 447891 : grow_reg_equivs ();
6017 : else
6018 : {
6019 1057059 : fix_reg_equiv_init ();
6020 :
6021 : #ifdef ENABLE_IRA_CHECKING
6022 1057059 : print_redundant_copies ();
6023 : #endif
6024 1057059 : if (! ira_use_lra_p)
6025 : {
6026 0 : ira_spilled_reg_stack_slots_num = 0;
6027 0 : ira_spilled_reg_stack_slots
6028 0 : = ((class ira_spilled_reg_stack_slot *)
6029 0 : ira_allocate (max_regno
6030 : * sizeof (class ira_spilled_reg_stack_slot)));
6031 0 : memset ((void *)ira_spilled_reg_stack_slots, 0,
6032 0 : max_regno * sizeof (class ira_spilled_reg_stack_slot));
6033 : }
6034 : }
6035 1504950 : allocate_initial_values ();
6036 :
6037 : /* See comment for find_moveable_pseudos call. */
6038 1504950 : if (ira_conflicts_p)
6039 1057059 : move_unallocated_pseudos ();
6040 :
6041 : /* Restore original values. */
6042 1504950 : if (lra_simple_p)
6043 : {
6044 5 : flag_caller_saves = saved_flag_caller_saves;
6045 5 : flag_ira_region = saved_flag_ira_region;
6046 : }
6047 1504950 : }
6048 :
6049 : /* Modify asm goto to avoid further trouble with this insn. We can
6050 : not replace the insn by USE as in other asm insns as we still
6051 : need to keep CFG consistency. */
6052 : void
6053 6 : ira_nullify_asm_goto (rtx_insn *insn)
6054 : {
6055 6 : ira_assert (JUMP_P (insn) && INSN_CODE (insn) < 0);
6056 6 : rtx tmp = extract_asm_operands (PATTERN (insn));
6057 6 : PATTERN (insn) = gen_rtx_ASM_OPERANDS (VOIDmode, ggc_strdup (""), "", 0,
6058 : rtvec_alloc (0),
6059 : rtvec_alloc (0),
6060 : ASM_OPERANDS_LABEL_VEC (tmp),
6061 : ASM_OPERANDS_SOURCE_LOCATION(tmp));
6062 6 : }
6063 :
6064 : static void
6065 1504950 : do_reload (void)
6066 : {
6067 1504950 : basic_block bb;
6068 1504950 : bool need_dce;
6069 1504950 : unsigned pic_offset_table_regno = INVALID_REGNUM;
6070 :
6071 1504950 : if (flag_ira_verbose < 10)
6072 1504950 : ira_dump_file = dump_file;
6073 :
6074 : /* If pic_offset_table_rtx is a pseudo register, then keep it so
6075 : after reload to avoid possible wrong usages of hard reg assigned
6076 : to it. */
6077 1504950 : if (pic_offset_table_rtx
6078 1504950 : && REGNO (pic_offset_table_rtx) >= FIRST_PSEUDO_REGISTER)
6079 : pic_offset_table_regno = REGNO (pic_offset_table_rtx);
6080 :
6081 1504950 : timevar_push (TV_RELOAD);
6082 1504950 : if (ira_use_lra_p)
6083 : {
6084 1504950 : if (current_loops != NULL)
6085 : {
6086 1010795 : loop_optimizer_finalize ();
6087 1010795 : free_dominance_info (CDI_DOMINATORS);
6088 : }
6089 19248389 : FOR_ALL_BB_FN (bb, cfun)
6090 17743439 : bb->loop_father = NULL;
6091 1504950 : current_loops = NULL;
6092 :
6093 1504950 : ira_destroy ();
6094 :
6095 1504950 : lra (ira_dump_file, internal_flag_ira_verbose);
6096 : /* ???!!! Move it before lra () when we use ira_reg_equiv in
6097 : LRA. */
6098 1504950 : vec_free (reg_equivs);
6099 1504950 : reg_equivs = NULL;
6100 1504950 : need_dce = false;
6101 : }
6102 : else
6103 : {
6104 0 : df_set_flags (DF_NO_INSN_RESCAN);
6105 0 : build_insn_chain ();
6106 :
6107 0 : need_dce = reload (get_insns (), ira_conflicts_p);
6108 : }
6109 :
6110 1504950 : timevar_pop (TV_RELOAD);
6111 :
6112 1504950 : timevar_push (TV_IRA);
6113 :
6114 1504950 : if (ira_conflicts_p && ! ira_use_lra_p)
6115 : {
6116 0 : ira_free (ira_spilled_reg_stack_slots);
6117 0 : ira_finish_assign ();
6118 : }
6119 :
6120 1504950 : if (internal_flag_ira_verbose > 0 && ira_dump_file != NULL
6121 96 : && overall_cost_before != ira_overall_cost)
6122 0 : fprintf (ira_dump_file, "+++Overall after reload %" PRId64 "\n",
6123 : ira_overall_cost);
6124 :
6125 1504950 : flag_ira_share_spill_slots = saved_flag_ira_share_spill_slots;
6126 :
6127 1504950 : if (! ira_use_lra_p)
6128 : {
6129 0 : ira_destroy ();
6130 0 : if (current_loops != NULL)
6131 : {
6132 0 : loop_optimizer_finalize ();
6133 0 : free_dominance_info (CDI_DOMINATORS);
6134 : }
6135 0 : FOR_ALL_BB_FN (bb, cfun)
6136 0 : bb->loop_father = NULL;
6137 0 : current_loops = NULL;
6138 :
6139 0 : regstat_free_ri ();
6140 0 : regstat_free_n_sets_and_refs ();
6141 : }
6142 :
6143 1504950 : if (optimize)
6144 1057059 : cleanup_cfg (CLEANUP_EXPENSIVE);
6145 :
6146 1504950 : finish_reg_equiv ();
6147 :
6148 1504950 : bitmap_obstack_release (&ira_bitmap_obstack);
6149 : #ifndef IRA_NO_OBSTACK
6150 : obstack_free (&ira_obstack, NULL);
6151 : #endif
6152 :
6153 : /* The code after the reload has changed so much that at this point
6154 : we might as well just rescan everything. Note that
6155 : df_rescan_all_insns is not going to help here because it does not
6156 : touch the artificial uses and defs. */
6157 1504950 : df_finish_pass (true);
6158 1504950 : df_scan_alloc (NULL);
6159 1504950 : df_scan_blocks ();
6160 :
6161 1504950 : if (optimize > 1)
6162 : {
6163 976481 : df_live_add_problem ();
6164 976481 : df_live_set_all_dirty ();
6165 : }
6166 :
6167 1504950 : if (optimize)
6168 1057059 : df_analyze ();
6169 :
6170 1504950 : if (need_dce && optimize)
6171 0 : run_fast_dce ();
6172 :
6173 : /* Diagnose uses of the hard frame pointer when it is used as a global
6174 : register. Often we can get away with letting the user appropriate
6175 : the frame pointer, but we should let them know when code generation
6176 : makes that impossible. */
6177 1504950 : if (global_regs[HARD_FRAME_POINTER_REGNUM] && frame_pointer_needed)
6178 : {
6179 2 : tree decl = global_regs_decl[HARD_FRAME_POINTER_REGNUM];
6180 2 : error_at (DECL_SOURCE_LOCATION (current_function_decl),
6181 : "frame pointer required, but reserved");
6182 2 : inform (DECL_SOURCE_LOCATION (decl), "for %qD", decl);
6183 : }
6184 :
6185 : /* If we are doing generic stack checking, give a warning if this
6186 : function's frame size is larger than we expect. */
6187 1504950 : if (flag_stack_check == GENERIC_STACK_CHECK)
6188 : {
6189 48 : poly_int64 size = get_frame_size () + STACK_CHECK_FIXED_FRAME_SIZE;
6190 :
6191 4464 : for (int i = 0; i < FIRST_PSEUDO_REGISTER; i++)
6192 4416 : if (df_regs_ever_live_p (i)
6193 233 : && !fixed_regs[i]
6194 4585 : && !crtl->abi->clobbers_full_reg_p (i))
6195 83 : size += UNITS_PER_WORD;
6196 :
6197 48 : if (constant_lower_bound (size) > STACK_CHECK_MAX_FRAME_SIZE)
6198 1 : warning (0, "frame size too large for reliable stack checking");
6199 : }
6200 :
6201 1504950 : if (pic_offset_table_regno != INVALID_REGNUM)
6202 80971 : pic_offset_table_rtx = gen_rtx_REG (Pmode, pic_offset_table_regno);
6203 :
6204 1504950 : timevar_pop (TV_IRA);
6205 1504950 : }
6206 :
6207 : /* Run the integrated register allocator. */
6208 :
6209 : namespace {
6210 :
6211 : const pass_data pass_data_ira =
6212 : {
6213 : RTL_PASS, /* type */
6214 : "ira", /* name */
6215 : OPTGROUP_NONE, /* optinfo_flags */
6216 : TV_IRA, /* tv_id */
6217 : 0, /* properties_required */
6218 : 0, /* properties_provided */
6219 : 0, /* properties_destroyed */
6220 : 0, /* todo_flags_start */
6221 : TODO_do_not_ggc_collect, /* todo_flags_finish */
6222 : };
6223 :
6224 : class pass_ira : public rtl_opt_pass
6225 : {
6226 : public:
6227 292371 : pass_ira (gcc::context *ctxt)
6228 584742 : : rtl_opt_pass (pass_data_ira, ctxt)
6229 : {}
6230 :
6231 : /* opt_pass methods: */
6232 1504958 : bool gate (function *) final override
6233 : {
6234 1504958 : return !targetm.no_register_allocation;
6235 : }
6236 1504950 : unsigned int execute (function *) final override
6237 : {
6238 1504950 : ira_in_progress = true;
6239 1504950 : ira (dump_file);
6240 1504950 : ira_in_progress = false;
6241 1504950 : return 0;
6242 : }
6243 :
6244 : }; // class pass_ira
6245 :
6246 : } // anon namespace
6247 :
6248 : rtl_opt_pass *
6249 292371 : make_pass_ira (gcc::context *ctxt)
6250 : {
6251 292371 : return new pass_ira (ctxt);
6252 : }
6253 :
6254 : namespace {
6255 :
6256 : const pass_data pass_data_reload =
6257 : {
6258 : RTL_PASS, /* type */
6259 : "reload", /* name */
6260 : OPTGROUP_NONE, /* optinfo_flags */
6261 : TV_RELOAD, /* tv_id */
6262 : 0, /* properties_required */
6263 : 0, /* properties_provided */
6264 : 0, /* properties_destroyed */
6265 : 0, /* todo_flags_start */
6266 : 0, /* todo_flags_finish */
6267 : };
6268 :
6269 : class pass_reload : public rtl_opt_pass
6270 : {
6271 : public:
6272 292371 : pass_reload (gcc::context *ctxt)
6273 584742 : : rtl_opt_pass (pass_data_reload, ctxt)
6274 : {}
6275 :
6276 : /* opt_pass methods: */
6277 1504958 : bool gate (function *) final override
6278 : {
6279 1504958 : return !targetm.no_register_allocation;
6280 : }
6281 1504950 : unsigned int execute (function *) final override
6282 : {
6283 1504950 : do_reload ();
6284 1504950 : return 0;
6285 : }
6286 :
6287 : }; // class pass_reload
6288 :
6289 : } // anon namespace
6290 :
6291 : rtl_opt_pass *
6292 292371 : make_pass_reload (gcc::context *ctxt)
6293 : {
6294 292371 : return new pass_reload (ctxt);
6295 : }
|
